From patchwork Tue Nov 17 14:41:09 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Clara Kowalsky X-Patchwork-Id: 5626 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 17 Nov 2026 15:42:59 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-ej1-f62.google.com (mail-ej1-f62.google.com [209.85.218.62]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 6AHEgxeK006894 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 17 Nov 2026 15:42:59 +0100 Received: by mail-ej1-f62.google.com with SMTP id a640c23a62f3a-c5101fd59e3sf62809766b.0 for ; Tue, 17 Nov 2026 06:42:59 -0800 (PST) ARC-Seal: i=3; a=rsa-sha256; t=1794926573; cv=pass; d=google.com; s=arc-20260327; b=jbptim70p1289Qk3/dZLUBbhw4ZkfhNGF+LR/U5Ygw4XCSW2yj3czm37cHU0kTTZge fXKyhnhkQMWyNho2Ko3yA6jyugYc1mDnLHb0lxIrRmUKANqlPIHaUbZnKdHBIpaVLWeS 0EslSOAJvTTqjs5rn6LQJRN5aDc1VBU7TdhKCZncdDmqJjm7COZO/qLhQWgyLBl0fL7X 0Scly3aGvrhoc8nZxzm7KPsjTGXqqm6f5jZ2//nSJJhgLSO0wGNGj9wVx9AA7Megp7f2 sweShYbEPYIraDb6/B+6yFDohCxmayBFCJohgelO1An048j54Q9ENNULv4oNt4S5s7BJ 4imQ== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:mime-version :content-transfer-encoding:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=HOTZ28tisJadMJOChEDhCP1zL35kO42sDk5XhBlqmxE=; fh=o4b33I5GxJQ1OYyAr8XiAl6kiACdnjfWQuRKz+DlfkE=; b=lMTXXMl+yzre3cWaxwuKlNo7Pj0LnvBfhfre26QyVrEPLw9g3ppgPJ2t4qJwWw+Y35 E9S0fHorgsDRVyBDPy3s9CuxvKolZo95FiO0uqAUiX8h35XVnENxmPWE9Uz7qh6UvQt/ 84R/BAzNt8BsFQNa9l5oYO1lf1JSkanz8ybCMCzj77FQNgfXAWaPUIl3dLfib93nc7Dc BTjpsxEDcK5dMBcLbXKTWEWFPTpO9Anwy9fxfdyGQKjs2X7PIU86El/pqHofyYSIbnsq gbeNFloeKLm8cBoLPpXrrTIn5NOmc9u/GuVj0HrU9rX4gVjvuvBkjOypNWkytdV9l7eF eHxQ==; darn=isar-build.org ARC-Authentication-Results: i=3; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=muTypZjC; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of clara.kowalsky@siemens.com designates 2a01:111:f403:c200::5 as permitted sender) smtp.mailfrom=clara.kowalsky@siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20260707; t=1794926572; x=1795531372; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:mime-version :content-transfer-encoding:content-type:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=HOTZ28tisJadMJOChEDhCP1zL35kO42sDk5XhBlqmxE=; b=O/WmeHCoWByr5uno0CagkNHoHm8ujJP69hqOHE2wlrGMRESi8W0RY9/MpUpG/pqKCi hS7NI4Q8S0phChavLY+xqBdCRlWS9V27ymJ1kGII6T8tdM9Freoj2B0WV7VokaqL1CpO KJV31uBrGwU7m25AJLJyTX1e4yZ0f8apA0v4ZDhgv1Dn60pvcn/GGUaH0eoCdWxj0xbD 84gLf6ZTO3unYnXCPM3OjHhOlS7ahn869ZRh4uKg2miVE2Grwnsbs9iAHqrjHM5C2JU+ s9IARX2m9IM0SE0NtYka0/B7z42IM3kg++wZrQCmYzRT/cWsPOAmb4iLRYGswWp2RGVT J38w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794926573; x=1795531373; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:mime-version :content-transfer-encoding:content-type:references:in-reply-to :message-id:date:subject:cc:to:from:x-beenthere:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to:content-type; bh=HOTZ28tisJadMJOChEDhCP1zL35kO42sDk5XhBlqmxE=; b=ewzTTO5W9s3eY9ayP27YkVjT50c7SL0LVvA61fOlXb/RQgXmrStShCUdo296HGe4qC 1WcYZ//tchh4oD03l+Igw4W4YZGg1YvebTFYRFxIQD1eSypg8oSnQ1AkmM+fMpi2OG61 N4VvrlG4JfHsADGIYGjgl10waNqb1Rk6UQ4Qs7fNZdl8z33LHlW8Vv5D6jYTz3vyjq// TeXRL3MAqxOnRn3zW8ofcjPnR9mJIc3Nfa+3q49mfSJaUvlfzAZ6LtfGFAHnB8hejVFr QOWsnaKPupo/DwtKflRVHMS+PMLGMB94E4uea41m0SskiEAZNth4Xx2bcbGVO1KWl/8r cYUA== X-Forwarded-Encrypted: i=3; AL+hErFl7B+C1Pi3T6PvSeRS9GiVeNbX7I3x/NIav1o4ge5tVsFT6WrYzIuM0rXDTQQKWEtJciLK4nY=@isar-build.org X-Gm-Message-State: APuU4Z9F4ZHKdso8029rNudF2RUkf5dim3bhNj0ysV8hBxMLg2vTzEBE Z6ACvxcD5xK5CcKSrlky3j+qK5e6CjIev5ZgPC8JzYQerUGb+Nf6/G5v X-Received: by 2002:a05:6402:c508:10b0:6b7:b872:27b6 with SMTP id 4fb4d7f45d1cf-6b7c0953947mr7735658a12.13.1794926572462; Tue, 17 Nov 2026 06:42:52 -0800 (PST) X-BeenThere: isar-users@googlegroups.com; h="ATskLde55KR3w0i4XuOmtdnMeA5UcdHKviFNuqXtiwRn+d5hEg==" Received: by 2002:a05:6402:90d:b0:6b8:6f03:ec19 with SMTP id 4fb4d7f45d1cf-6b86f03ecb2ls694981a12.1.-pod-prod-06-eu; Tue, 17 Nov 2026 06:42:49 -0800 (PST) X-Received: by 2002:a05:6402:c504:20b0:6b7:182c:5d4e with SMTP id 4fb4d7f45d1cf-6b7c088ac9dmr8167527a12.5.1794926569221; Tue, 17 Nov 2026 06:42:49 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1794926569; cv=pass; d=google.com; s=arc-20260327; b=ELvh345/gjcoyFa6Cdk3ZmSYaDHChjVnsiOToIDmIP7BJsrLia/mThCVMAF02bL4of ICIMKmeLBmJV+RBTkx4quz9K/lQaayYxiPNKUWs0SRm5XX99yISe+t1/qztlyofv/f4E VJL8/xJnzt+0ZUtCZa3BrdRqqx6RZRjFtOYb4eLWPqz42JBUNhzBhPXgvx6ewKma341D r9Qf8HbD+OKYmC/EGL4SInEFpRnFxFyCV4BUgb1Av9ImnpvYbsxNjinTRseKQEUOkp4p dDLhqVooppYq5O6x/8Tyt17ez8XbkeYHCuy1WQKXb84eCfJc1u5iGxUw30/Dt8TaB7Hw WIaw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=mime-version:content-transfer-encoding:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=mviQIzbDyFSJRqreLPzhfsKjdrzwdu78M8fw+Q389hc=; fh=gPnyYwFwya925hIKsufnyacyuqnFsKXzQMHP3i58878=; b=jRrWk55DJqTRSvIQfHthUQA6bhD2+mBccU+xdfCULvREMk8W5F/vOMkA6P1/avBsCe tSRAh079Szt6Je3wIsv+WviQO4MuS+LkYVn2xwe22K5JfSuuNcDbHui+FGM2JUNiH92z B9jimML/sodGFXOd9+X8s57n7T+9vRf4M74U7TjTry3teq111Puy2gF/4P6hHee+gDYR 9U4LAfQnFGwcEJXwS1dIRUPW64h4p1CPF7R2b3oJJ4xdH0bDv3wiPqAiSu/EjvSKqhnh xykjMQuJ/dji0DMq7x4gn6aD5C7wFmZRarhxNEjRIBdN7Fs9ODXAn/289N1KxjcUW8+8 O83w==; dara=google.com ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=muTypZjC; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of clara.kowalsky@siemens.com designates 2a01:111:f403:c200::5 as permitted sender) smtp.mailfrom=clara.kowalsky@siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from DUZPR83CU001.outbound.protection.outlook.com (mail-northeuropeazlp170120005.outbound.protection.outlook.com. [2a01:111:f403:c200::5]) by gmr-mx.google.com with ESMTPS id 4fb4d7f45d1cf-6b7c23eea10si184859a12.10.2026.11.17.06.42.48 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Nov 2026 06:42:49 -0800 (PST) Received-SPF: pass (google.com: domain of clara.kowalsky@siemens.com designates 2a01:111:f403:c200::5 as permitted sender) client-ip=2a01:111:f403:c200::5; ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=No2K6yE2CTj3I0yLQHJTKq4palnVqVudkiM4ApHE83JTjcJ385+RYl7//aIIto8pwQqyU5doRZzF7FH0S6VLQPkrNLZgLtI6NKtChhMuvTbCa7KbXvGE+h/3jpEXwUgAKzzcE/FhzvjzJ7teow3i3QAKzfXD5ZCAqsHYighiHLte/0MGfB8l/Pl5TeBJsOjyxZMM2Bupzsl0BfMX9xPnKCjKnuOlNmFL8KG0qaHXuDkKAjn9NI/0kin7KCd6LlOJIMRbyNzM1B/qNZsQSOUzyx7klKAYxUt4KaqNZrMxDk61fxIwWayihqgV2PIgSZ9OsSEMUD9ZvNhXV1HcXE9ZfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mviQIzbDyFSJRqreLPzhfsKjdrzwdu78M8fw+Q389hc=; b=P057YvvLpcqz/psqd9weYHwGdpLs3God5PcffN/SvgBiXG3jEUfMddCvhLFSqB+iGr0RTRjGau41Gl5jHRzOeaXnqkc6OmVpiH2Q99lFXjcHAtndMQEsqSF/J28g58Ji+BUyDn0rgK2dQv2PMYGxyvtz3HcQzmv4Ox5zB3PEzumumm50pFjqvqa8qhSXhsXFPE0jJPzhqKI+83wVVrWLRvzAl5LFdpss6H4x9M+vNL+LQtYO6a1w19KOGTa5yI00nvIsdpCJMvDDFrP6le+oEP7nOdrNQdigxsgLalrgmdDI+prSS6cJisOrbPQjlFbIHui6lK2QDOiz84nKjGm5+w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none Received: from AM0PR10MB9645.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:740::12) by AMDPR10MB992270.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:70d::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.639.8; Tue, 17 Nov 2026 14:42:37 +0000 Received: from AM0PR10MB9645.EURPRD10.PROD.OUTLOOK.COM ([fe80::527e:14a7:bd3e:86be]) by AM0PR10MB9645.EURPRD10.PROD.OUTLOOK.COM ([fe80::527e:14a7:bd3e:86be%6]) with mapi id 15.21.0639.006; Tue, 17 Nov 2026 14:42:37 +0000 X-Patchwork-Original-From: "'Clara Kowalsky' via isar-users" From: Clara Kowalsky To: isar-users@googlegroups.com Cc: Clara Kowalsky , Felix Moessbauer Subject: [PATCH v2 6/6] bitbake: update to branch 2.18, tag yocto-6.0.4 Date: Tue, 17 Nov 2026 15:41:09 +0100 Message-ID: <20261117144109.2417395-7-clara.kowalsky@siemens.com> X-Mailer: git-send-email 2.56.1 In-Reply-To: <20261117144109.2417395-1-clara.kowalsky@siemens.com> References: <20261117144109.2417395-1-clara.kowalsky@siemens.com> X-ClientProxiedBy: FR4P281CA0371.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:f8::7) To AM0PR10MB9645.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:740::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM0PR10MB9645:EE_|AMDPR10MB992270:EE_ X-MS-Office365-Filtering-Correlation-Id: 1ae32e71-45e9-43e1-f233-08df44b904b1 X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7049299003|3109299003|23010399003|1800799024|69100299015|366016|376014|261102123227099003|261102124527099003|261102130727099003|261028144727099003|261022113327099003|261022075727099003|261022115727099003|261021121427099003|261021123027099003|261021130227099003|261022105027099003|261021133027099003|261021135927099003|261026140227099003|261026130827099003|261026135527099003|261026121527099003|5023799004|11063799006|56012099006|10067099003|260925022911599003|260925021311599003|260925021911599003|17002099007|18002099003|22082099003|55112099003|20052099010|6133799003|10086099003|10136099003|12006099003|261009223327099003|25016099003|16056099006|261009221227099003|261009223027099003|261009220027099003|261009221727099003|9063799003|29003799003|13003099007|3023799007|4133799003|15866825006|191535003; X-Microsoft-Antispam-Message-Info: l+9gq2UEwD1jUAtWfBJl+BKHgDQ8rwCpZ+6VVNrLTlajQELK/8DDYxYR0ZGXT6rTqelRBH82jm66kWznuN9rNY7H8K+atJBk+NBdeXW0Q+Yv8J5l0rnP6UjiJp+ssMa60ZxFjoPs+Uej70OyPxiB+9TveIhoR6cPLJRotUSkHT1TcGsHA8F3wZDai084m68nhkBpq2xM8aO2Yz7FRErgWgozRghaqVzXRbP0m6utwn0ADcQru/BYhm2b1SNeN9yXRk3kPgKhbjTwC4Gi8KfHz8PaNZ599vA5IDEtdSKa3jv95aiK0bZl/fQ4bN9OT3UQ6OfHeYZGBJzzbr+Zj8m8QjGUBr70tfT3W99hp60qR078Ss8mrjf4UL5ucPh1PBPijfA1amHD+JVtT6U0orruud6PgT/FV2zjW8LcvufXLiJ+/eV+JNh1G/VKRDW5OFLRq49ivAxkCJWjuP1UUf8UsexEOWhekcacGjWssOBN4dVIVUkTs/GSInql3zZcIOqfBMrrGl5AXRCEq/1ZM1aG+TQXZlXeCI/rUdsHLYIX+NfYCXdJeKLwiEEXNFOY2x/hpFTfaWBUAJNQkZYYZTCjJ3ijWmlBN7K2FAScRPbQbISn8kpcv69xjcla6+IpWzyQIX/AYvssl0jhkzU43P0qQU15LfH/OaYVQRxPPP3118fNhh8arpAUMm21iumWJR6c X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM0PR10MB9645.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(7049299003)(3109299003)(23010399003)(1800799024)(69100299015)(366016)(376014)(261102123227099003)(261102124527099003)(261102130727099003)(261028144727099003)(261022113327099003)(261022075727099003)(261022115727099003)(261021121427099003)(261021123027099003)(261021130227099003)(261022105027099003)(261021133027099003)(261021135927099003)(261026140227099003)(261026130827099003)(261026135527099003)(261026121527099003)(5023799004)(11063799006)(56012099006)(10067099003)(260925022911599003)(260925021311599003)(260925021911599003)(17002099007)(18002099003)(22082099003)(55112099003)(20052099010)(6133799003)(10086099003)(10136099003)(12006099003)(261009223327099003)(25016099003)(16056099006)(261009221227099003)(261009223027099003)(261009220027099003)(261009221727099003)(9063799003)(29003799003)(13003099007)(3023799007)(4133799003)(15866825006)(191535003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: vIeHWmpCt4/h/rph8TTb2AidWNH730rKfChi3M/JfZ6vQBAVMIdn3Jl5oNJkn7Sdi6TCao3As4r7KOtq3+di0CG+OPPFx1HsQ84zJf+j6V8SbEcp9vgNgy3Dtj/SYYFijzOCEd964yDqUk5UOOQXQ5sGyvrdhrgQPZJvlKEysate+DcZhusqYLI622UQGIisvlYzvQDUAYFjSlFY0P+jRxSHHc0G+MxJ6Rd1IaLbYOP7YtpGR9XUZfhSbhTrCaLuRNpRxx6U6EfZ6BRPab9VctzncoCBMHvU30N1fFwXArq4pU+v2D076fhjUNj6FmTpRefHu3l9NCLQ0iLwFz5Vcia/N3wExvB+eDYVEPI3shvG0KI1sh5dfbnQ40aF42CxIA+CkaDMv7gLBFtikI60VJXNoqpAnZ2kPVVjMb6+A5fPB3cO1D0Qm34FvLBOeGwfy5jbxtt08TjGyj6dxIWolrGdNaGCxmOAlUxHLc0RSl79o1ry2GmUNmkExKVe0ZEskzEl4o/pY0x++n7HYN32GU/Sjtsv5a12yKmRd6He7U5rl4k8HRm6ABcYOhFx9KjhhdZFunDIF31h8+KQP2E+IxW42SHh0fD9ZGLmoUcjLFVIlqRjmuEFwiTLDMJXDVg+p08RfJg1IQ+d9qFTSQuFKOl9Z7/W6ucRVUV5TXqY/PaA6bMjMK3q47GI3CR9Zgo5KvMS7ac6Dl/JKPBLuLAdAs3V9ZIBjqJBtisuWbbQ1O7CTdWmJVG+vHTNMJI7Tj8NV515E7pN66NtjM7b5k7ualEfTKEJZvxboLVCJbERQdxKNLwljepyaspzI76ZfO36ToPtsHfZDlcaDmlPtArsnQg0yOA28mbP43RjgTSzBW/EyCZNypsMC6ITgNzYIhxHUKxiLGIcUQaG75+0ueHQcEbzbBKphVRINOzWMGUUeeYMr16OCY62A1bnRE6oRs++c1FgKkNFnWCaU51Eeesz1lgC5CSckMOEYqQPIN2nNyb9mkDLnL2xPrOiO6ck/BjP/imfZ7XK0oq9hGucIbtDJId2MC56AHJas7u1N4IYPqJD5F/sTVoB9W7PSH5koGhJwpmiytRlC+p2f92Wm1QKCqmnurcH5SmkSzAIL+2P6cACuX1qS6oGpprUVgNCru6K3zoKrgbJKEZr0KmKnjTVRdqrDjhJqL8OB+1aCtznTGG8e9kw9YPNAxtXgR9naLhbVQYnFn7/oHAOIFiCaB3P4AB3rT+bKrAsAiyJWU3ymKh9viz5QKCFxI6NwpXvZJv3sOEXYTsVOofFx1QKGwDLsojVslaukMT7NH9KGFzn0OsWQogK+uc0zkkEM6gr7S0TcFMi6Slg6rhfRkzeZ4oYfZID3Huglv88d6pslVd6tkLyBpJiBc/ye+kFB3oltG+jX8eZJoRpQQkPZhjMaTk8GFG3s/2+z9SYywlY50xoU6EOATsQ2//ZqS2WNZI+R6DZFuzOn7CyPEIel6aDVddQXeFC5Hj26QgPoB6ytSSBHmoZqR5fA/DBVCkT3AvVWgbMVo3D0/ZNIcHrSeEYkyhZhP6HEXa4IMDJ7bDQj2jAXQRdi3hICSp1LhcSMHk8RnwoHW+UVZBgn3b0n/wpXLFY0gH3HVieGZ8YmmTbfk0gqhRhjDtkSqOONJmOtAo85O6ya178v68UksDkbzjCgIfChCAMdFY+eUmVMUac6rqhScp0Cf0YZ9fpK3M1JELVan0l9NWUgm0NdcUAgzzAQBijq+IH1+x+AwKc9sQGV4fL8q0= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1ae32e71-45e9-43e1-f233-08df44b904b1 X-MS-Exchange-CrossTenant-AuthSource: AM0PR10MB9645.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Nov 2026 14:42:37.0090 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: FA9T1Llr8K9dqEOLlaxCCpTnuphrgHLHXu5S2GQaNwP3vNEr0DAsUNfiIgefjGm7sEF8smooTUj/gjzqcnC+aHE3wesrFw0HFZ7IrCZ663Q= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AMDPR10MB992270 X-Original-Sender: clara.kowalsky@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=muTypZjC; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of clara.kowalsky@siemens.com designates 2a01:111:f403:c200::5 as permitted sender) smtp.mailfrom=clara.kowalsky@siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Clara Kowalsky Reply-To: Clara Kowalsky Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= Update BitBake from v2.8.0 to branch 2.18, tag yocto-6.0.4. This is upstream commit 51fd94a67fa17650a9ff623492005972bbc7d26f. Drop Isar's lowered Python version check and retain upstream's Python 3.8 minimum. WIC now runs with the build host's Python, so imaging Debian buster no longer requires this workaround. In gitignore, set /build* so Git does not ignore BitBake's new bs4/builder directory. Signed-off-by: Clara Kowalsky Tested-by: Felix Moessbauer --- .gitignore | 2 +- bitbake/.b4-config | 5 + bitbake/.gitignore | 1 + bitbake/.vscode/settings.json | 14 + bitbake/README | 4 +- bitbake/TODO | 65 +- bitbake/bin/bitbake | 2 +- bitbake/bin/bitbake-config-build | 1 + bitbake/bin/bitbake-diffsigs | 9 +- bitbake/bin/bitbake-getvar | 15 +- bitbake/bin/bitbake-hashclient | 107 +- bitbake/bin/bitbake-hashserv | 10 +- bitbake/bin/bitbake-layers | 40 +- bitbake/bin/bitbake-prserv | 26 +- bitbake/bin/bitbake-selftest | 5 +- bitbake/bin/bitbake-server | 8 +- bitbake/bin/bitbake-setup | 1384 +++++++ bitbake/bin/bitbake-worker | 33 +- bitbake/bin/git-make-shallow | 4 +- bitbake/bin/toaster | 21 +- bitbake/contrib/b4-wrapper-bitbake.py | 40 + bitbake/contrib/hashserv/Dockerfile | 2 +- bitbake/contrib/vim/ftdetect/bitbake.vim | 14 +- .../oe-nodistro-master.conf.json | 47 + .../oe-nodistro-whinlatter.conf.json | 56 + .../oe-nodistro-wrynose.conf.json | 48 + .../configurations/poky-master.conf.json | 73 + .../configurations/poky-whinlatter.conf.json | 80 + .../configurations/poky-wrynose.conf.json | 74 + bitbake/doc/.gitignore | 3 + bitbake/doc/Makefile | 2 +- .../bitbake-user-manual-environment-setup.rst | 1296 ++++++ .../bitbake-user-manual-execution.rst | 35 +- .../bitbake-user-manual-fetching.rst | 287 +- .../bitbake-user-manual-intro.rst | 267 +- .../bitbake-user-manual-library-functions.rst | 59 + .../bitbake-user-manual-metadata.rst | 394 +- .../bitbake-user-manual-ref-variables.rst | 379 +- bitbake/doc/conf.py | 41 +- bitbake/doc/index.rst | 2 + bitbake/doc/releases.rst | 174 - bitbake/doc/setversions.py | 356 ++ bitbake/doc/sphinx-static/switchers.js | 233 -- bitbake/doc/sphinx-static/switchers.js.in | 249 ++ bitbake/lib/bb/COW.py | 10 +- bitbake/lib/bb/__init__.py | 144 +- bitbake/lib/bb/acl.py | 2 - bitbake/lib/bb/asyncrpc/__init__.py | 2 +- bitbake/lib/bb/asyncrpc/client.py | 162 +- bitbake/lib/bb/asyncrpc/serv.py | 44 +- bitbake/lib/bb/build.py | 27 +- bitbake/lib/bb/cache.py | 35 +- bitbake/lib/bb/checksum.py | 25 + bitbake/lib/bb/codeparser.py | 97 +- bitbake/lib/bb/command.py | 39 +- bitbake/lib/bb/compress/lz4.py | 4 +- bitbake/lib/bb/cooker.py | 280 +- bitbake/lib/bb/cookerdata.py | 30 +- bitbake/lib/bb/daemonize.py | 3 + bitbake/lib/bb/data.py | 19 +- bitbake/lib/bb/data_smart.py | 147 +- bitbake/lib/bb/event.py | 29 +- bitbake/lib/bb/fetch2/__init__.py | 345 +- bitbake/lib/bb/fetch2/az.py | 9 +- bitbake/lib/bb/fetch2/bzr.py | 128 - bitbake/lib/bb/fetch2/clearcase.py | 6 +- bitbake/lib/bb/fetch2/crate.py | 56 +- bitbake/lib/bb/fetch2/cvs.py | 157 - bitbake/lib/bb/fetch2/gcp.py | 15 +- bitbake/lib/bb/fetch2/git.py | 553 ++- bitbake/lib/bb/fetch2/gitsm.py | 127 +- bitbake/lib/bb/fetch2/gomod.py | 273 ++ bitbake/lib/bb/fetch2/local.py | 9 +- bitbake/lib/bb/fetch2/npm.py | 30 +- bitbake/lib/bb/fetch2/npmsw.py | 104 +- bitbake/lib/bb/fetch2/osc.py | 165 - bitbake/lib/bb/fetch2/s3.py | 2 +- bitbake/lib/bb/fetch2/sftp.py | 2 +- bitbake/lib/bb/fetch2/ssh.py | 3 +- bitbake/lib/bb/fetch2/svn.py | 5 +- bitbake/lib/bb/fetch2/wget.py | 195 +- bitbake/lib/bb/filter.py | 142 + bitbake/lib/bb/main.py | 6 +- bitbake/lib/bb/parse/__init__.py | 49 +- bitbake/lib/bb/parse/ast.py | 193 +- bitbake/lib/bb/parse/parse_py/BBHandler.py | 53 +- bitbake/lib/bb/parse/parse_py/ConfHandler.py | 24 +- bitbake/lib/bb/persist_data.py | 271 -- bitbake/lib/bb/runqueue.py | 345 +- bitbake/lib/bb/server/process.py | 86 +- bitbake/lib/bb/server/xmlrpcserver.py | 11 +- bitbake/lib/bb/siggen.py | 93 +- bitbake/lib/bb/taskdata.py | 8 +- bitbake/lib/bb/tests/codeparser.py | 40 + bitbake/lib/bb/tests/compression.py | 4 +- bitbake/lib/bb/tests/data.py | 207 +- .../software/miniupnp/download.php | 3528 +++++++++++++++++ bitbake/lib/bb/tests/fetch.py | 2069 ++++++++-- bitbake/lib/bb/tests/filter.py | 88 + .../bb/tests/parse-tests/classes/base.bbclass | 5 + .../classes/recipe-file-class.bbclass | 2 + .../bb/tests/parse-tests/conf/bitbake.conf | 15 + bitbake/lib/bb/tests/parse.py | 413 +- bitbake/lib/bb/tests/persist_data.py | 129 - .../tests/runqueue-tests/classes/base.bbclass | 4 +- .../lib/bb/tests/runqueue-tests/recipes/g1.bb | 2 + .../lib/bb/tests/runqueue-tests/recipes/h1.bb | 0 bitbake/lib/bb/tests/runqueue.py | 11 +- bitbake/lib/bb/tests/setup.py | 728 ++++ bitbake/lib/bb/tests/siggen.py | 47 + bitbake/lib/bb/tests/support/httpserver.py | 4 +- bitbake/lib/bb/tests/utils.py | 19 + bitbake/lib/bb/tinfoil.py | 192 +- bitbake/lib/bb/ui/buildinfohelper.py | 5 +- bitbake/lib/bb/ui/knotty.py | 199 +- bitbake/lib/bb/ui/taskexp_ncurses.py | 3 + bitbake/lib/bb/ui/teamcity.py | 14 +- bitbake/lib/bb/ui/uihelper.py | 12 +- bitbake/lib/bb/utils.py | 813 +++- bitbake/lib/bblayers/action.py | 4 +- bitbake/lib/bblayers/layerindex.py | 12 +- bitbake/lib/bblayers/query.py | 37 +- bitbake/lib/bs4/AUTHORS | 49 + bitbake/lib/bs4/AUTHORS.txt | 43 - bitbake/lib/bs4/{NEWS.txt => CHANGELOG} | 779 +++- bitbake/lib/bs4/{COPYING.txt => LICENSE} | 11 +- bitbake/lib/bs4/__init__.py | 681 +++- bitbake/lib/bs4/builder/__init__.py | 382 +- bitbake/lib/bs4/builder/_html5lib.py | 251 +- bitbake/lib/bs4/builder/_htmlparser.py | 433 +- bitbake/lib/bs4/builder/_lxml.py | 212 +- bitbake/lib/bs4/css.py | 274 ++ bitbake/lib/bs4/dammit.py | 411 +- bitbake/lib/bs4/diagnose.py | 84 +- bitbake/lib/bs4/element.py | 2219 +++++++---- bitbake/lib/bs4/formatter.py | 185 + bitbake/lib/bs4/testing.py | 686 ---- bitbake/lib/bs4/tests/__init__.py | 1 - .../lib/bs4/tests/test_builder_registry.py | 147 - bitbake/lib/bs4/tests/test_docs.py | 32 - bitbake/lib/bs4/tests/test_html5lib.py | 98 - bitbake/lib/bs4/tests/test_htmlparser.py | 31 - bitbake/lib/bs4/tests/test_lxml.py | 70 - bitbake/lib/bs4/tests/test_soup.py | 479 --- bitbake/lib/bs4/tests/test_tree.py | 2004 ---------- bitbake/lib/hashserv/__init__.py | 13 +- bitbake/lib/hashserv/client.py | 220 +- bitbake/lib/hashserv/server.py | 56 +- bitbake/lib/hashserv/sqlite.py | 17 + bitbake/lib/hashserv/tests.py | 295 +- bitbake/lib/layerindexlib/__init__.py | 6 +- bitbake/lib/layerindexlib/restapi.py | 2 +- bitbake/lib/ply/yacc.py | 1 - bitbake/lib/progressbar/progressbar.py | 16 +- bitbake/lib/prserv/__init__.py | 97 +- bitbake/lib/prserv/client.py | 15 +- bitbake/lib/prserv/db.py | 452 +-- bitbake/lib/prserv/serv.py | 140 +- bitbake/lib/prserv/tests.py | 388 ++ .../bldcontrol/localhostbecontroller.py | 154 +- .../toaster/orm/fixtures/check_fixtures.py | 38 + .../lib/toaster/orm/fixtures/gen_fixtures.py | 57 +- bitbake/lib/toaster/orm/fixtures/oe-core.xml | 52 +- bitbake/lib/toaster/orm/fixtures/poky.xml | 166 +- .../orm/management/commands/lsupdates.py | 9 +- bitbake/lib/toaster/orm/models.py | 3 +- .../tests/browser/selenium_helpers_base.py | 50 +- .../tests/browser/test_all_builds_page.py | 7 +- .../tests/browser/test_all_projects_page.py | 12 +- .../tests/browser/test_builddashboard_page.py | 2 +- .../tests/browser/test_landing_page.py | 14 +- .../tests/browser/test_layerdetails_page.py | 52 +- .../browser/test_new_custom_image_page.py | 2 +- .../tests/browser/test_new_project_page.py | 12 +- bitbake/lib/toaster/tests/builds/buildtest.py | 2 +- .../tests/functional/functional_helpers.py | 86 + .../functional/test_create_new_project.py | 106 +- .../tests/functional/test_functional_basic.py | 198 +- .../tests/functional/test_project_config.py | 97 +- .../tests/functional/test_project_page.py | 415 +- .../test_project_page_tab_config.py | 311 +- bitbake/lib/toaster/tests/functional/utils.py | 21 +- .../tests/toaster-tests-requirements.txt | 2 + .../toastergui/templatetags/projecttags.py | 1 - bitbake/lib/toaster/toastergui/views.py | 18 +- bitbake/lib/toaster/toastermain/settings.py | 1 - .../setup-schema/bitbake-setup.schema.json | 142 + bitbake/setup-schema/layers.schema.json | 91 + 188 files changed, 22475 insertions(+), 10063 deletions(-) create mode 100644 bitbake/.b4-config create mode 100644 bitbake/.vscode/settings.json create mode 120000 bitbake/bin/bitbake-config-build create mode 100755 bitbake/bin/bitbake-setup create mode 100755 bitbake/contrib/b4-wrapper-bitbake.py create mode 100644 bitbake/default-registry/configurations/oe-nodistro-master.conf.json create mode 100644 bitbake/default-registry/configurations/oe-nodistro-whinlatter.conf.json create mode 100644 bitbake/default-registry/configurations/oe-nodistro-wrynose.conf.json create mode 100644 bitbake/default-registry/configurations/poky-master.conf.json create mode 100644 bitbake/default-registry/configurations/poky-whinlatter.conf.json create mode 100644 bitbake/default-registry/configurations/poky-wrynose.conf.json create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-library-functions.rst delete mode 100644 bitbake/doc/releases.rst create mode 100755 bitbake/doc/setversions.py delete mode 100644 bitbake/doc/sphinx-static/switchers.js create mode 100644 bitbake/doc/sphinx-static/switchers.js.in delete mode 100644 bitbake/lib/bb/fetch2/bzr.py delete mode 100644 bitbake/lib/bb/fetch2/cvs.py create mode 100644 bitbake/lib/bb/fetch2/gomod.py delete mode 100644 bitbake/lib/bb/fetch2/osc.py create mode 100644 bitbake/lib/bb/filter.py delete mode 100644 bitbake/lib/bb/persist_data.py create mode 100644 bitbake/lib/bb/tests/fetch-testdata/software/miniupnp/download.php create mode 100644 bitbake/lib/bb/tests/filter.py create mode 100644 bitbake/lib/bb/tests/parse-tests/classes/base.bbclass create mode 100644 bitbake/lib/bb/tests/parse-tests/classes/recipe-file-class.bbclass create mode 100644 bitbake/lib/bb/tests/parse-tests/conf/bitbake.conf delete mode 100644 bitbake/lib/bb/tests/persist_data.py create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/h1.bb create mode 100644 bitbake/lib/bb/tests/setup.py create mode 100644 bitbake/lib/bs4/AUTHORS delete mode 100644 bitbake/lib/bs4/AUTHORS.txt rename bitbake/lib/bs4/{NEWS.txt => CHANGELOG} (56%) rename bitbake/lib/bs4/{COPYING.txt => LICENSE} (80%) create mode 100644 bitbake/lib/bs4/css.py create mode 100644 bitbake/lib/bs4/formatter.py delete mode 100644 bitbake/lib/bs4/testing.py delete mode 100644 bitbake/lib/bs4/tests/__init__.py delete mode 100644 bitbake/lib/bs4/tests/test_builder_registry.py delete mode 100644 bitbake/lib/bs4/tests/test_docs.py delete mode 100644 bitbake/lib/bs4/tests/test_html5lib.py delete mode 100644 bitbake/lib/bs4/tests/test_htmlparser.py delete mode 100644 bitbake/lib/bs4/tests/test_lxml.py delete mode 100644 bitbake/lib/bs4/tests/test_soup.py delete mode 100644 bitbake/lib/bs4/tests/test_tree.py create mode 100644 bitbake/lib/prserv/tests.py create mode 100755 bitbake/lib/toaster/orm/fixtures/check_fixtures.py create mode 100644 bitbake/setup-schema/bitbake-setup.schema.json create mode 100644 bitbake/setup-schema/layers.schema.json diff --git a/.gitignore b/.gitignore index b5f6149e..f397ff51 100644 --- a/.gitignore +++ b/.gitignore @@ -4,4 +4,4 @@ __pycache__ .config.yaml* .kas_shell_history .isar_test_shell_history -build* +/build* diff --git a/bitbake/.b4-config b/bitbake/.b4-config new file mode 100644 index 00000000..4d9ab7bd --- /dev/null +++ b/bitbake/.b4-config @@ -0,0 +1,5 @@ +[b4] + send-series-to = bitbake-devel@lists.openembedded.org + send-auto-cc-cmd = ./contrib/b4-wrapper-bitbake.py send-auto-cc-cmd + prep-pre-flight-checks = disable-needs-checking + send-prefixes = 2.18 diff --git a/bitbake/.gitignore b/bitbake/.gitignore index fbf9eb20..a6a256b2 100644 --- a/bitbake/.gitignore +++ b/bitbake/.gitignore @@ -3,6 +3,7 @@ *~ pyshtables.py /doc/manual/html/ +**/bitbake-builds/ /build/ /bin/bitbakec *.swp diff --git a/bitbake/.vscode/settings.json b/bitbake/.vscode/settings.json new file mode 100644 index 00000000..a5480a93 --- /dev/null +++ b/bitbake/.vscode/settings.json @@ -0,0 +1,14 @@ +{ + "python.autoComplete.extraPaths": [ + "${workspaceFolder}/sources/poky/bitbake/lib", + "${workspaceFolder}/sources/poky/meta/lib" + ], + "python.analysis.extraPaths": [ + "${workspaceFolder}/sources/poky/bitbake/lib", + "${workspaceFolder}/sources/poky/meta/lib" + ], + "files.associations": { + "*.conf": "bitbake", + "*.inc": "bitbake" + } +} \ No newline at end of file diff --git a/bitbake/README b/bitbake/README index e9f4c858..0701dbbe 100644 --- a/bitbake/README +++ b/bitbake/README @@ -24,12 +24,12 @@ for full details on how to submit changes. As a quick guide, patches should be sent to bitbake-devel@lists.openembedded.org The git command to do that would be: - git send-email -M -1 --to bitbake-devel@lists.openembedded.org + git send-email -M -1 --to bitbake-devel@lists.openembedded.org --subject-prefix="2.18][PATCH If you're sending a patch related to the BitBake manual, make sure you copy the Yocto Project documentation mailing list: - git send-email -M -1 --to bitbake-devel@lists.openembedded.org --cc docs@lists.yoctoproject.org + git send-email -M -1 --to bitbake-devel@lists.openembedded.org --cc docs@lists.yoctoproject.org --subject-prefix="2.18][PATCH Mailing list: diff --git a/bitbake/TODO b/bitbake/TODO index 54d091fb..4e0bd9a7 100644 --- a/bitbake/TODO +++ b/bitbake/TODO @@ -1,36 +1,31 @@ -- Reimplement the interactive mode as a proper ui -- Continue dropping fatal/SystemExit/sys.exit usage in favor of raising - appropriate exceptions -- Continue pylint / pyflakes / pychecker / pep8 fixups + +- Interfaces + - Add an interactive mode to knotty (similar to the interactive mode we dropped) + - Fix up or drop the ncurses interface? Currently too limited to be useful. + - Concerns about whether teamcity interface works. No testing. + +- Improve exception handling: + - dropping fatal/SystemExit/sys.exit usage in favor of raising + appropriate exceptions + - Audit bb.fatal usage - these should all be able to be replaced with + exceptions + +- Continue pylint / pyflakes / pychecker / pep8 fixups - focus on real issues + or performance issues, not cosmetic ones. + - Drop os.system usage in favor of direct subprocess usage or a subprocess wrapper -- Kill the execution of 'tee' for the task log file in build.py -- Fix up the exception handling - - - Kill exec_task's catch of FuncFailed, instead catch it in the other - callers of exec_task/exec_func - - What exactly is the purpose of the "EventException"? I can see using an - exception like that, *perhaps*, to abstract away exceptions raised by - event handlers, but it has no place in bb.build.exec_task - BUG: if you chmod 000 local.conf, it silently doesn't parse it, when it should really fail, so the user can fix the problem. -- Audit bb.fatal usage - these should all be able to be replaced with - exceptions -- Figure out how to handle the ncurses UI. Should some of our logging - formatting stuff be made common to all of bitbake, or perhaps all UIs via - the UIHelper? Long term, high impact: - - Change override application to actually *move* it over -- so the original - override specific version of the variable goes away, rather than sticking - around as a duplicate. - - Change the behavior when a variable is referenced and is unset. Today, it - evaluates to ${FOO} and then shell has a chance to expand it, but this is - far from ideal. We had considered evaluating it to the empty string, but - that has other potential problems. Frans Meulenbroeks has proposed just + - Consider changing the behavior when a variable is referenced and is unset + Today, it evaluates to ${FOO} and then shell has a chance to expand it, but + this is far from ideal. We had considered evaluating it to the empty string, + but that has other potential problems. Frans Meulenbroeks has proposed just erroring when this occurs, as we can always define default values for the variables in bitbake.conf. This seems reasonable. My only concern with that is the case where you want to reference a shell variable with odd @@ -38,25 +33,3 @@ Long term, high impact: expansion rather than normal $. To handle that case, we'd really need a way to escape / disable bitbake variable expansion, \${} perhaps. -Uncertain: - - - Leverage the python 2.6 multiprocessing module - - - Worker processes for bb.cooker - - Server / UI processes - - - Create a bitbake configuration class which is utilized by the library, not - just bin/bitbake. This class should be responsible for extracting - configuration parameters from the metadata for bitbake internal use, as well - as pulling specific items like BBDEBUG, and importing settings from an - optparse options object. - - - Python version bits - - - Utilize the new string formatting where appropriate - - Do we need to take into account the bytes literals changes? - - Do we have any file-like objects that would benefit from using the "io" - module? - - Do we want to leverage the abstract base classes in collections? - - Aside: Set methods now accept multiple iterables - diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake index f494eaa1..5a820417 100755 --- a/bitbake/bin/bitbake +++ b/bitbake/bin/bitbake @@ -27,7 +27,7 @@ from bb.main import bitbake_main, BitBakeConfigParameters, BBMainException bb.utils.check_system_locale() -__version__ = "2.8.0" +__version__ = "2.18.0" if __name__ == "__main__": if __version__ != bb.__version__: diff --git a/bitbake/bin/bitbake-config-build b/bitbake/bin/bitbake-config-build new file mode 120000 index 00000000..11e6df80 --- /dev/null +++ b/bitbake/bin/bitbake-config-build @@ -0,0 +1 @@ +bitbake-layers \ No newline at end of file diff --git a/bitbake/bin/bitbake-diffsigs b/bitbake/bin/bitbake-diffsigs index 8202c786..9d6cb8c9 100755 --- a/bitbake/bin/bitbake-diffsigs +++ b/bitbake/bin/bitbake-diffsigs @@ -72,16 +72,17 @@ def find_siginfo_task(bbhandler, pn, taskname, sig1=None, sig2=None): elif sig2 not in sigfiles: logger.error('No sigdata files found matching %s %s with signature %s' % (pn, taskname, sig2)) sys.exit(1) + + latestfiles = [sigfiles[sig1]['path'], sigfiles[sig2]['path']] else: sigfiles = find_siginfo(bbhandler, pn, taskname) latestsigs = sorted(sigfiles.keys(), key=lambda h: sigfiles[h]['time'])[-2:] if not latestsigs: logger.error('No sigdata files found matching %s %s' % (pn, taskname)) sys.exit(1) - sig1 = latestsigs[0] - sig2 = latestsigs[1] - - latestfiles = [sigfiles[sig1]['path'], sigfiles[sig2]['path']] + latestfiles = [sigfiles[latestsigs[0]]['path']] + if len(latestsigs) > 1: + latestfiles.append(sigfiles[latestsigs[1]]['path']) return latestfiles diff --git a/bitbake/bin/bitbake-getvar b/bitbake/bin/bitbake-getvar index 8901f99a..378fb135 100755 --- a/bitbake/bin/bitbake-getvar +++ b/bitbake/bin/bitbake-getvar @@ -10,12 +10,14 @@ import io import os import sys import warnings +import logging warnings.simplefilter("default") bindir = os.path.dirname(__file__) topdir = os.path.dirname(bindir) sys.path[0:0] = [os.path.join(topdir, 'lib')] +import bb.providers import bb.tinfoil if __name__ == "__main__": @@ -37,13 +39,22 @@ if __name__ == "__main__": sys.exit("--flag only makes sense with --value") quiet = args.quiet or args.value + if quiet: + logger = logging.getLogger("BitBake") + logger.setLevel(logging.WARNING) + with bb.tinfoil.Tinfoil(tracking=True, setup_logging=not quiet) as tinfoil: if args.recipe: tinfoil.prepare(quiet=3 if quiet else 2) - d = tinfoil.parse_recipe(args.recipe) + try: + d = tinfoil.parse_recipe(args.recipe) + except bb.providers.NoProvider as e: + sys.exit(str(e)) else: tinfoil.prepare(quiet=2, config_only=True) - d = tinfoil.config_data + # Expand keys and run anonymous functions to get identical result to + # "bitbake -e" + d = tinfoil.finalizeData() value = None if args.flag: diff --git a/bitbake/bin/bitbake-hashclient b/bitbake/bin/bitbake-hashclient index 610787ed..b8755c57 100755 --- a/bitbake/bin/bitbake-hashclient +++ b/bitbake/bin/bitbake-hashclient @@ -16,6 +16,8 @@ import time import warnings import netrc import json +import statistics +import textwrap warnings.simplefilter("default") try: @@ -81,6 +83,7 @@ def main(): nonlocal found_hashes nonlocal missed_hashes nonlocal max_time + nonlocal times with hashserv.create_client(args.address) as client: for i in range(args.requests): @@ -98,29 +101,41 @@ def main(): else: missed_hashes += 1 - max_time = max(elapsed, max_time) + times.append(elapsed) pbar.update() max_time = 0 found_hashes = 0 missed_hashes = 0 lock = threading.Lock() - total_requests = args.clients * args.requests + times = [] start_time = time.perf_counter() - with ProgressBar(total=total_requests) as pbar: + with ProgressBar(total=args.clients * args.requests) as pbar: threads = [threading.Thread(target=thread_main, args=(pbar, lock), daemon=False) for _ in range(args.clients)] for t in threads: t.start() for t in threads: t.join() + total_elapsed = time.perf_counter() - start_time - elapsed = time.perf_counter() - start_time with lock: - print("%d requests in %.1fs. %.1f requests per second" % (total_requests, elapsed, total_requests / elapsed)) - print("Average request time %.8fs" % (elapsed / total_requests)) - print("Max request time was %.8fs" % max_time) - print("Found %d hashes, missed %d" % (found_hashes, missed_hashes)) + mean = statistics.mean(times) + median = statistics.median(times) + stddev = statistics.pstdev(times) + + print(f"Number of clients: {args.clients}") + print(f"Requests per client: {args.requests}") + print(f"Number of requests: {len(times)}") + print(f"Total elapsed time: {total_elapsed:.3f}s") + print(f"Total request rate: {len(times)/total_elapsed:.3f} req/s") + print(f"Average request time: {mean:.3f}s") + print(f"Median request time: {median:.3f}s") + print(f"Request time std dev: {stddev:.3f}s") + print(f"Maximum request time: {max(times):.3f}s") + print(f"Minimum request time: {min(times):.3f}s") + print(f"Hashes found: {found_hashes}") + print(f"Hashes missed: {missed_hashes}") if args.report: with ProgressBar(total=args.requests) as pbar: @@ -212,6 +227,27 @@ def main(): print("New hashes marked: %d" % result["count"]) return 0 + def handle_gc_mark_stream(args, client): + stdin = (l.strip() for l in sys.stdin) + marked_hashes = 0 + + try: + result = client.gc_mark_stream(args.mark, stdin) + marked_hashes = result["count"] + except ConnectionError: + logger.warning( + "Server doesn't seem to support `gc-mark-stream`. Sending " + "hashes sequentially using `gc-mark` API." + ) + for line in stdin: + pairs = line.split() + condition = dict(zip(pairs[::2], pairs[1::2])) + result = client.gc_mark(args.mark, condition) + marked_hashes += result["count"] + + print("New hashes marked: %d" % marked_hashes) + return 0 + def handle_gc_sweep(args, client): result = client.gc_sweep(args.mark) print("Removed %d rows" % result["count"]) @@ -225,7 +261,45 @@ def main(): print("true" if result else "false") return 0 - parser = argparse.ArgumentParser(description='Hash Equivalence Client') + def handle_ping(args, client): + times = [] + for i in range(1, args.count + 1): + if not args.quiet: + print(f"Ping {i} of {args.count}... ", end="") + start_time = time.perf_counter() + client.ping() + elapsed = time.perf_counter() - start_time + times.append(elapsed) + if not args.quiet: + print(f"{elapsed:.3f}s") + + mean = statistics.mean(times) + median = statistics.median(times) + std_dev = statistics.pstdev(times) + + if not args.quiet: + print("------------------------") + print(f"Number of pings: {len(times)}") + print(f"Average round trip time: {mean:.3f}s") + print(f"Median round trip time: {median:.3f}s") + print(f"Round trip time std dev: {std_dev:.3f}s") + print(f"Min time is: {min(times):.3f}s") + print(f"Max time is: {max(times):.3f}s") + return 0 + + parser = argparse.ArgumentParser( + formatter_class=argparse.RawDescriptionHelpFormatter, + description='Hash Equivalence Client', + epilog=textwrap.dedent( + """ + Possible ADDRESS options are: + unix://PATH Connect to UNIX domain socket at PATH + ws://HOST[:PORT] Connect to websocket at HOST:PORT (default port is 80) + wss://HOST[:PORT] Connect to secure websocket at HOST:PORT (default port is 443) + HOST:PORT Connect to TCP server at HOST:PORT + """ + ), + ) parser.add_argument('--address', default=DEFAULT_ADDRESS, help='Server address (default "%(default)s")') parser.add_argument('--log', default='WARNING', help='Set logging level') parser.add_argument('--login', '-l', metavar="USERNAME", help="Authenticate as USERNAME") @@ -313,6 +387,16 @@ def main(): help="Keep entries in table where KEY == VALUE") gc_mark_parser.set_defaults(func=handle_gc_mark) + gc_mark_parser_stream = subparsers.add_parser( + 'gc-mark-stream', + help=( + "Mark multiple hashes to be retained for garbage collection. Input should be provided via stdin, " + "with each line formatted as key-value pairs separated by spaces, for example 'column1 foo column2 bar'." + ) + ) + gc_mark_parser_stream.add_argument("mark", help="Mark for this garbage collection operation") + gc_mark_parser_stream.set_defaults(func=handle_gc_mark_stream) + gc_sweep_parser = subparsers.add_parser('gc-sweep', help="Perform garbage collection and delete any entries that are not marked") gc_sweep_parser.add_argument("mark", help="Mark for this garbage collection operation") gc_sweep_parser.set_defaults(func=handle_gc_sweep) @@ -322,6 +406,11 @@ def main(): unihash_exists_parser.add_argument("unihash", help="Unihash to check") unihash_exists_parser.set_defaults(func=handle_unihash_exists) + ping_parser = subparsers.add_parser('ping', help="Ping server") + ping_parser.add_argument("-n", "--count", type=int, help="Number of pings. Default is %(default)s", default=10) + ping_parser.add_argument("-q", "--quiet", action="store_true", help="Don't print each ping; only print results") + ping_parser.set_defaults(func=handle_ping) + args = parser.parse_args() logger = logging.getLogger('hashserv') diff --git a/bitbake/bin/bitbake-hashserv b/bitbake/bin/bitbake-hashserv index 4bfb7abf..01503736 100755 --- a/bitbake/bin/bitbake-hashserv +++ b/bitbake/bin/bitbake-hashserv @@ -125,6 +125,11 @@ The following permissions are supported by the server: default=os.environ.get("HASHSERVER_ADMIN_PASSWORD", None), help="Create default admin user with password ADMIN_PASSWORD ($HASHSERVER_ADMIN_PASSWORD)", ) + parser.add_argument( + "--reuseport", + action="store_true", + help="Enable SO_REUSEPORT, allowing multiple servers to bind to the same port for load balancing", + ) args = parser.parse_args() @@ -132,7 +137,9 @@ The following permissions are supported by the server: level = getattr(logging, args.log.upper(), None) if not isinstance(level, int): - raise ValueError("Invalid log level: %s (Try ERROR/WARNING/INFO/DEBUG)" % args.log) + raise ValueError( + "Invalid log level: %s (Try ERROR/WARNING/INFO/DEBUG)" % args.log + ) logger.setLevel(level) console = logging.StreamHandler() @@ -155,6 +162,7 @@ The following permissions are supported by the server: anon_perms=anon_perms, admin_username=args.admin_user, admin_password=args.admin_password, + reuseport=args.reuseport, ) server.serve_forever() return 0 diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers index d4b1d1aa..c49a5f30 100755 --- a/bitbake/bin/bitbake-layers +++ b/bitbake/bin/bitbake-layers @@ -18,24 +18,31 @@ import warnings warnings.simplefilter("default") bindir = os.path.dirname(__file__) +toolname = os.path.basename(__file__).split(".")[0] topdir = os.path.dirname(bindir) sys.path[0:0] = [os.path.join(topdir, 'lib')] import bb.tinfoil +import bb.cooker import bb.msg -logger = bb.msg.logger_create('bitbake-layers', sys.stdout) +logger = bb.msg.logger_create(toolname, sys.stdout) def main(): parser = argparse.ArgumentParser( - description="BitBake layers utility", + description={"bitbake-layers":"BitBake layers utility", "bitbake-config-build":"BitBake build configuration utility"}[toolname], epilog="Use %(prog)s --help to get help on a specific command", add_help=False) parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true') parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true') - parser.add_argument('-F', '--force', help='Force add without recipe parse verification', action='store_true') + if toolname == "bitbake-layers": + parser.add_argument('-F', '--force', help='Forced execution: can be specified multiple times. -F will force add without recipe parse verification and -FF will additionally force the run without layer parsing.', action='count') parser.add_argument('--color', choices=['auto', 'always', 'never'], default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', metavar='COLOR') + # Set the default for --force here rather than above so that it is always + # defined even if the option is not available. + parser.set_defaults(force=0) + global_args, unparsed_args = parser.parse_known_args() # Help is added here rather than via add_help=True, as we don't want it to @@ -57,18 +64,27 @@ def main(): level=logger.getEffectiveLevel()) plugins = [] - tinfoil = bb.tinfoil.Tinfoil(tracking=True) - tinfoil.logger.setLevel(logger.getEffectiveLevel()) - try: - tinfoil.prepare(True) - for path in ([topdir] + - tinfoil.config_data.getVar('BBPATH').split(':')): - pluginpath = os.path.join(path, 'lib', 'bblayers') + with bb.tinfoil.Tinfoil(tracking=True) as tinfoil: + tinfoil.logger.setLevel(logger.getEffectiveLevel()) + + if global_args.force > 1: + bbpaths = [] + else: + if toolname == "bitbake-config-build": + extra_features = [bb.cooker.CookerFeatures.SKIP_FRAGMENTS] + else: + extra_features = None + tinfoil.prepare(True, extra_features=extra_features) + bbpaths = tinfoil.config_data.getVar('BBPATH').split(':') + + for path in ([topdir] + bbpaths): + pluginbasepath = {"bitbake-layers":'bblayers', 'bitbake-config-build':'bbconfigbuild'}[toolname] + pluginpath = os.path.join(path, 'lib', pluginbasepath) bb.utils.load_plugins(logger, plugins, pluginpath) registered = False for plugin in plugins: - if hasattr(plugin, 'tinfoil_init'): + if hasattr(plugin, 'tinfoil_init') and global_args.force <= 1: plugin.tinfoil_init(tinfoil) if hasattr(plugin, 'register_commands'): registered = True @@ -86,8 +102,6 @@ def main(): tinfoil.config_data.enableTracking() return args.func(args) - finally: - tinfoil.shutdown() if __name__ == "__main__": diff --git a/bitbake/bin/bitbake-prserv b/bitbake/bin/bitbake-prserv index ad0a0694..3992e84e 100755 --- a/bitbake/bin/bitbake-prserv +++ b/bitbake/bin/bitbake-prserv @@ -16,11 +16,18 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), "lib import prserv import prserv.serv -VERSION = "1.1.0" +VERSION = "2.0.0" PRHOST_DEFAULT="0.0.0.0" PRPORT_DEFAULT=8585 +def init_logger(logfile, loglevel): + numeric_level = getattr(logging, loglevel.upper(), None) + if not isinstance(numeric_level, int): + raise ValueError("Invalid log level: %s" % loglevel) + FORMAT = "%(asctime)-15s %(message)s" + logging.basicConfig(level=numeric_level, filename=logfile, format=FORMAT) + def main(): parser = argparse.ArgumentParser( description="BitBake PR Server. Version=%s" % VERSION, @@ -70,12 +77,25 @@ def main(): action="store_true", help="open database in read-only mode", ) + parser.add_argument( + "-u", + "--upstream", + default=os.environ.get("PRSERV_UPSTREAM", None), + help="Upstream PR service (host:port)", + ) args = parser.parse_args() - prserv.init_logger(os.path.abspath(args.log), args.loglevel) + init_logger(os.path.abspath(args.log), args.loglevel) if args.start: - ret=prserv.serv.start_daemon(args.file, args.host, args.port, os.path.abspath(args.log), args.read_only) + ret=prserv.serv.start_daemon( + args.file, + args.host, + args.port, + os.path.abspath(args.log), + args.read_only, + args.upstream + ) elif args.stop: ret=prserv.serv.stop_daemon(args.host, args.port) else: diff --git a/bitbake/bin/bitbake-selftest b/bitbake/bin/bitbake-selftest index f25f23b1..fb7c57dd 100755 --- a/bitbake/bin/bitbake-selftest +++ b/bitbake/bin/bitbake-selftest @@ -15,6 +15,7 @@ import unittest try: import bb import hashserv + import prserv import layerindexlib except RuntimeError as exc: sys.exit(str(exc)) @@ -27,12 +28,14 @@ tests = ["bb.tests.codeparser", "bb.tests.event", "bb.tests.fetch", "bb.tests.parse", - "bb.tests.persist_data", "bb.tests.runqueue", + "bb.tests.setup", "bb.tests.siggen", "bb.tests.utils", "bb.tests.compression", + "bb.tests.filter", "hashserv.tests", + "prserv.tests", "layerindexlib.tests.layerindexobj", "layerindexlib.tests.restapi", "layerindexlib.tests.cooker"] diff --git a/bitbake/bin/bitbake-server b/bitbake/bin/bitbake-server index 454a3919..1428f72a 100755 --- a/bitbake/bin/bitbake-server +++ b/bitbake/bin/bitbake-server @@ -29,7 +29,7 @@ logfile = sys.argv[4] lockname = sys.argv[5] sockname = sys.argv[6] timeout = float(sys.argv[7]) -profile = bool(int(sys.argv[8])) +profile = sys.argv[8] xmlrpcinterface = (sys.argv[9], int(sys.argv[10])) if xmlrpcinterface[0] == "None": xmlrpcinterface = (None, xmlrpcinterface[1]) @@ -38,9 +38,9 @@ if xmlrpcinterface[0] == "None": with open('/dev/null', 'r') as si: os.dup2(si.fileno(), sys.stdin.fileno()) -so = open(logfile, 'a+') -os.dup2(so.fileno(), sys.stdout.fileno()) -os.dup2(so.fileno(), sys.stderr.fileno()) +with open(logfile, 'a+') as so: + os.dup2(so.fileno(), sys.stdout.fileno()) + os.dup2(so.fileno(), sys.stderr.fileno()) # Have stdout and stderr be the same so log output matches chronologically # and there aren't two seperate buffers diff --git a/bitbake/bin/bitbake-setup b/bitbake/bin/bitbake-setup new file mode 100755 index 00000000..4d75ad61 --- /dev/null +++ b/bitbake/bin/bitbake-setup @@ -0,0 +1,1384 @@ +#!/usr/bin/env python3 + +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import argparse +import configparser +import copy +import datetime +import functools +import glob +import json +import logging +import os +import shutil +import signal +import string +import subprocess +import sys +import textwrap +import time + + +bindir = os.path.abspath(os.path.dirname(__file__)) +sys.path[0:0] = [os.path.join(os.path.dirname(bindir), 'lib')] + +import bb.msg # noqa: E402 +import bb.process # noqa: E402 + +logger = bb.msg.logger_create('bitbake-setup', sys.stdout) + +# These settings can only be set in the global context +GLOBAL_ONLY_SETTINGS = ( + "top-dir-prefix", + "top-dir-name", +) + +def color_enabled() -> bool: + """ + Our logger has a BBLogFormatter formatter which holds whether color is + enabled or not. Return this value. + """ + return logger.handlers[0].formatter.color_enabled + +def get_diff_color_param() -> str: + return "--color=always" if color_enabled() else "--color=never" + +def print_configs(prompt: str, choices: list[str], descriptions: list[str] = []): + """ + Helper function to print a list of choices and align the output. + Each option name is made bold to stand out, unless color is not enabled in + our logger. + """ + if not prompt.endswith(':'): + prompt += ":" + logger.plain(prompt) + + if not descriptions: + descriptions = ["" for _ in choices] + + # maximum size of all choices, for alignment + cmax = max([len(c) for c in choices]) + 1 + + for n, c in enumerate(choices): + msg = f"{n + 1}. " + if color_enabled(): + # make it bold + msg += "\033[1m" + msg += f"{c:<{cmax}}" + if color_enabled(): + msg += "\033[0m" + msg += f" {descriptions[n]}" + logger.plain(msg) + +# If bitbake is from a release tarball or somewhere like pypi where +# updates may not be straightforward, prefer to use the git repo as the +# default registry +def get_default_registry(): + internal_registry = os.path.normpath(os.path.dirname(__file__) + "/../default-registry") + git_registry = "git://git.openembedded.org/bitbake;protocol=https;branch=master;rev=master" + if os.path.exists(os.path.dirname(__file__) + "/../.git"): + return internal_registry + else: + return git_registry + +def cache_dir(top_dir): + return os.path.join(top_dir, '.bitbake-setup-cache') + +def init_bb_cache(top_dir, settings, args): + dldir = settings["default"]["dl-dir"] + bb_cachedir = os.path.join(cache_dir(top_dir), 'bitbake-cache') + os.makedirs(bb_cachedir, exist_ok=True) + + d = bb.data.init() + d.setVar("DL_DIR", dldir) + d.setVar("BB_CACHEDIR", bb_cachedir) + d.setVar("__BBSRCREV_SEEN", "1") + if args.no_network: + d.setVar("BB_SRCREV_POLICY", "cache") + bb.fetch.fetcher_init(d) + return d + +def save_bb_cache(): + bb.fetch2.fetcher_parse_save() + bb.fetch2.fetcher_parse_done() + +def get_config_name(config): + suffix = '.conf.json' + config_file = os.path.basename(config) + if config_file.endswith(suffix): + return config_file[:-len(suffix)] + else: + raise Exception("Config file {} does not end with {}, please rename the file.".format(config, suffix)) + +def write_upstream_config(config_dir, config_data): + with open(os.path.join(config_dir, "config-upstream.json"),'w') as s: + json.dump(config_data, s, sort_keys=True, indent=4) + +def write_sources_fixed_revisions(config_dir, layer_dir, config_data): + json_path = os.path.join(config_dir, "sources-fixed-revisions.json") + json_link = os.path.join(layer_dir, "sources-fixed-revisions.json") + sources = {} + sources['sources'] = config_data + with open(os.path.join(config_dir, "sources-fixed-revisions.json"),'w') as s: + json.dump(sources, s, sort_keys=True, indent=4) + if not os.path.lexists(json_link): + os.symlink(os.path.relpath(json_path ,layer_dir), json_link) + +def commit_config(config_dir): + bb.process.run("git -C {} add .".format(config_dir)) + bb.process.run("git -C {} commit --allow-empty --no-verify -a -m 'Configuration at {}'".format(config_dir, time.asctime())) + +def _write_layer_list(dest, repodirs): + layers = [] + for r in repodirs: + for root, dirs, files in os.walk(os.path.join(dest,r)): + if os.path.basename(root) == 'conf' and 'layer.conf' in files: + layers.append(os.path.relpath(os.path.dirname(root), dest)) + layers_f = os.path.join(dest, ".oe-layers.json") + with open(layers_f, 'w') as f: + json.dump({"version":"1.0","layers":layers}, f, sort_keys=True, indent=4) + +def add_unique_timestamp_to_path(path): + timestamp = time.strftime("%Y%m%d%H%M%S") + path_unique = "{}.{}".format(path, timestamp) + if os.path.exists(path_unique): + import itertools + for i in itertools.count(start=1): + path_unique = "{}.{}.{}".format(path, timestamp, i) + if not os.path.exists(path_unique): + break + return path_unique + +def _get_remotes(r_remote): + remotes = [] + + if 'remotes' not in r_remote and 'uri' not in r_remote: + raise Exception("Expected key(s): 'remotes', 'uri'") + + if 'remotes' in r_remote: + for remote in r_remote['remotes']: + remotes.append(r_remote['remotes'][remote]['uri']) + + if 'uri' in r_remote: + remotes.append(r_remote['uri']) + + return remotes + +def checkout_layers(layers, confdir, layerdir, d, rebase_conflicts_strategy='abort'): + def _checkout_git_remote(r_remote, repodir, layers_fixed_revisions): + rev = r_remote['rev'] + branch = r_remote.get('branch', None) + + remotes = _get_remotes(r_remote) + + for remote in remotes: + prot,host,path,user,pswd,params = bb.fetch.decodeurl(remote) + fetchuri = bb.fetch.encodeurl(('git',host,path,user,pswd,params)) + logger.plain(" {}".format(r_name)) + if branch: + src_uri = f"{fetchuri};protocol={prot};rev={rev};branch={branch};destsuffix={repodir}" + else: + src_uri = f"{fetchuri};protocol={prot};rev={rev};nobranch=1;destsuffix={repodir}" + fetcher = bb.fetch.Fetch([src_uri], d) + repodir_path = os.path.join(layerdir, repodir) + try: + do_fetch(fetcher, layerdir) + except (bb.fetch2.LocalModificationsError, bb.fetch2.RebaseError) as e: + if rebase_conflicts_strategy != 'backup': + e.msg += ("\nUse 'bitbake-setup update --rebase-conflicts-strategy=backup'" + " to automatically back up the directory and re-clone from upstream," + " or use 'bitbake-setup init -L %s /path/to/local/checkout'" + " to work with a local checkout instead." % r_name) + raise + backup_path = add_unique_timestamp_to_path(repodir_path + '-backup') + logger.warning( + "%s\n" + "Renaming %s to %s to preserve your work, then re-cloning from upstream.", + e, repodir_path, backup_path) + os.rename(repodir_path, backup_path) + fetcher.unpack(layerdir) + urldata = fetcher.ud[src_uri] + revision = urldata.revision + layers_fixed_revisions[r_name]['git-remote']['rev'] = revision + + def _symlink_local(src, dst): + logger.plain("Making a symbolic link {} pointing to {}".format(dst, src)) + os.symlink(src, dst) + + layers_fixed_revisions = copy.deepcopy(layers) + repodirs = [] + oesetupbuild = None + logger.plain("Fetching layer/tool repositories into {}".format(layerdir)) + for r_name in layers: + r_data = layers[r_name] + repodir = r_data.get("path", r_name) + repodirs.append(repodir) + + r_remote = r_data.get('git-remote') + r_local = r_data.get('local') + if r_remote and r_local: + raise Exception("Source {} contains both git-remote and local properties.".format(r_name)) + + repodir_path = os.path.join(layerdir, repodir) + if os.path.lexists(repodir_path): + if os.path.islink(repodir_path): + os.remove(repodir_path) + elif r_local: + backup_path = add_unique_timestamp_to_path(repodir_path + '-backup') + logger.warning("""Source {} in {} contains local modifications. Renaming to {} to preserve them. +For local development work it is recommended to clone the needed layers separately and re-initialize using -L option: +bitbake-setup init -L {} /path/to/repo/checkout""".format( + r_name, repodir_path, backup_path, r_name)) + os.rename(repodir_path, backup_path) + + if r_remote: + _checkout_git_remote(r_remote, repodir, layers_fixed_revisions) + if r_local: + _symlink_local(os.path.expanduser(r_local["path"]), repodir_path) + + if os.path.exists(os.path.join(layerdir, repodir, 'scripts/oe-setup-build')): + oesetupbuild = os.path.join(layerdir, repodir, 'scripts/oe-setup-build') + oeinitbuildenvdir = os.path.join(layerdir, repodir) + + logger.plain(" ") + _write_layer_list(layerdir, repodirs) + + if oesetupbuild: + links = {'setup-build': oesetupbuild, 'oe-scripts': os.path.dirname(oesetupbuild), 'oe-init-build-env-dir': oeinitbuildenvdir} + for l,t in links.items(): + symlink = os.path.join(layerdir, l) + if os.path.lexists(symlink): + os.remove(symlink) + os.symlink(os.path.relpath(t,layerdir),symlink) + + return layers_fixed_revisions + +def setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, update_bb_conf, init_vscode=False): + def _setup_build_conf(layers, filerelative_layers, build_conf_dir): + os.makedirs(build_conf_dir) + layers_s = [] + + for l in layers: + l = os.path.join(layerdir, l) + layers_s.append(" {} \\".format(l)) + + for l in filerelative_layers: + if thisdir: + l = os.path.join(thisdir, l) + else: + raise Exception("Configuration is using bb-layers-file-relative to specify " \ + "a layer path relative to itself. This can be done only " \ + "when the configuration is specified by its path on local " \ + "disk, not when it's in a registry or is fetched over http.") + layers_s.append(" {} \\".format(l)) + + layers_s = "\n".join(layers_s) + bblayers_conf = """BBLAYERS ?= " \\ +{} + " +""".format(layers_s) + with open(os.path.join(build_conf_dir, "bblayers.conf"), 'w') as f: + f.write(bblayers_conf) + + local_conf = """# +# This file is intended for local configuration tweaks. +# +# If you would like to publish and share changes made to this file, +# it is recommended to put them into a distro config, or to create +# layer fragments from changes made here. +# +""" + with open(os.path.join(build_conf_dir, "local.conf"), 'w') as f: + f.write(local_conf) + + with open(os.path.join(build_conf_dir, "templateconf.cfg"), 'w') as f: + f.write("") + + with open(os.path.join(build_conf_dir, "conf-summary.txt"), 'w') as f: + f.write(bitbake_config["description"] + "\n") + + with open(os.path.join(build_conf_dir, "conf-notes.txt"), 'w') as f: + f.write("") + + def _make_init_build_env(builddir, oeinitbuildenvdir): + builddir = os.path.realpath(builddir) + cmd = 'cd {}\nset {}\n. ./oe-init-build-env\nset --\n'.format(oeinitbuildenvdir, builddir) + initbuild_in_builddir = os.path.join(builddir, 'init-build-env') + + with open(initbuild_in_builddir, 'w') as f: + f.write("# init-build-env wrapper created by bitbake-setup\n") + f.write(cmd + '\n') + + def _prepend_passthrough_to_init_build_env(builddir): + env = bitbake_config.get("bb-env-passthrough-additions") + if not env: + return + + initbuild_in_builddir = os.path.join(builddir, 'init-build-env') + with open(initbuild_in_builddir) as f: + content = f.read() + + joined = " \\\n".join(env) + env = "export BB_ENV_PASSTHROUGH_ADDITIONS=\" \\\n" + env += "${BB_ENV_PASSTHROUGH_ADDITIONS} \\\n" + env += joined + env += '"' + + with open(initbuild_in_builddir, 'w') as f: + f.write("# environment passthrough added by bitbake-setup\n") + f.write(env + '\n') + f.write('\n') + f.write(content) + + bitbake_builddir = os.path.join(setupdir, "build") + logger.plain("Setting up bitbake configuration in\n {}\n".format(bitbake_builddir)) + + template = bitbake_config.get("oe-template") + layers = bitbake_config.get("bb-layers") + if not template and not layers: + logger.error("Bitbake configuration does not contain a reference to an OpenEmbedded build template via 'oe-template' or a list of layers via 'bb-layers'; please use oe-setup-build, oe-init-build-env or another mechanism manually to complete the setup.") + return + oesetupbuild = os.path.join(layerdir, 'setup-build') + if template and not os.path.exists(oesetupbuild): + raise Exception("Cannot complete setting up a bitbake build directory from OpenEmbedded template '{}' as oe-setup-build was not found in any layers; please use oe-init-build-env manually.".format(template)) + + bitbake_confdir = os.path.join(bitbake_builddir, 'conf') + backup_bitbake_confdir = add_unique_timestamp_to_path(os.path.join(bitbake_builddir, 'conf-backup')) + upstream_bitbake_confdir = add_unique_timestamp_to_path(os.path.join(bitbake_builddir, 'conf-upstream')) + + if os.path.exists(bitbake_confdir): + os.rename(bitbake_confdir, backup_bitbake_confdir) + + if layers: + filerelative_layers = bitbake_config.get("bb-layers-file-relative") or [] + _setup_build_conf(layers, filerelative_layers, bitbake_confdir) + + if template: + bb.process.run("{} setup -c {} -b {} --no-shell".format(oesetupbuild, template, bitbake_builddir)) + else: + oeinitbuildenvdir = os.path.join(layerdir, 'oe-init-build-env-dir') + if not os.path.exists(os.path.join(oeinitbuildenvdir, "oe-init-build-env")): + logger.error("Could not find oe-init-build-env in any of the layers; please use another mechanism to initialize the bitbake environment") + return + _make_init_build_env(bitbake_builddir, os.path.realpath(oeinitbuildenvdir)) + + _prepend_passthrough_to_init_build_env(bitbake_builddir) + + siteconf_symlink = os.path.join(bitbake_confdir, "site.conf") + siteconf = os.path.normpath(os.path.join(setupdir, '..', "site.conf")) + if os.path.lexists(siteconf_symlink): + os.remove(siteconf_symlink) + os.symlink(os.path.relpath(siteconf, bitbake_confdir), siteconf_symlink) + + + init_script = os.path.join(bitbake_builddir, "init-build-env") + workspace_file = os.path.join(setupdir, "bitbake.code-workspace") + shell = "bash" + fragments = bitbake_config.get("oe-fragments", []) + sorted(bitbake_config.get("oe-fragment-choices",{}).values()) + if fragments: + bb.process.run("{} -c '. {} && bitbake-config-build enable-fragment {}'".format(shell, init_script, " ".join(fragments))) + + if os.path.exists(backup_bitbake_confdir): + conf_diff = get_diff(backup_bitbake_confdir, bitbake_confdir) + if not conf_diff: + logger.plain('New bitbake configuration from upstream is the same as the current one, no need to update it.') + shutil.rmtree(bitbake_confdir) + os.rename(backup_bitbake_confdir, bitbake_confdir) + if init_vscode or os.path.exists(workspace_file): + configure_vscode(setupdir, layerdir, bitbake_builddir, init_script) + return + + logger.plain('Upstream bitbake configuration changes were found:') + logger.plain(conf_diff) + + if update_bb_conf == 'prompt': + y_or_n = input('Apply these changes to the current configuration? (y/N): ') + if y_or_n != 'y': + update_bb_conf = 'no' + + if update_bb_conf == 'no': + logger.plain('Ignoring upstream bitbake configuration changes') + logger.plain(f'Leaving the upstream configuration in {upstream_bitbake_confdir}') + os.rename(bitbake_confdir, upstream_bitbake_confdir) + os.rename(backup_bitbake_confdir, bitbake_confdir) + if init_vscode or os.path.exists(workspace_file): + configure_vscode(setupdir, layerdir, bitbake_builddir, init_script) + return + + logger.plain('Applying upstream bitbake configuration changes') + logger.plain(f'Leaving the previous configuration in {backup_bitbake_confdir}') + + fragment_note = "Run 'bitbake-config-build enable-fragment ' to enable additional fragments or replace built-in ones (e.g. machine/ or distro/ to change MACHINE or DISTRO)." + + readme_extra = "" + if init_vscode: + readme_extra = "\nTo edit the code in VSCode, open the workspace: code {}\n".format(workspace_file) + + readme = """{}\n\nAdditional information is in {} and {}\n +Source the environment using '. {}' to run builds from the command line.\n +{}\n +The bitbake configuration files (local.conf, bblayers.conf and more) can be found in {}/conf +{}""".format( + bitbake_config["description"], + os.path.join(bitbake_builddir,'conf/conf-summary.txt'), + os.path.join(bitbake_builddir,'conf/conf-notes.txt'), + init_script, + fragment_note, + bitbake_builddir, + readme_extra + ) + readme_file = os.path.join(bitbake_builddir, "README") + with open(readme_file, 'w') as f: + f.write(readme) + + logger.plain("This bitbake configuration provides:\n {}\n".format(bitbake_config["description"])) + logger.plain("Usage instructions and additional information are in\n {}\n".format(readme_file)) + logger.plain("To run builds, source the environment using\n . {}\n".format(init_script)) + logger.plain("{}\n".format(fragment_note)) + logger.plain("The bitbake configuration files (local.conf, bblayers.conf and more) can be found in\n {}/conf\n".format(bitbake_builddir)) + if init_vscode: + logger.plain("To edit the code in VSCode, open the workspace:\n code {}\n".format(workspace_file)) + + if init_vscode or os.path.exists(workspace_file): + configure_vscode(setupdir, layerdir, bitbake_builddir, init_script) + +def get_registry_config(registry_path, id): + for root, dirs, files in os.walk(registry_path): + for f in files: + if f.endswith('.conf.json') and id == get_config_name(f): + return os.path.join(root, f) + raise Exception("Unable to find {} in available configurations; use 'list' sub-command to see what is available".format(id)) + +def merge_overrides_into_sources(sources, overrides): + layers = copy.deepcopy(sources) + for k,v in overrides.items(): + if k in layers: + layers[k] = v + return layers + +def update_build(config, confdir, setupdir, layerdir, d, update_bb_conf="prompt", init_vscode=False, rebase_conflicts_strategy='abort'): + layer_config = merge_overrides_into_sources(config["data"]["sources"], config["source-overrides"]["sources"]) + sources_fixed_revisions = checkout_layers(layer_config, confdir, layerdir, d, rebase_conflicts_strategy=rebase_conflicts_strategy) + bitbake_config = config["bitbake-config"] + thisdir = os.path.dirname(config["path"]) if config["type"] == 'local' else None + setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, update_bb_conf, init_vscode) + write_sources_fixed_revisions(confdir, layerdir, sources_fixed_revisions) + commit_config(confdir) + +def int_input(allowed_values, prompt=''): + n = None + while n is None: + try: + n = int(input(prompt)) + except ValueError: + prompt = 'Not a valid number, please try again: ' + continue + if n not in allowed_values: + prompt = 'Number {} not one of {}, please try again: '.format(n, allowed_values) + n = None + return n + +def flatten_bitbake_configs(configs): + def merge_configs(c1,c2): + c_merged = {} + for k,v in c2.items(): + if k not in c1.keys(): + c_merged[k] = v + for k,v in c1.items(): + if k not in c2.keys(): + c_merged[k] = v + else: + c_merged[k] = c1[k] + c2[k] + del c_merged['configurations'] + return c_merged + + flattened_configs = [] + for c in configs: + if 'configurations' not in c: + flattened_configs.append(c) + else: + for sub_c in flatten_bitbake_configs(c['configurations']): + flattened_configs.append(merge_configs(c, sub_c)) + return flattened_configs + +def choose_bitbake_config(configs, parameters, non_interactive): + flattened_configs = flatten_bitbake_configs(configs) + configs_dict = {i["name"]:i for i in flattened_configs} + + if parameters: + config_id = parameters[0] + if config_id not in configs_dict: + raise Exception("Bitbake configuration {} not found; replace with one of {}".format(config_id, configs_dict)) + return configs_dict[config_id] + + enumerated_configs = list(enumerate(flattened_configs, 1)) + if len(enumerated_configs) == 1: + only_config = flattened_configs[0] + logger.plain("\nSelecting the only available bitbake configuration {}".format(only_config["name"])) + return only_config + + if non_interactive: + raise Exception("Unable to choose from bitbake configurations in non-interactive mode: {}".format(configs_dict)) + + logger.plain("") + print_configs("Available bitbake configurations", + [c["name"] for c in flattened_configs], + [c["description"] for c in flattened_configs]) + config_n = int_input([i[0] for i in enumerated_configs], + "\nPlease select one of the above bitbake configurations by its number: ") - 1 + return flattened_configs[config_n] + +def choose_config(configs, non_interactive): + not_expired_configs = [k for k in sorted(configs.keys()) if not has_expired(configs[k].get("expires", None))] + if len(not_expired_configs) == 1: + only_config = not_expired_configs[0] + logger.plain("\nSelecting the only available configuration {}\n".format(only_config)) + return only_config + + if non_interactive: + raise Exception("Unable to choose from configurations in non-interactive mode: {}".format(not_expired_configs)) + + descs = [] + for c in not_expired_configs: + d = configs[c]["description"] + expiry_date = configs[c].get("expires", None) + if expiry_date: + d += f" (supported until {expiry_date})" + descs.append(d) + + logger.plain("") + print_configs("Available Configuration Templates", + [c for c in not_expired_configs], + descs) + config_n = int_input([i[0] for i in list(enumerate(not_expired_configs, 1))], + "\nPlease select one of the above configurations by its number: ") - 1 + return not_expired_configs[config_n] + +def choose_fragments(possibilities, parameters, non_interactive, skip_selection): + choices = {} + for k,v in possibilities.items(): + if skip_selection and k in skip_selection: + logger.info("Skipping a selection of {}, as requested on command line. The resulting bitbake configuration may require further manual adjustments.".format(k)) + continue + + # options can be a list of strings or a list of dicts + options = v["options"] + if len(options) > 0 and isinstance(v["options"][0], str): + options = [{"name": o, "description": ""} for o in v["options"]] + + choice = [o["name"] for o in options if o["name"] in parameters] + if len(choice) > 1: + raise Exception("Options specified on command line do not allow a single selection " + f"from possibilities {[o['name'] for o in options]}, please " + f"remove one or more from {parameters}") + if len(choice) == 1: + choices[k] = choice[0] + continue + + if non_interactive: + raise Exception(f"Unable to choose from options in non-interactive mode: {[o['name'] for o in options]}") + + logger.plain("") + print_configs(v["description"], + [o['name'] for o in options], + [o['description'] for o in options]) + options_enumerated = list(enumerate(options, 1)) + option_n = int_input([i[0] for i in options_enumerated], + "\nPlease select one of the above options by its number: ") - 1 + choices[k] = options_enumerated[option_n][1]["name"] + return choices + +def obtain_config(top_dir, registry, args, source_overrides, d): + if args.config: + config_id = args.config[0] + config_parameters = args.config[1:] + if os.path.exists(config_id): + config_id = os.path.abspath(config_id) + logger.info("Reading configuration from local file\n {}".format(config_id)) + upstream_config = {'type':'local', + 'path':config_id, + 'name':get_config_name(config_id), + 'data':json.load(open(config_id)) + } + elif config_id.startswith("http://") or config_id.startswith("https://"): + logger.info("Reading configuration from network URI\n {}".format(config_id)) + import urllib.request + try: + with urllib.request.urlopen(config_id) as f: + json_data = json.load(f) + upstream_config = {'type':'network','uri':config_id,'name':get_config_name(config_id),'data':json_data} + except json.JSONDecodeError as e: + raise Exception ("Invalid JSON from {}. Are you pointing to an HTML page? {}".format(config_id, e)) from e + else: + logger.info("Looking up config {} in configuration registry".format(config_id)) + registry_path = update_registry(registry, cache_dir(top_dir), d) + registry_configs = list_registry(registry_path, with_expired=True) + if config_id not in registry_configs: + raise Exception("Config {} not found in configuration registry, re-run 'init' without parameters to choose from available configurations.".format(config_id)) + upstream_config = {'type':'registry','registry':registry,'name':config_id,'data':json.load(open(get_registry_config(registry_path,config_id)))} + expiry_date = upstream_config['data'].get("expires", None) + if has_expired(expiry_date): + logger.warning("This configuration is no longer supported after {}. Please consider changing to a supported configuration.".format(expiry_date)) + else: + registry_path = update_registry(registry, cache_dir(top_dir), d) + registry_configs = list_registry(registry_path, with_expired=True) + config_id = choose_config(registry_configs, args.non_interactive) + config_parameters = [] + upstream_config = {'type':'registry','registry':registry,'name':config_id,'data':json.load(open(get_registry_config(registry_path,config_id)))} + + upstream_config['bitbake-config'] = choose_bitbake_config(upstream_config['data']['bitbake-setup']['configurations'], config_parameters, args.non_interactive) + upstream_config['bitbake-config']['oe-fragment-choices'] = choose_fragments(upstream_config['bitbake-config'].get('oe-fragments-one-of',{}), config_parameters[1:], args.non_interactive, args.skip_selection) + upstream_config['non-interactive-cmdline-options'] = [config_id, upstream_config['bitbake-config']['name']] + sorted(upstream_config['bitbake-config']['oe-fragment-choices'].values()) + upstream_config['source-overrides'] = source_overrides + upstream_config['skip-selection'] = args.skip_selection + return upstream_config + +def obtain_overrides(args): + overrides = {'sources':{}} + if args.source_overrides: + overrides = json.load(open(args.source_overrides)) + overrides_dir = os.path.dirname(os.path.abspath(args.source_overrides)) + for s,v in overrides['sources'].items(): + local = v.get('local') + if local: + path = os.path.expanduser(local['path']) + if not os.path.isabs(path): + overrides['sources'][s]['local']['path'] = os.path.join(overrides_dir, path) + + for local_name, local_path in args.use_local_source: + overrides['sources'][local_name] = {'local':{'path':os.path.abspath(os.path.expanduser(local_path))}} + + return overrides + +def configure_vscode(setupdir, layerdir, builddir, init_script): + """ + Configure the VSCode environment by creating or updating a workspace file. + + Create or update a bitbake.code-workspace file with folders for the layers and build/conf. + Managed folders are regenerated; user-added folders are kept. Settings are merged, with + managed keys (bitbake.*, python extra paths) always overwritten. + """ + logger.debug("configure_vscode: setupdir={}, layerdir={}, builddir={}, init_script={}".format( + setupdir, layerdir, builddir, init_script)) + + # Get git repository directories + git_repos = [] + if os.path.exists(layerdir): + for entry in os.listdir(layerdir): + entry_path = os.path.join(layerdir, entry) + if os.path.isdir(entry_path) and not os.path.islink(entry_path): + # Check if it's a git repository + if os.path.exists(os.path.join(entry_path, '.git')): + git_repos.append(entry) + logger.debug("configure_vscode: found {} git repos: {}".format(len(git_repos), git_repos)) + + conf_path = os.path.relpath(os.path.join(builddir, "conf"), setupdir) + repo_paths = [os.path.relpath(os.path.join(layerdir, repo), setupdir) for repo in git_repos] + logger.debug("configure_vscode: conf_path={}, repo_paths={}".format(conf_path, repo_paths)) + + # Load existing workspace + workspace_file = os.path.join(setupdir, "bitbake.code-workspace") + workspace = { + "extensions": { + "recommendations": [ + "yocto-project.yocto-bitbake" + ] + } + } + if os.path.exists(workspace_file): + logger.debug("configure_vscode: loading existing workspace file: {}".format(workspace_file)) + try: + with open(workspace_file, 'r') as f: + workspace = json.load(f) + logger.debug("configure_vscode: loaded workspace with {} folders, {} settings".format( + len(workspace.get("folders", [])), len(workspace.get("settings", {})))) + except (json.JSONDecodeError, OSError) as e: + logger.error( + "Unable to read existing workspace file {}: {}. Skipping update.".format( + workspace_file, str(e) + ) + ) + return + else: + logger.debug("configure_vscode: creating new workspace file: {}".format(workspace_file)) + + # Update folders + existing_folders = workspace.get("folders", []) + new_folders = [{"name": "conf", "path": conf_path}] + for rp in repo_paths: + repo_name = os.path.basename(rp) + new_folders.append({"name": repo_name, "path": rp}) + # Keep any user-added folders that are not managed + managed_paths = {f["path"] for f in new_folders} + for f in existing_folders: + if f["path"] not in managed_paths: + new_folders.append(f) + logger.debug("configure_vscode: keeping user-added folder: {}".format(f["path"])) + workspace["folders"] = new_folders + logger.debug("configure_vscode: updated workspace with {} folders".format(len(new_folders))) + + # Build Python extra paths for each layer - only check top level of each repo + extra_paths = [] + subdirs_to_check = ['lib', 'scripts'] + for repo in git_repos: + repo_path_abs = os.path.join(layerdir, repo) + for subdir in subdirs_to_check: + sub_path = os.path.join(repo_path_abs, subdir) + if os.path.isdir(sub_path): + extra_paths.append(sub_path) + + # Update settings + existing_settings = workspace.get("settings", {}) + new_settings = { + "bitbake.disableConfigModification": True, + "bitbake.pathToBitbakeFolder": os.path.join(layerdir, "bitbake"), + "bitbake.pathToBuildFolder": builddir, + "bitbake.pathToEnvScript": init_script, + "bitbake.workingDirectory": builddir, + "files.associations": { + "*.conf": "bitbake", + "*.inc": "bitbake" + }, + "files.exclude": { + "**/.git/**": True + }, + "search.exclude": { + "**/.git/**": True, + "**/logs/**": True + }, + "files.watcherExclude": { + "**/.git/**": True, + "**/logs/**": True + }, + "python.analysis.exclude": [ + "**/.git/**", + "**/logs/**" + ], + "python.autoComplete.extraPaths": extra_paths, + "python.analysis.extraPaths": extra_paths + } + + # Merge settings: add missing, always update bitbake paths and python extra paths + for key, value in new_settings.items(): + if key not in existing_settings: + existing_settings[key] = value + elif key.startswith("bitbake.") or key in [ + "python.autoComplete.extraPaths", + "python.analysis.extraPaths", + ]: + # Always replace - these are managed/machine-generated settings + existing_settings[key] = value + elif key in [ + "files.associations", + "files.exclude", + "search.exclude", + "files.watcherExclude", + "python.analysis.exclude", + ]: + # For dicts and lists, merge new values in without removing user additions + if isinstance(value, dict): + if not isinstance(existing_settings[key], dict): + existing_settings[key] = {} + for k, v in value.items(): + if k not in existing_settings[key]: + existing_settings[key][k] = v + elif isinstance(value, list): + if not isinstance(existing_settings[key], list): + existing_settings[key] = [] + for item in value: + if item not in existing_settings[key]: + existing_settings[key].append(item) + + workspace["settings"] = existing_settings + logger.debug("configure_vscode: merged settings, total {} keys".format(len(existing_settings))) + + with open(workspace_file, 'w') as f: + json.dump(workspace, f, indent=4) + logger.debug("configure_vscode: wrote workspace file: {}".format(workspace_file)) + +def init_config(top_dir, settings, args): + create_siteconf(top_dir, args.non_interactive, settings) + + d = init_bb_cache(top_dir, settings, args) + + def handle_task_progress(event, d): + rate = event.rate if event.rate else '' + progress = event.progress if event.progress > 0 else 0 + logger.handlers[0].terminator = '\r' + logger.plain("{}% {} ".format(progress, rate)) + logger.handlers[0].terminator = '\n' + + source_overrides = obtain_overrides(args) + upstream_config = obtain_config(top_dir, settings["default"]["registry"], args, source_overrides, d) + logger.info("Run 'bitbake-setup init --non-interactive {}' to select this configuration non-interactively.\n".format(" ".join(upstream_config['non-interactive-cmdline-options']))) + + if args.setup_dir_name: + setup_dir_name = args.setup_dir_name + else: + setup_dir_name = "{}-{}".format(upstream_config['name']," ".join(upstream_config['non-interactive-cmdline-options'][1:]).replace(" ","-").replace("/","_")) + if 'setup-dir-name' in upstream_config['bitbake-config']: + mapping = { + k: v.partition("/")[2].replace(" ", "-").replace("/", "_") + for k, v in upstream_config['bitbake-config']['oe-fragment-choices'].items() + } + config_setup_dir_name = string.Template(upstream_config['bitbake-config']['setup-dir-name']).substitute(mapping) + config_setup_dir = os.path.join(top_dir, config_setup_dir_name) + if os.path.exists(config_setup_dir): + logger.info("Setup directory {} (as suggested by configuration) already exists, using the full name instead.\n".format(config_setup_dir)) + elif settings['default']['use-full-setup-dir-name'] != 'no': + logger.info("Using the full setup directory name instead of {} suggested by configuration, as set in the settings.\n".format(config_setup_dir)) + else: + setup_dir_name = config_setup_dir_name + + if not args.non_interactive: + n = input(f"Enter setup directory name [{setup_dir_name}]: ") + if n: + setup_dir_name = n + + setupdir = os.path.join(os.path.abspath(top_dir), setup_dir_name) + if os.path.exists(os.path.join(setupdir, "layers")): + logger.info(f"Setup already initialized in:\n {setupdir}\nUse 'bitbake-setup status' to check if it needs to be updated, or 'bitbake-setup update' to perform the update.\nIf you would like to start over and re-initialize in this directory, remove it, and run 'bitbake-setup init' again.") + return + + logger.plain("Initializing a setup directory in\n {}".format(setupdir)) + if not args.non_interactive: + y_or_n = input('Continue? (y/N): ') + if y_or_n != 'y': + exit() + logger.plain("") + + os.makedirs(setupdir, exist_ok=True) + + confdir = os.path.join(setupdir, "config") + layerdir = os.path.join(setupdir, "layers") + + os.makedirs(confdir) + os.makedirs(layerdir) + + bb.process.run("git -C {} init -b main".format(confdir)) + # Make sure commiting doesn't fail if no default git user is configured on the machine + bb.process.run("git -C {} config user.name bitbake-setup".format(confdir)) + bb.process.run("git -C {} config user.email bitbake-setup@not.set".format(confdir)) + bb.process.run("git -C {} commit --no-verify --allow-empty -m 'Initial commit'".format(confdir)) + + bb.event.register("bb.build.TaskProgress", handle_task_progress, data=d) + + write_upstream_config(confdir, upstream_config) + update_build(upstream_config, confdir, setupdir, layerdir, d, update_bb_conf="yes", init_vscode=args.init_vscode) + + bb.event.remove("bb.build.TaskProgress", None) + +def get_diff(file1, file2): + try: + bb.process.run('diff {} -uNr {} {}'.format(get_diff_color_param(), file1, file2)) + except bb.process.ExecutionError as e: + if e.exitcode == 1: + return e.stdout + else: + raise e + return None + +def are_layers_changed(layers, layerdir, d): + def _is_git_remote_changed(r_remote, repodir): + from bb.fetch2.git import sha1_re + + rev = r_remote['rev'] + branch = r_remote.get('branch', None) + + rev_parse_result = bb.process.run('git -C {} rev-parse HEAD'.format(os.path.join(layerdir, repodir))) + local_revision = rev_parse_result[0].strip() + if sha1_re.match(rev): + if rev != local_revision: + logger.info('Layer repository checked out into {} is at revision {} but should be at {}'.format(os.path.join(layerdir, repodir),local_revision, rev)) + return True + return False + + remotes = _get_remotes(r_remote) + changed = False + for remote in remotes: + type,host,path,user,pswd,params = bb.fetch.decodeurl(remote) + fetchuri = bb.fetch.encodeurl(('git',host,path,user,pswd,params)) + if branch: + fetcher = bb.fetch.FetchData("{};protocol={};rev={};branch={};destsuffix={}".format(fetchuri,type,rev,branch,repodir), d) + else: + fetcher = bb.fetch.FetchData("{};protocol={};rev={};nobranch=1;destsuffix={}".format(fetchuri,type,rev,repodir), d) + upstream_revision = fetcher.method.latest_revision(fetcher, d, 'default') + if upstream_revision != local_revision: + changed = True + logger.info('Layer repository {} checked out into {} updated revision {} from {} to {}'.format(remote, os.path.join(layerdir, repodir), rev, local_revision, upstream_revision)) + return changed + + changed = False + for r_name in layers: + r_data = layers[r_name] + repodir = r_data.get("path", r_name) + + git_remote = r_data.get('git-remote') + if git_remote: + changed = changed | _is_git_remote_changed(git_remote, repodir) + + return changed + +def build_status(top_dir, settings, args, d, update=False): + setupdir = os.path.abspath(args.setup_dir) + + confdir = os.path.join(setupdir, "config") + layerdir = os.path.join(setupdir, "layers") + + current_upstream_config = json.load(open(os.path.join(confdir, "config-upstream.json"))) + + args.config = current_upstream_config['non-interactive-cmdline-options'] + args.non_interactive = True + args.skip_selection = current_upstream_config['skip-selection'] + source_overrides = current_upstream_config["source-overrides"] + registry = current_upstream_config.get("registry") + new_upstream_config = obtain_config(top_dir, registry, args, source_overrides, d) + + write_upstream_config(confdir, new_upstream_config) + config_diff = bb.process.run('git -C {} diff {}'.format(confdir, get_diff_color_param()))[0] + + if config_diff: + logger.plain('\nConfiguration in {} has changed:\n{}'.format(setupdir, config_diff)) + if update: + update_build(new_upstream_config, confdir, setupdir, layerdir, d, + update_bb_conf=args.update_bb_conf, rebase_conflicts_strategy=args.rebase_conflicts_strategy) + else: + bb.process.run('git -C {} restore config-upstream.json'.format(confdir)) + return + + layer_config = merge_overrides_into_sources(current_upstream_config["data"]["sources"], current_upstream_config["source-overrides"]["sources"]) + if are_layers_changed(layer_config, layerdir, d): + if update: + update_build(current_upstream_config, confdir, setupdir, layerdir, + d, update_bb_conf=args.update_bb_conf, rebase_conflicts_strategy=args.rebase_conflicts_strategy) + return + + logger.plain("\nConfiguration in {} has not changed.".format(setupdir)) + +def build_update(top_dir, settings, args, d): + build_status(top_dir, settings, args, d, update=True) + +def do_fetch(fetcher, dir): + # git fetcher simply dumps git output to stdout; in bitbake context that is redirected to temp/log.do_fetch + # and we need to set up smth similar here + fetchlogdir = os.path.join(dir, 'logs') + os.makedirs(fetchlogdir, exist_ok=True) + fetchlog = os.path.join(fetchlogdir, 'fetch_log.{}'.format(datetime.datetime.now().strftime("%Y%m%d%H%M%S"))) + with open(fetchlog, 'a') as f: + oldstdout = sys.stdout + sys.stdout = f + try: + fetcher.download() + fetcher.unpack_update(dir) + finally: + sys.stdout = oldstdout + +def update_registry(registry, cachedir, d): + registrydir = 'configurations' + if registry.startswith("."): + full_registrydir = os.path.join(os.getcwd(), registry, registrydir) + elif registry.startswith("/"): + full_registrydir = os.path.join(registry, registrydir) + else: + full_registrydir = os.path.join(cachedir, registrydir) + logger.info("Fetching configuration registry\n {}\ninto\n {}".format(registry, full_registrydir)) + fetcher = bb.fetch.Fetch(["{};destsuffix={}".format(registry, registrydir)], d) + do_fetch(fetcher, cachedir) + return full_registrydir + +def has_expired(expiry_date): + if expiry_date: + return datetime.datetime.now() > datetime.datetime.fromisoformat(expiry_date) + return False + +def list_registry(registry_path, with_expired): + json_data = {} + + for root, dirs, files in os.walk(registry_path): + for f in files: + if f.endswith('.conf.json'): + config_name = get_config_name(f) + config_data = json.load(open(os.path.join(root, f))) + config_desc = config_data["description"] + expiry_date = config_data.get("expires", None) + if expiry_date: + if with_expired or not has_expired(expiry_date): + json_data[config_name] = {"description": config_desc, "expires": expiry_date} + else: + json_data[config_name] = {"description": config_desc} + return json_data + +def list_configs(settings, args): + import tempfile + top_dir = tempfile.mkdtemp(prefix="bitbake-setup-list-") + settings['default']['dl-dir'] = os.path.join(top_dir, '.bitbake-setup-downloads') + d = init_bb_cache(top_dir, settings, args) + registry_path = update_registry(settings["default"]["registry"], cache_dir(top_dir), d) + json_data = list_registry(registry_path, args.with_expired) + shutil.rmtree(top_dir) + + logger.plain("Available configurations:") + for config_name, config_data in sorted(json_data.items()): + expiry_date = config_data.get("expires", None) + config_desc = config_data["description"] + if expiry_date: + if args.with_expired or not has_expired(expiry_date): + logger.plain("{}\t{} (supported until {})".format(config_name, config_desc, expiry_date)) + else: + logger.plain("{}\t{}".format(config_name, config_desc)) + logger.plain("\nRun 'init' with one of the above configuration identifiers to set up a build.") + + if args.write_json: + with open(args.write_json, 'w') as f: + json.dump(json_data, f, sort_keys=True, indent=4) + logger.plain("Available configurations written into {}".format(args.write_json)) + +def install_buildtools(top_dir, settings, args, d): + buildtools_install_dir = os.path.join(args.setup_dir, 'buildtools') + if os.path.exists(buildtools_install_dir): + if not args.force: + logger.plain("Buildtools are already installed in {}.".format(buildtools_install_dir)) + env_scripts = glob.glob(os.path.join(buildtools_install_dir, 'environment-setup-*')) + if env_scripts: + logger.plain("If you wish to use them, you need to source the environment setup script e.g.") + for s in env_scripts: + logger.plain("$ . {}".format(s)) + logger.plain("You can also re-run bitbake-setup install-buildtools with --force option to force a reinstallation.") + return + shutil.rmtree(buildtools_install_dir) + + install_buildtools = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools') + buildtools_download_dir = add_unique_timestamp_to_path(os.path.join(args.setup_dir, 'buildtools-downloads/buildtools')) + logger.plain("Buildtools archive is downloaded into {} and its content installed into {}".format(buildtools_download_dir, buildtools_install_dir)) + subprocess.check_call("{} -d {} --downloads-directory {}".format(install_buildtools, buildtools_install_dir, buildtools_download_dir), shell=True) + +def create_siteconf(top_dir, non_interactive, settings): + siteconfpath = os.path.join(top_dir, 'site.conf') + if os.path.exists(siteconfpath): + logger.info('A site.conf file already exists. Please remove it if you would like to replace it with a default one') + else: + logger.plain(f'{top_dir} looks like a new top directory. If you would like to use a different directory, answer "n" below and either:') + logger.plain('\t1) Change the default bitbake-setup settings:') + logger.plain('\t\tbitbake-setup settings set default top-dir-prefix ') + logger.plain('\t\tbitbake-setup settings set default top-dir-name ') + logger.plain('\t2) Pass one or more options on the command line to change the top level directory in that invocation only:') + logger.plain('\t\tbitbake-setup --setting default top-dir-prefix ...') + logger.plain('\t\tbitbake-setup --setting default top-dir-name ...') + logger.plain('') + + logger.plain('A common site.conf file will be created, please check it is correct before running builds\n {}\n'.format(siteconfpath)) + if not non_interactive: + y_or_n = input('Proceed? (y/N): ') + if y_or_n != 'y': + exit() + + os.makedirs(top_dir, exist_ok=True) + with open(siteconfpath, 'w') as siteconffile: + sstate_settings = textwrap.dedent( + """ + # + # Where to place shared-state files + # + # BitBake has the capability to accelerate builds based on previously built output. + # This is done using "shared state" files which can be thought of as cache objects + # and this option determines where those files are placed. + # + # You can wipe out TMPDIR leaving this directory intact and the build would regenerate + # from these files if no changes were made to the configuration. If changes were made + # to the configuration, only shared state files where the state was still valid would + # be used (done using checksums). + SSTATE_DIR ?= "{sstate_dir}" + # + # Hash Equivalence database location + # + # Hash equivalence improves reuse of sstate by detecting when a given sstate + # artifact can be reused as equivalent, even if the current task hash doesn't + # match the one that generated the artifact. This variable controls where the + # Hash Equivalence database ("hashserv.db") is stored and can be shared between + # concurrent builds. + BB_HASHSERVE_DB_DIR ?= "${{SSTATE_DIR}}" + """.format(sstate_dir=os.path.join(top_dir, ".sstate-cache")) + ) + siteconffile.write( + textwrap.dedent( + """\ + # This file is intended for build host-specific bitbake settings + + # Where to place downloads + # + # During a first build the system will download many different source code + # tarballs from various upstream projects. This can take a while, particularly + # if your network connection is slow. These are all stored in DL_DIR. When + # wiping and rebuilding you can preserve this directory to speed up this part of + # subsequent builds. This directory is safe to share between multiple builds on + # the same machine too. + DL_DIR ?= "{dl_dir}" + """.format( + dl_dir=settings["default"]["dl-dir"], + ) + ) + (sstate_settings if settings["default"]["common-sstate"] == 'yes' else "") + ) + + +def topdir_settings_path(top_dir): + return os.path.join(top_dir, 'settings.conf') + +def global_settings_path(args): + return os.path.abspath(args.global_settings) if args.global_settings else os.path.join(os.path.expanduser('~'), '.config', 'bitbake-setup', 'settings.conf') + +def load_settings(settings_path): + settings = configparser.ConfigParser() + if os.path.exists(settings_path): + logger.info('Loading settings from {}'.format(settings_path)) + settings.read_file(open(settings_path)) + + for section in settings.sections(): + for key, value in settings[section].items(): + settings[section][key] = value.strip('\"\'') + + return settings + +def change_setting(top_dir, args): + if vars(args)['global']: + settings_path = global_settings_path(args) + elif args.setting in GLOBAL_ONLY_SETTINGS: + logger.info(f"{args.setting} can only be set in the global config; '--global' is implied") + settings_path = global_settings_path(args) + else: + settings_path = topdir_settings_path(top_dir) + settings = load_settings(settings_path) + + if args.subcommand == 'set': + if args.section not in settings.keys(): + settings[args.section] = {} + settings[args.section][args.setting] = args.value + logger.plain(f"From section '{args.section}' the setting '{args.setting}' was changed to '{args.value}'") + if args.subcommand == 'unset': + if args.section in settings.keys() and args.setting in settings[args.section].keys(): + del settings[args.section][args.setting] + logger.plain(f"From section '{args.section}' the setting '{args.setting}' has been removed") + + os.makedirs(os.path.dirname(settings_path), exist_ok=True) + with open(settings_path, 'w') as settingsfile: + settings.write(settingsfile) + logger.info(f"Settings written to {settings_path}") + +def list_settings(all_settings): + for section, section_settings in all_settings.items(): + for key, value in section_settings.items(): + logger.plain("{} {} {}".format(section, key, value)) + +def settings_func(top_dir, all_settings, args): + if args.subcommand == 'list': + list_settings(all_settings) + elif args.subcommand == 'set' or args.subcommand == 'unset': + change_setting(top_dir, args) + +def get_setup_dir_via_bbpath(): + bbpath = os.environ.get('BBPATH') + if bbpath: + bitbake_dir = os.path.normpath(bbpath.split(':')[0]) + if os.path.exists(os.path.join(bitbake_dir,'init-build-env')): + setup_dir = os.path.dirname(bitbake_dir) + return setup_dir + return None + +def get_top_dir(args, settings): + setup_dir_via_bbpath = get_setup_dir_via_bbpath() + if setup_dir_via_bbpath: + top_dir = os.path.dirname(setup_dir_via_bbpath) + if os.path.exists(cache_dir(top_dir)): + return top_dir + + if hasattr(args, 'setup_dir'): + top_dir = os.path.dirname(os.path.normpath(args.setup_dir)) + return top_dir + + top_dir_prefix = settings['default']['top-dir-prefix'] + top_dir_name = settings['default']['top-dir-name'] + return os.path.join(top_dir_prefix, top_dir_name) + +def merge_settings(builtin_settings, global_settings, topdir_settings, cmdline_settings): + all_settings = builtin_settings + + for s in (global_settings, topdir_settings): + for section, section_settings in s.items(): + for setting, value in section_settings.items(): + if section not in all_settings.keys(): + all_settings[section] = {} + all_settings[section][setting] = value + + for (section, setting, value) in cmdline_settings: + if section not in all_settings.keys(): + all_settings[section] = {} + all_settings[section][setting] = value + + return all_settings + +def sigint_handler(sig, frame, func, top_dir): + logger.plain('\nShutting down...') + if isinstance(top_dir, str) and os.path.exists(top_dir): + if func in [init_config, build_update]: + logger.warning(f'{top_dir} may contain an incomplete setup!') + elif func == install_buildtools: + logger.warning(f'{top_dir} may contain an incomplete buildtools installation!') + exit() + +def main(): + def add_setup_dir_arg(parser): + setup_dir = get_setup_dir_via_bbpath() + if setup_dir: + parser.add_argument('--setup-dir', default=setup_dir, help="Path to the setup, default is %(default)s via BBPATH") + else: + parser.add_argument('--setup-dir', required=True, help="Path to the setup") + + parser = argparse.ArgumentParser( + description="BitBake setup utility. Run with 'init' argument to get started.", + epilog="Use %(prog)s --help to get help on a specific command" + ) + parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true') + parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true') + parser.add_argument('--color', choices=['auto', 'always', 'never'], default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', metavar='COLOR') + parser.add_argument('--no-network', action='store_true', help='Do not check whether configuration repositories and layer repositories have been updated; use only the local cache.') + parser.add_argument('--global-settings', action='store', metavar='PATH', help='Path to the global settings file.') + parser.add_argument('--setting', default=[], action='append', dest='cmdline_settings', + nargs=3, metavar=('SECTION', 'SETTING', 'VALUE'), + help='Modify a setting (for this bitbake-setup invocation only), for example "--setting default top-dir-prefix /path/to/top/dir".') + + subparsers = parser.add_subparsers() + + parser_list = subparsers.add_parser('list', help='List available configurations') + parser_list.add_argument('--with-expired', action='store_true', help='List also configurations that are no longer supported due to reaching their end-of-life dates.') + parser_list.add_argument('--write-json', action='store', help='Write available configurations into a json file so they can be programmatically processed.') + parser_list.set_defaults(func=list_configs) + + parser_init = subparsers.add_parser('init', help='Select a configuration and initialize a setup from it') + parser_init.add_argument('config', nargs='*', help="path/URL/id to a configuration file (use 'list' command to get available ids), followed by configuration options. Bitbake-setup will ask to choose from available choices if command line doesn't completely specify them.") + parser_init.add_argument('--non-interactive', action='store_true', help='Do not ask to interactively choose from available options; if bitbake-setup cannot make a decision it will stop with a failure.') + parser_init.add_argument('--source-overrides', action='store', help='Override sources information (repositories/revisions) with values from a local json file.') + parser_init.add_argument('--setup-dir-name', action='store', help='A custom setup directory name under the top directory.') + parser_init.add_argument('--skip-selection', action='append', help='Do not select and set an option/fragment from available choices; the resulting bitbake configuration may be incomplete.') + parser_init.add_argument('-L', '--use-local-source', default=[], action='append', nargs=2, metavar=('SOURCE_NAME', 'PATH'), + help='Symlink local source into a build, instead of getting it as prescribed by a configuration (useful for local development).') + parser_init.add_argument('--init-vscode', action=argparse.BooleanOptionalAction, default=bool(shutil.which('code')), + help='Generate VSCode workspace configuration (default: %(default)s)') + parser_init.set_defaults(func=init_config) + + parser_status = subparsers.add_parser('status', help='Check if the setup needs to be synchronized with configuration') + add_setup_dir_arg(parser_status) + parser_status.set_defaults(func=build_status) + + parser_update = subparsers.add_parser('update', help='Update a setup to be in sync with configuration') + add_setup_dir_arg(parser_update) + parser_update.add_argument('--update-bb-conf', choices=['prompt', 'yes', 'no'], default='prompt', help='Update bitbake configuration files (bblayers.conf, local.conf) (default: prompt)') + parser_update.add_argument('--rebase-conflicts-strategy', choices=['abort', 'backup'], default='abort', + help="What to do when a layer repository has local modifications that prevent " + "an in-place update: 'abort' (default) aborts with an error message; " + "'backup' renames the directory to a timestamped backup and re-clones from upstream.") + parser_update.set_defaults(func=build_update) + + parser_install_buildtools = subparsers.add_parser('install-buildtools', help='Install buildtools which can help fulfil missing or incorrect dependencies on the host machine') + add_setup_dir_arg(parser_install_buildtools) + parser_install_buildtools.add_argument('--force', action='store_true', help='Force a reinstall of buildtools over the previous installation.') + parser_install_buildtools.set_defaults(func=install_buildtools) + + parser_settings_arg_global = argparse.ArgumentParser(add_help=False) + parser_settings_arg_global.add_argument('--global', action='store_true', help="Modify the setting in a global settings file, rather than one specific to a top directory") + + parser_settings = subparsers.add_parser('settings', + help='List current settings, or set or unset a setting in a settings file (e.g. the default prefix and name of the top directory, the location of configuration registry, downloads directory and other settings specific to a top directory)') + parser_settings.set_defaults(func=settings_func) + + subparser_settings = parser_settings.add_subparsers(dest="subcommand", required=True, help="The action to perform on the settings file") + + subparser_settings.add_parser('list', help="List all settings with their values") + + parser_settings_set = subparser_settings.add_parser('set', parents=[parser_settings_arg_global], + help="In a Section, set a setting to a certain value") + parser_settings_set.add_argument("section", metavar="
", help="Section in a settings file, typically 'default'") + parser_settings_set.add_argument("setting", metavar="", help="Name of a setting") + parser_settings_set.add_argument("value", metavar="", help="The setting value") + + parser_settings_unset = subparser_settings.add_parser('unset', parents=[parser_settings_arg_global], + help="Unset a setting, e.g. 'bitbake-setup settings unset default registry' would revert to the registry setting in a global settings file") + parser_settings_unset.add_argument("section", metavar="
", help="Section in a settings file, typically 'default'") + parser_settings_unset.add_argument("setting", metavar="", help="The setting to remove") + + args = parser.parse_args() + + if args.debug: + logger.setLevel(logging.DEBUG) + elif args.quiet: + logger.setLevel(logging.ERROR) + + # Need to re-run logger_create with color argument + # (will be the same logger since it has the same name) + bb.msg.logger_create('bitbake-setup', output=sys.stdout, + color=args.color, + level=logger.getEffectiveLevel()) + + if 'func' in args: + if hasattr(args, 'setup_dir'): + if not os.path.exists(os.path.join(args.setup_dir,'build', 'init-build-env')): + logger.error("Not a valid setup directory: build/init-build-env does not exist in {}".format(args.setup_dir)) + return + + if not hasattr(args, 'non_interactive'): + args.non_interactive = True + + builtin_settings = {} + builtin_settings['default'] = { + 'top-dir-prefix':os.getcwd(), + 'top-dir-name':'bitbake-builds', + 'registry':get_default_registry(), + 'use-full-setup-dir-name':'no', + 'common-sstate':'yes', + } + + global_settings = load_settings(global_settings_path(args)) + top_dir = get_top_dir(args, merge_settings(builtin_settings, global_settings, {}, args.cmdline_settings)) + + # register handler now to pass top_dir + _handler = functools.partial(sigint_handler, + func=args.func, + top_dir=os.path.abspath(top_dir)) + signal.signal(signal.SIGINT, _handler) + + # This cannot be set with the rest of the builtin settings as top_dir needs to be determined first + builtin_settings['default']['dl-dir'] = os.path.join(top_dir, '.bitbake-setup-downloads') + + topdir_settings = load_settings(topdir_settings_path(top_dir)) + all_settings = merge_settings(builtin_settings, global_settings, topdir_settings, args.cmdline_settings) + + if args.func == settings_func: + settings_func(top_dir, all_settings, args) + return + if args.func == list_configs: + list_configs(all_settings, args) + return + + logger.info('Bitbake-setup is using {} as top directory.'.format(top_dir)) + + try: + if args.func == init_config: + init_config(top_dir, all_settings, args) + else: + d = init_bb_cache(top_dir, all_settings, args) + args.func(top_dir, all_settings, args, d) + + save_bb_cache() + except (SystemExit, KeyboardInterrupt): + raise + except Exception as e: + if args.debug: + raise + logger.error(str(e)) + sys.exit(1) + else: + parser.print_help() + +if __name__ == '__main__': + main() diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker index e8073f2a..ae21f277 100755 --- a/bitbake/bin/bitbake-worker +++ b/bitbake/bin/bitbake-worker @@ -9,6 +9,7 @@ import os import sys import warnings warnings.simplefilter("default") +warnings.filterwarnings("ignore", category=DeprecationWarning, message=".*use.of.fork.*may.lead.to.deadlocks.in.the.child.*") sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib')) from bb import fetch2 import logging @@ -21,9 +22,15 @@ import traceback import queue import shlex import subprocess -from multiprocessing import Lock +import fcntl from threading import Thread +Lock = bb.multiprocessing.Lock + +# Remove when we have a minimum of python 3.10 +if not hasattr(fcntl, 'F_SETPIPE_SZ'): + fcntl.F_SETPIPE_SZ = 1031 + bb.utils.check_system_locale() # Users shouldn't be running this code directly @@ -44,7 +51,6 @@ if sys.argv[1].startswith("decafbadbad"): # updates to log files for use with tail try: if sys.stdout.name == '': - import fcntl fl = fcntl.fcntl(sys.stdout.fileno(), fcntl.F_GETFL) fl |= os.O_SYNC fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, fl) @@ -56,6 +62,12 @@ logger = logging.getLogger("BitBake") worker_pipe = sys.stdout.fileno() bb.utils.nonblockingfd(worker_pipe) +# Try to make the pipe buffers larger as it is much more efficient. If we can't +# e.g. out of buffer space (/proc/sys/fs/pipe-user-pages-soft) then just pass over. +try: + fcntl.fcntl(worker_pipe, fcntl.F_SETPIPE_SZ, 512 * 1024) +except: + pass # Need to guard against multiprocessing being used in child processes # and multiple processes trying to write to the parent at the same time worker_pipe_lock = None @@ -105,7 +117,7 @@ def worker_flush(worker_queue): if not worker_queue.empty(): worker_queue_int.extend(worker_queue.get()) written = os.write(worker_pipe, worker_queue_int) - worker_queue_int = worker_queue_int[written:] + del worker_queue_int[0:written] except (IOError, OSError) as e: if e.errno != errno.EAGAIN and e.errno != errno.EPIPE: raise @@ -171,11 +183,8 @@ def fork_off_task(cfg, data, databuilder, workerdata, extraconfigdata, runtask): elif workerdata["umask"]: umask = workerdata["umask"] if umask: - # umask might come in as a number or text string.. - try: - umask = int(umask, 8) - except TypeError: - pass + # Convert to a python numeric value as it could be a string + umask = bb.utils.to_filemode(umask) dry_run = cfg.dry_run or runtask['dry_run'] @@ -279,8 +288,10 @@ def fork_off_task(cfg, data, databuilder, workerdata, extraconfigdata, runtask): else: logger.debug("Skipping disable network for %s since %s is not a local uid." % (taskname, uid)) - # exported_vars() returns a generator which *cannot* be passed to os.environ.update() - # successfully. We also need to unset anything from the environment which shouldn't be there + # exported_vars() expands each value, so it must be called before + # empty_environment() below: an expansion that shells out needs the + # environment bitbake was started with. We also need to unset + # anything from the environment which shouldn't be there exports = bb.data.exported_vars(the_data) bb.utils.empty_environment() @@ -357,7 +368,7 @@ class runQueueWorkerPipe(): def read(self): start = len(self.queue) try: - self.queue.extend(self.input.read(102400) or b"") + self.queue.extend(self.input.read(512*1024) or b"") except (OSError, IOError) as e: if e.errno != errno.EAGAIN: raise diff --git a/bitbake/bin/git-make-shallow b/bitbake/bin/git-make-shallow index 9de557c1..e6c180b4 100755 --- a/bitbake/bin/git-make-shallow +++ b/bitbake/bin/git-make-shallow @@ -115,8 +115,8 @@ def filter_refs(refs): all_refs = get_all_refs() to_remove = set(all_refs) - set(refs) if to_remove: - check_output(['xargs', '-0', '-n', '1'] + git_cmd + ['update-ref', '-d', '--no-deref'], - input=''.join(l + '\0' for l in to_remove)) + check_output(git_cmd + ['update-ref', '--no-deref', '--stdin', '-z'], + input=''.join('delete ' + l + '\0\0' for l in to_remove)) def follow_history_intersections(revisions, refs): diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster index f002c8c1..d20cbe90 100755 --- a/bitbake/bin/toaster +++ b/bitbake/bin/toaster @@ -164,14 +164,20 @@ fi # in the local layers that currently make using an arbitrary # toasterconf.json difficult. -. $OE_ROOT/.templateconf -if [ -n "$TEMPLATECONF" ]; then - if [ ! -d "$TEMPLATECONF" ]; then - # Allow TEMPLATECONF=meta-xyz/conf as a shortcut - if [ -d "$OE_ROOT/$TEMPLATECONF" ]; then - TEMPLATECONF="$OE_ROOT/$TEMPLATECONF" +# BBSETUP no longer supports .templateconf +if [ -f $OE_ROOT/.templateconf ] ; then + . $OE_ROOT/.templateconf + if [ -n "$TEMPLATECONF" ]; then + if [ ! -d "$TEMPLATECONF" ]; then + # Allow TEMPLATECONF=meta-xyz/conf as a shortcut + if [ -d "$OE_ROOT/$TEMPLATECONF" ]; then + TEMPLATECONF="$OE_ROOT/$TEMPLATECONF" + fi fi fi +else + # Force poky + TEMPLATECONF="layers/meta-yocto/meta-poky/conf/templates/default" fi unset OE_ROOT @@ -255,7 +261,8 @@ fi # 2) the build dir (in build) # 3) the sqlite db if that is being used. # 4) pid's we need to clean up on exit/shutdown -export TOASTER_DIR=$TOASTERDIR +# By default we move this all into the Toaster's parent build directory +export TOASTER_DIR=$BUILDDIR export BB_ENV_PASSTHROUGH_ADDITIONS="$BB_ENV_PASSTHROUGH_ADDITIONS TOASTER_DIR" # Determine the action. If specified by arguments, fine, if not, toggle it diff --git a/bitbake/contrib/b4-wrapper-bitbake.py b/bitbake/contrib/b4-wrapper-bitbake.py new file mode 100755 index 00000000..87dff2c3 --- /dev/null +++ b/bitbake/contrib/b4-wrapper-bitbake.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# +# This script is to be called by b4: +# - through b4.send-auto-cc-cmd with "send-auto-cc-cmd" as first argument, +# +# When send-auto-cc-cmd is passed: +# +# This returns the list of Cc recipients for a patch. +# +# This script takes as stdin a patch. + +import subprocess +import sys + +cmd = sys.argv[1] +if cmd != "send-auto-cc-cmd": + sys.exit(-1) + +patch = sys.stdin.read() + +if subprocess.call(["which", "lsdiff"], stdout=subprocess.DEVNULL) != 0: + print("lsdiff missing from host, please install patchutils") + sys.exit(-1) + +files = subprocess.check_output(["lsdiff", "--strip-match=1", "--strip=1", "--include=doc/*"], + input=patch, text=True) +if len(files): + print("docs@lists.yoctoproject.org") +else: +# Handle patches made with --no-prefix + files = subprocess.check_output(["lsdiff", "--include=doc/*"], + input=patch, text=True) + if len(files): + print("docs@lists.yoctoproject.org") + +sys.exit(0) diff --git a/bitbake/contrib/hashserv/Dockerfile b/bitbake/contrib/hashserv/Dockerfile index 74b4a3be..aec1f86f 100644 --- a/bitbake/contrib/hashserv/Dockerfile +++ b/bitbake/contrib/hashserv/Dockerfile @@ -11,7 +11,7 @@ FROM alpine:3.13.1 -RUN apk add --no-cache python3 +RUN apk add --no-cache python3 libgcc COPY bin/bitbake-hashserv /opt/bbhashserv/bin/ COPY lib/hashserv /opt/bbhashserv/lib/hashserv/ diff --git a/bitbake/contrib/vim/ftdetect/bitbake.vim b/bitbake/contrib/vim/ftdetect/bitbake.vim index 09fc4dc7..1d958fec 100644 --- a/bitbake/contrib/vim/ftdetect/bitbake.vim +++ b/bitbake/contrib/vim/ftdetect/bitbake.vim @@ -11,10 +11,18 @@ if &compatible || version < 600 || exists("b:loaded_bitbake_plugin") endif " .bb, .bbappend and .bbclass -au BufNewFile,BufRead *.{bb,bbappend,bbclass} set filetype=bitbake +au BufNewFile,BufRead *.{bb,bbappend,bbclass} setfiletype bitbake -" .inc -au BufNewFile,BufRead *.inc set filetype=bitbake +" .inc -- meanwhile included upstream +if !has("patch-9.1.1732") + au BufNewFile,BufRead *.inc call s:BBIncDetect() + def s:BBIncDetect() + var lines = getline(1) .. getline(2) .. getline(3) + if lines =~# '\<\%(require\|inherit\)\>' || lines =~# '[A-Z][A-Za-z0-9_:${}/]*\s\+\%(??\|[?:+.]\)\?=.\? ' + set filetype=bitbake + endif + enddef +endif " .conf au BufNewFile,BufRead *.conf diff --git a/bitbake/default-registry/configurations/oe-nodistro-master.conf.json b/bitbake/default-registry/configurations/oe-nodistro-master.conf.json new file mode 100644 index 00000000..5aa148d4 --- /dev/null +++ b/bitbake/default-registry/configurations/oe-nodistro-master.conf.json @@ -0,0 +1,47 @@ +{ + "description": "OpenEmbedded - 'nodistro' basic configuration", + "sources": { + "bitbake": { + "git-remote": { + "uri": "https://git.openembedded.org/bitbake", + "branch": "master", + "rev": "master" + } + }, + "openembedded-core": { + "git-remote": { + "uri": "https://git.openembedded.org/openembedded-core", + "branch": "master", + "rev": "master" + } + }, + "yocto-docs": { + "git-remote": { + "uri": "https://git.yoctoproject.org/yocto-docs", + "branch": "master", + "rev": "master" + } + } + }, + "bitbake-setup": { + "configurations": [ + { + "name": "nodistro", + "description": "OpenEmbedded 'nodistro'", + "setup-dir-name": "oe-nodistro-master", + "bb-layers": ["openembedded-core/meta"], + "oe-fragments-one-of": { + "machine": { + "description": "Target machines", + "options" : [ + { "name": "machine/qemux86-64", "description": "x86-64 system on QEMU" }, + { "name": "machine/qemuarm64", "description": "ARMv8 system on QEMU" }, + { "name": "machine/qemuriscv64", "description": "RISC-V system on QEMU" } + ] + } + } + } + ] + }, + "version": "1.0" +} diff --git a/bitbake/default-registry/configurations/oe-nodistro-whinlatter.conf.json b/bitbake/default-registry/configurations/oe-nodistro-whinlatter.conf.json new file mode 100644 index 00000000..7b183ae1 --- /dev/null +++ b/bitbake/default-registry/configurations/oe-nodistro-whinlatter.conf.json @@ -0,0 +1,56 @@ +{ + "description": "OpenEmbedded - 'nodistro' basic configuration, release 5.3 'whinlatter'", + "expires": "2026-05-31", + "sources": { + "bitbake": { + "git-remote": { + "remotes": { + "origin": { + "uri": "https://git.openembedded.org/bitbake" + } + }, + "branch": "2.16", + "rev": "2.16" + } + }, + "openembedded-core": { + "git-remote": { + "remotes": { + "origin": { + "uri": "https://git.openembedded.org/openembedded-core" + } + }, + "branch": "whinlatter", + "rev": "whinlatter" + } + }, + "yocto-docs": { + "git-remote": { + "remotes": { + "origin": { + "uri": "https://git.yoctoproject.org/yocto-docs" + } + }, + "branch": "whinlatter", + "rev": "whinlatter" + } + } + }, + "bitbake-setup": { + "configurations": [ + { + "name": "nodistro", + "description": "OpenEmbedded 'nodistro'", + "setup-dir-name": "oe-nodistro-whinlatter", + "bb-layers": ["openembedded-core/meta"], + "oe-fragments-one-of": { + "machine": { + "description": "Target machines", + "options" : ["machine/qemux86-64", "machine/qemuarm64", "machine/qemuriscv64"] + } + } + } + ] + }, + "version": "1.0" +} diff --git a/bitbake/default-registry/configurations/oe-nodistro-wrynose.conf.json b/bitbake/default-registry/configurations/oe-nodistro-wrynose.conf.json new file mode 100644 index 00000000..278ce4dd --- /dev/null +++ b/bitbake/default-registry/configurations/oe-nodistro-wrynose.conf.json @@ -0,0 +1,48 @@ +{ + "description": "OpenEmbedded - 'nodistro' basic configuration, release 6.0 'wrynose'", + "expires": "2030-05-31", + "sources": { + "bitbake": { + "git-remote": { + "uri": "https://git.openembedded.org/bitbake", + "branch": "2.18", + "rev": "2.18" + } + }, + "openembedded-core": { + "git-remote": { + "uri": "https://git.openembedded.org/openembedded-core", + "branch": "wrynose", + "rev": "wrynose" + } + }, + "yocto-docs": { + "git-remote": { + "uri": "https://git.yoctoproject.org/yocto-docs", + "branch": "wrynose", + "rev": "wrynose" + } + } + }, + "bitbake-setup": { + "configurations": [ + { + "name": "nodistro", + "description": "OpenEmbedded 'nodistro'", + "setup-dir-name": "oe-nodistro-wrynose", + "bb-layers": ["openembedded-core/meta"], + "oe-fragments-one-of": { + "machine": { + "description": "Target machines", + "options" : [ + { "name": "machine/qemux86-64", "description": "x86-64 system on QEMU" }, + { "name": "machine/qemuarm64", "description": "ARMv8 system on QEMU" }, + { "name": "machine/qemuriscv64", "description": "RISC-V system on QEMU" } + ] + } + } + } + ] + }, + "version": "1.0" +} diff --git a/bitbake/default-registry/configurations/poky-master.conf.json b/bitbake/default-registry/configurations/poky-master.conf.json new file mode 100644 index 00000000..58d37964 --- /dev/null +++ b/bitbake/default-registry/configurations/poky-master.conf.json @@ -0,0 +1,73 @@ +{ + "description": "Poky - The Yocto Project testing distribution configurations and hardware test platforms", + "sources": { + "bitbake": { + "git-remote": { + "uri": "https://git.openembedded.org/bitbake", + "branch": "master", + "rev": "master" + } + }, + "openembedded-core": { + "git-remote": { + "uri": "https://git.openembedded.org/openembedded-core", + "branch": "master", + "rev": "master" + } + }, + "meta-yocto": { + "git-remote": { + "uri": "https://git.yoctoproject.org/meta-yocto", + "branch": "master", + "rev": "master" + } + }, + "yocto-docs": { + "git-remote": { + "uri": "https://git.yoctoproject.org/yocto-docs", + "branch": "master", + "rev": "master" + } + } + }, + "bitbake-setup": { + "configurations": [ + { + "bb-layers": ["openembedded-core/meta","meta-yocto/meta-yocto-bsp","meta-yocto/meta-poky"], + "setup-dir-name": "$distro-master", + "oe-fragments-one-of": { + "machine": { + "description": "Target machines", + "options" : [ + { "name": "machine/qemux86-64", "description": "x86-64 system on QEMU" }, + { "name": "machine/qemuarm64", "description": "ARMv8 system on QEMU" }, + { "name": "machine/qemuriscv64", "description": "RISC-V system on QEMU" }, + { "name": "machine/genericarm64", "description": "Arm64 SystemReady IR/ES platforms" }, + { "name": "machine/genericx86-64", "description": "x86_64 (64-bit) PCs and servers" } + ] + }, + "distro": { + "description": "Target distributions", + "options" : [ + { "name": "distro/poky", "description": "Yocto Project Reference Distro" }, + { "name": "distro/poky-altcfg", "description": "Poky alternative with systemd as init manager" }, + { "name": "distro/poky-tiny", "description": "Poky alternative optimized for size" } + ] + } + }, + "configurations": [ + { + "name": "poky", + "description": "Poky - The Yocto Project testing distribution" + }, + { + "name": "poky-with-sstate", + "description": "Poky - The Yocto Project testing distribution with internet sstate acceleration. Use with caution as it requires a completely robust local network with sufficient bandwidth.", + "oe-fragments": ["core/yocto/sstate-mirror-cdn"] + } + ] + } + ] + }, + "version": "1.0" +} diff --git a/bitbake/default-registry/configurations/poky-whinlatter.conf.json b/bitbake/default-registry/configurations/poky-whinlatter.conf.json new file mode 100644 index 00000000..4827ebe1 --- /dev/null +++ b/bitbake/default-registry/configurations/poky-whinlatter.conf.json @@ -0,0 +1,80 @@ +{ + "description": "Poky - The Yocto Project testing distribution configurations and hardware test platforms, release 5.3 'whinlatter'", + "expires": "2026-05-31", + "sources": { + "bitbake": { + "git-remote": { + "remotes": { + "origin": { + "uri": "https://git.openembedded.org/bitbake" + } + }, + "branch": "2.16", + "rev": "2.16" + } + }, + "openembedded-core": { + "git-remote": { + "remotes": { + "origin": { + "uri": "https://git.openembedded.org/openembedded-core" + } + }, + "branch": "whinlatter", + "rev": "whinlatter" + } + }, + "meta-yocto": { + "git-remote": { + "remotes": { + "origin": { + "uri": "https://git.yoctoproject.org/meta-yocto" + } + }, + "branch": "whinlatter", + "rev": "whinlatter" + } + }, + "yocto-docs": { + "git-remote": { + "remotes": { + "origin": { + "uri": "https://git.yoctoproject.org/yocto-docs" + } + }, + "branch": "whinlatter", + "rev": "whinlatter" + } + } + }, + "bitbake-setup": { + "configurations": [ + { + "bb-layers": ["openembedded-core/meta","meta-yocto/meta-yocto-bsp","meta-yocto/meta-poky"], + "setup-dir-name": "$distro-whinlatter", + "oe-fragments-one-of": { + "machine": { + "description": "Target machines", + "options" : ["machine/qemux86-64", "machine/qemuarm64", "machine/qemuriscv64", "machine/genericarm64", "machine/genericx86-64"] + }, + "distro": { + "description": "Target Distributions", + "options" : ["distro/poky", "distro/poky-altcfg", "distro/poky-tiny"] + } + }, + "configurations": [ + { + "name": "poky", + "description": "Poky - The Yocto Project testing distribution" + }, + { + "name": "poky-with-sstate", + "description": "Poky - The Yocto Project testing distribution with internet sstate acceleration. Use with caution as it requires a completely robust local network with sufficient bandwidth.", + "oe-fragments": ["core/yocto/sstate-mirror-cdn"] + } + ] + } + ] + }, + "version": "1.0" +} diff --git a/bitbake/default-registry/configurations/poky-wrynose.conf.json b/bitbake/default-registry/configurations/poky-wrynose.conf.json new file mode 100644 index 00000000..d2792843 --- /dev/null +++ b/bitbake/default-registry/configurations/poky-wrynose.conf.json @@ -0,0 +1,74 @@ +{ + "description": "Poky - The Yocto Project testing distribution configurations and hardware test platforms, release 6.0 'wrynose'", + "expires": "2030-05-31", + "sources": { + "bitbake": { + "git-remote": { + "uri": "https://git.openembedded.org/bitbake", + "branch": "2.18", + "rev": "2.18" + } + }, + "openembedded-core": { + "git-remote": { + "uri": "https://git.openembedded.org/openembedded-core", + "branch": "wrynose", + "rev": "wrynose" + } + }, + "meta-yocto": { + "git-remote": { + "uri": "https://git.yoctoproject.org/meta-yocto", + "branch": "wrynose", + "rev": "wrynose" + } + }, + "yocto-docs": { + "git-remote": { + "uri": "https://git.yoctoproject.org/yocto-docs", + "branch": "wrynose", + "rev": "wrynose" + } + } + }, + "bitbake-setup": { + "configurations": [ + { + "bb-layers": ["openembedded-core/meta","meta-yocto/meta-yocto-bsp","meta-yocto/meta-poky"], + "setup-dir-name": "$distro-wrynose", + "oe-fragments-one-of": { + "machine": { + "description": "Target machines", + "options" : [ + { "name": "machine/qemux86-64", "description": "x86-64 system on QEMU" }, + { "name": "machine/qemuarm64", "description": "ARMv8 system on QEMU" }, + { "name": "machine/qemuriscv64", "description": "RISC-V system on QEMU" }, + { "name": "machine/genericarm64", "description": "Arm64 SystemReady IR/ES platforms" }, + { "name": "machine/genericx86-64", "description": "x86_64 (64-bit) PCs and servers" } + ] + }, + "distro": { + "description": "Target distributions", + "options" : [ + { "name": "distro/poky", "description": "Yocto Project Reference Distro" }, + { "name": "distro/poky-altcfg", "description": "Poky alternative with systemd as init manager" }, + { "name": "distro/poky-tiny", "description": "Poky alternative optimized for size" } + ] + } + }, + "configurations": [ + { + "name": "poky", + "description": "Poky - The Yocto Project testing distribution" + }, + { + "name": "poky-with-sstate", + "description": "Poky - The Yocto Project testing distribution with internet sstate acceleration. Use with caution as it requires a completely robust local network with sufficient bandwidth.", + "oe-fragments": ["core/yocto/sstate-mirror-cdn"] + } + ] + } + ] + }, + "version": "1.0" +} diff --git a/bitbake/doc/.gitignore b/bitbake/doc/.gitignore index 69fa449d..3a570d8c 100644 --- a/bitbake/doc/.gitignore +++ b/bitbake/doc/.gitignore @@ -1 +1,4 @@ _build/ +sphinx-static/switchers.js +releases.json +releases.rst diff --git a/bitbake/doc/Makefile b/bitbake/doc/Makefile index 996f01b7..2f8e898c 100644 --- a/bitbake/doc/Makefile +++ b/bitbake/doc/Makefile @@ -27,7 +27,7 @@ publish: Makefile html singlehtml sed -i -e 's@index.html#@singleindex.html#@g' $(BUILDDIR)/$(DESTDIR)/singleindex.html clean: - @rm -rf $(BUILDDIR) + @rm -rf $(BUILDDIR) sphinx-static/switchers.js releases.json releases.rst # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst new file mode 100644 index 00000000..b3ed2129 --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst @@ -0,0 +1,1296 @@ +.. SPDX-License-Identifier: CC-BY-2.5 + +============================================= +Setting Up The Environment With bitbake-setup +============================================= + +| + +Setting up layers and configuring builds can be done with the +``bitbake-setup`` tool. This tool acts as a top-level utility which can perform +the following tasks: + +- Parse a JSON configuration file that describes what layers and which snippets + of configuration to use. + +- Clone the layers onto the versions specified in the configuration file. + +- Create and setup a directory ready for building what is specified in + configuration files. + +- Behave according to global or per-project settings. + +- Synchronize with upstream configuration changes. + +Quick Start +=========== + +#. First, run the following command to install the ``bitbake-setup`` + command-line tool in a Python `virtual environment + `__: + + .. code-block:: shell + + $ python3 -m venv --clear ./bitbake-setup-venv + $ . ./bitbake-setup-venv/bin/activate + $ pip install bitbake-setup + + .. tip:: + + To update ``bitbake-setup`` when already installed, you can run: + + .. code-block:: console + + $ pip install --upgrade bitbake-setup + +#. To start using ``bitbake-setup``, run: + + .. code-block:: shell + + $ bitbake-setup init + + This command will ask you to choose which configurations to use available as + part of the default BitBake :term:`Configuration Templates `. + + .. note:: + + These default configurations are located in ``./bitbake/default-registry/``. + See the :ref:`ref-bbsetup-section-config-reference` section to learn more + about ``bitbake-setup`` input configuration files. + +#. Here is an example of files and directories structure created by ``bitbake-setup init``:: + + ./bitbake-builds/ + ├── site.conf + └── poky-master-poky-distro_poky-machine_qemux86-64/ +    ├── build/ +    ├── config/ +    └── layers/ + + With: + + - ``./bitbake-builds``: the :term:`Top Directory`, where ``bitbake-setup`` + configures everything. This directory can be configured with the + :ref:`ref-bbsetup-setting-top-dir-prefix` and + :ref:`ref-bbsetup-setting-top-dir-name` settings. + + - ``poky-master-poky-distro_poky-machine_qemux86-64``: a :term:`Setup` + directory, which holds a :term:`Setup`: a result of the choices made + during the ``bitbake-setup init`` execution. + + The name of the directory will vary depending on the choices. + + - ``config/``: holds the :term:`Configuration Instance`, which embeds the + :term:`Configuration Template` (first choice of the :ref:`ref-bbsetup-command-init` command) + and the choices made during the initialization. + + - ``build/``: the :term:`BitBake Build` directory, where BitBake stores + its own configuration and outputs for the builds. + + - ``layers/``: the directory where :ref:`layers + ` and other + repositories managed by ``bitbake-setup`` are stored and updated. + + - ``site.conf``: a BitBake configuration file that contains site specific + configurations for your build environment. When it is created, it contains + some variable definitions that are based on your current :term:`settings`. + Comments in this file will help you understand what these variables + correspond to. + + These configurations are shared between the :term:`setups `. + +#. Source the ``init-build-env`` file present in the :term:`BitBake Build` + directory: + + .. code-block:: shell + + $ source ./poky-master-poky-distro_poky-machine_qemux86-64/build/init-build-env + + This command will prepare your current shell with the BitBake environment. + +#. You can then start running ``bitbake`` in the current shell. For more information + on how to use ``bitbake``, see the :doc:`/bitbake-user-manual/bitbake-user-manual-execution` + section of this manual. + +.. note:: + + The ``bitbake-setup`` tool is developed and maintained in the BitBake + repository and can also be used directly from there: + + .. code-block:: console + + $ git clone https://git.openembedded.org/bitbake ./bitbake + $ ./bitbake/bin/bitbake-setup ... + +Terminology +=========== + +The ``bitbake-setup`` tool revolves around some common terms we define in this +section. + +``bitbake-setup`` works with a specific hierarchy of files and directories, that +can be represented as follows:: + + Top Directory + ├── site.conf + ├── Setup 1 + │   ├── build/ + │ ├── config/ + │   └── layers/ + ├── Setup 2 + │   ├── build/ + │ ├── config/ + │   └── layers/ + ... + +The "Top Directory" and "Setup" directories are defined as follows: + +.. glossary:: + :term:`Top Directory` + The top directory is the working directory of ``bitbake-setup``, where its + outputs end-up (unless otherwise configured by :term:`settings` such as + :ref:`ref-bbsetup-setting-dl-dir`). + + The location of this directory can be changed with the + :ref:`ref-bbsetup-setting-top-dir-prefix` and + :ref:`ref-bbsetup-setting-top-dir-name` settings. + + The top directory contains one or more :term:`Setup` directories, each of + them containing a :term:`Setup`, and a :term:`Site Configuration File` + (named ``site.conf``). + + :term:`Setup` + A Setup is the result of the :ref:`ref-bbsetup-command-init` + command, which creates a :term:`Setup` directory. It is constructed from a + :term:`Configuration Template` and choices made during the ``init`` command. + + It contains at least: + + - A :term:`BitBake Build` (``build/`` directory). + - A :term:`Configuration Instance` (``config/`` directory). + - Sources such as :ref:`layers + ` or other + repositories managed by ``bitbake-setup`` (``layers/`` directory). + + :term:`Site Configuration File` + This file named ``site.conf`` is a unique file located in the :term:`Top + Directory`, and holds top-level BitBake configuration statements shared + between the :term:`Setups `. + + When it is created, it contains some variable definitions that are based + on your current :term:`settings`. Comments in this file will help you + understand what these variables correspond to. + +The following components are involved to create the content of these directories: + +.. glossary:: + :term:`BitBake Build` + A BitBake Build is a sub-tree inside a :term:`Setup` that BitBake itself + operates on. The files in the ``conf/`` directory of a :term:`BitBake + Build` constitute the :ref:`BitBake configuration + `. + + :term:`Configuration Template` + A Configuration Template is a file in JSON format containing a template to + create a :term:`Setup`. These files are used during the :ref:`ref-bbsetup-command-init` + command as a starting point to configure the :term:`Setup`. When the + command runs, the user may be prompted with choices to further specify the + :term:`Setup` to create. + + It is also possible to specify the choices on the command line for a + completely non-interactive initialization. + + :term:`Configuration Template` files are stored in :term:`registries + `, and can be listed with the :ref:`ref-bbsetup-command-list` + command. + + :term:`Configuration Template` files must end with the ``.conf.json`` + suffix for ``bitbake-setup`` to locate them. + + .. note:: + + The default :term:`Configuration Templates ` are + located in the BitBake repository in a local registry. the + ``default-registry/`` directory. This can be modified with the + :ref:`ref-bbsetup-setting-registry` setting. + + :ref:`ref-bbsetup-command-status` will tell if a :term:`Setup` + is in sync with the :term:`Configuration Template` it was constructed from + (typically: layer updates). + + :ref:`ref-bbsetup-command-update` will bring a :term:`Setup` + in sync with its :term:`Configuration Template`. + + :term:`Configuration Instance` + The :term:`Configuration Instance` is stored in the ``config/`` directory + in a :term:`Setup`. It embeds the :term:`Configuration Template` and the + choices made during the initialization. + + It is also a Git repository, that contains a history of the configuration + instance and updates made to it via :ref:`ref-bbsetup-command-update`. + + :term:`Registry` + A configuration registry is a place where one or more :term:`Configuration + Templates ` are stored. + + The directory structure of the registry can be any: ``bitbake-setup`` + recursively find files ending with ``.conf.json`` and consider it a + :term:`Configuration Template`. + + The registry location is configured through the + :ref:`ref-bbsetup-setting-registry` setting. This location can be the URL to + a Git repository, a local directory, or any URI supported by the BitBake + fetcher (see the :doc:`/bitbake-user-manual/bitbake-user-manual-fetching` + section for more information on fetchers). + + :term:`Settings` + Settings are operational parameters that are global to all builds under a + :term:`Top Directory`, stored in a ``settings.conf`` file. For example, + this could be the location of the configuration registry, or where the + BitBake fetcher should store the downloads. + + There are also global settings, common to all top directories that are + stored in ``~/.config/bitbake-setup/settings.conf``. + + See the :ref:`bitbake-user-manual/bitbake-user-manual-environment-setup:Settings` + section to see the supported settings and where they can be stored. + + :term:`Source Override` + A source override is a JSON file that can be used to modify revisions and + origins of layers or other sources that need to be checked out into a + :term:`Setup` (in the ``layers/`` directory). It can be useful for example + when the master branches need to be changed to master-next for the purpose + of testing, or to set up a CI pipeline that tests code in a pull request + coming from a developer's repository and branch. + + Such a file is specified with a command-line option to + :ref:`ref-bbsetup-command-init`. + + See the :ref:`ref-bbsetup-source-overrides` section for more information on + the format of these files. + +The ``bitbake-setup`` command +============================= + +The ``bitbake-setup`` program has general options and sub-commands. These can be +obtained using ``bitbake-setup --help``. + +The general options, common to all commands, are: + +- ``-h`` or ``--help``: Show the help message and exit. +- ``-d`` or ``--debug``: Enable debug outputs. +- ``-q`` or ``--quiet``: Print only errors. +- ``--color``: Colorize output (where COLOR is auto, always, never). +- ``--no-network``: Do not check whether configuration repositories and layer + repositories have been updated; use only the local cache. +- ``--global-settings``: Path to the global settings file. +- ``--setting``: Modify a setting (for this bitbake-setup invocation only). + For example ``--setting default top-dir-prefix /path/to/top-dir``. + +.. _ref-bbsetup-command-init: + +``bitbake-setup init`` +---------------------- + +The ``bitbake-setup init`` sub-command helps initializing a :term:`Setup`. + +This command can be run without any arguments to prompt the user with +configuration options to choose from. These configuration options are taken from +the input :term:`Configuration Template` files in the :term:`registry`. + +.. note:: + + The registry location can be set with the :ref:`ref-bbsetup-setting-registry` + setting and the :ref:`ref-bbsetup-command-settings` command. + +Otherwise, the first argument to :ref:`ref-bbsetup-command-init` can be: + +- A Configuration Template ID in the registry. +- A path to a Configuration Template file on a local disk. +- An HTTP URI to the Configuration Template file. + +The choices made during the bare ``bitbake-setup init`` command can also be +passed directly on the command-line, for example:: + + bitbake-setup init poky distro/poky-tiny ... + +``bitbake-setup`` will stop and ask to make a choice if the above command does +not contain all of the required configurations to complete the sequence of +choices. + +In addition, the command can take the following arguments: + +- ``--non-interactive``: can be used to create :term:`Setups ` + without interactions from the user. The command will fail if not all the + required choices are provided in the command. + +- ``--source-overrides``: can be used to pass one or more + :ref:`source override `. See the + :ref:`ref-bbsetup-source-overrides` section. + +- ``--setup-dir-name``: can be used to configure the name of the :term:`Setup` + directory. + +- ``--skip-selection``: can be used to skip some of the choices + (which may result in an incomplete :term:`Setup`!) + +- ``-L`` or ``--use-local-source``: instead of getting a source as prescribed in + a configuration, symlink it into a :term:`Setup` from a path on local disk. This + is useful for local development where that particular source directory is managed + separately, and bitbake-setup will include it in a build but will not otherwise + touch or modify it. This option can be specified multiple times to specify multiple + local sources. + + The option can be seen as a command line shortcut to providing an override file + with a ``local`` source in it. See the :ref:`ref-bbsetup-source-overrides` section + for more information on source overrides. + +- ``--init-vscode`` / ``--no-init-vscode``: generate (or skip generating) a + ``bitbake.code-workspace`` file in the :term:`Setup` directory. The workspace + file configures the `Yocto Project BitBake + `_ + VS Code extension with paths to the build directory and the init script, and + lists the layer directories as workspace folders. Any user-added folders or + settings in an existing workspace file are preserved across updates. + The default is ``true`` when ``code`` (the VS Code binary) is found on + ``PATH``, and ``false`` otherwise. + +``bitbake-setup init`` Examples +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- This example prompts the user to choose a :term:`Configuration Template` from + a custom registry: + + .. code-block:: shell + + $ bitbake-setup \ + --setting default registry 'git://example.com/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main' \ + init + +- This example takes a :term:`Configuration Template` from a remote location + (here, one of the default configuration in BitBake): + + .. code-block:: shell + + $ bitbake-setup init https://git.openembedded.org/bitbake/plain/default-registry/configurations/oe-nodistro-master.conf.json + +- This example initializes a :term:`Setup` with: + + - A custom :ref:`ref-bbsetup-setting-top-dir-prefix` and :ref:`ref-bbsetup-setting-top-dir-name` + - A :term:`source override`, and additionally overriding openembedded-core with a locally managed directory. + - A standalone :term:`Configuration Template` file. + - Choices passed on the command-line, applied non-interactively. + + .. code-block:: shell + + $ bitbake-setup \ + --setting default top-dir-prefix /work/bitbake-setup \ + --setting default top-dir-name custom-project \ + init \ + --non-interactive \ + --source-overrides develop-branch.json \ + -L openembedded-core ~/development/openembedded-core-gadget \ + ./gadget_master.conf.json \ + gadget distro/gadget machine/gadget + +.. _ref-bbsetup-command-list: + +``bitbake-setup list`` +---------------------- + +The ``bitbake-setup list`` sub-command lists the available :term:`Configuration +Templates ` in the current :term:`registry`. + +In addition, the command can take the following arguments: + +- ``--with-expired``: list the expired configuration (e.g. older Yocto releases + that have reached their End-Of-Life dates). + +- ``--write-json``: write the configurations into a JSON file so they can be + programmatically processed. + +.. _ref-bbsetup-command-status: + +``bitbake-setup status`` +------------------------ + +The ``bitbake-setup status`` sub-command shows the status of a +:term:`Setup`. Any differences between the local copy of the :term:`Configuration +Template` and the upstream one are printed on the console. + +If the BitBake environment is sourced and ready to build, the ``bitbake-setup +status`` command (without any arguments) will show the status of the current +:term:`Setup`. + +In addition, the command can take the following arguments: + +- ``--setup-dir``: path to the :term:`Setup` to check to status for. Not + required if the command is invoked from an initialized BitBake environment + that contains :term:`BBPATH`. + +.. _ref-bbsetup-command-update: + +``bitbake-setup update`` +------------------------ + +The ``bitbake-setup update`` sub-command updates a :term:`Setup` to sync with +the latest changes from the :term:`Configuration Template` it was constructed from. +The :ref:`ref-bbsetup-command-status` command can be used to show the current +status of the :term:`Setup` before updating it. + +This command is intended to be run from a shell where the BitBake environment +has been sourced (e.g. after ``source build/init-build-env``), so that +``bitbake-setup`` can automatically identify the current :term:`Setup` without +requiring the ``--setup-dir`` argument. + +When a layer repository already exists in the :term:`Setup` (i.e. it has been +previously checked out by ``bitbake-setup init`` or a prior ``update``), the +update is performed *in place* using the fetcher's +:ref:`unpack_update ` method: the new upstream revision +is fetched into the local download cache and then rebased on top of the +checkout's current HEAD. This means any local commits in the layer directory +are preserved and rebased onto the new upstream revision. If the working tree +contains staged or unstaged changes to tracked files, the update is blocked +until those changes are committed, stashed or discarded. + +.. note:: + + ``bitbake-setup`` performs the rebase to fast-forward local commits onto + the new upstream revision, but intentionally does not go further by using + ``--autostash`` (which would silently stash uncommitted changes before the + rebase and pop them afterwards). Any uncommitted modifications are surfaced + to the user before the update proceeds, so there are no surprises from an + automatic stash/pop cycle. + +In addition, the command can take the following arguments: + +- ``--update-bb-conf``: whether to update the :term:`BitBake Build` + configuration (``local.conf``, ``bblayers.conf``, etc.). This argument can + take up to three values: + + - ``prompt`` (default): ask the user whether to update. + - ``yes``: update the configuration files. + - ``no``: don't update the configuration files. + +- ``--rebase-conflicts-strategy``: what to do when a layer repository has + local modifications or commits that prevent an in-place update. Accepted + values are: + + - ``abort`` (default): stop with an error message describing the problem. + The repository is left in its previous state (the failed rebase is + automatically aborted). The error message includes a hint to re-run with + ``--rebase-conflicts-strategy=backup``. + - ``backup``: rename the conflicting layer directory to a timestamped + ``-backup-`` path (preserving local work), then + re-clone the layer from upstream into a fresh directory. + +- ``--setup-dir``: path to the :term:`Setup` to update. Not required if the + command is invoked from an initialized BitBake environment that contains + :term:`BBPATH`. + +``bitbake-setup update`` Examples +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- This example performs a standard update from an initialized BitBake + environment. Local commits in any layer directory are rebased on top of + the new upstream revision and preserved: + + .. code-block:: shell + + $ bitbake-setup update + NOTE: Bitbake-setup is using /path/to/bitbake-builds as top directory. + NOTE: Looking up config poky-master in configuration registry + NOTE: Layer repository https://git.openembedded.org/openembedded-core + checked out into /path/to/bitbake-builds/poky-master/layers/openembedded-core + updated revision master from d383ea3... to b50d6de... + Fetching layer/tool repositories into /path/to/bitbake-builds/poky-master/layers + bitbake + meta-yocto + openembedded-core + +- This example shows what happens when a layer directory contains staged or + unstaged changes to tracked files. The update is blocked with an error: + + .. code-block:: shell + + $ bitbake-setup update + NOTE: Bitbake-setup is using /path/to/bitbake-builds as top directory. + NOTE: Looking up config poky-master in configuration registry + Fetching layer/tool repositories into /path/to/bitbake-builds/poky-master/layers + bitbake + meta-yocto + openembedded-core + ERROR: Unpack failure for URL: + 'git://git.openembedded.org/openembedded-core;protocol=https;rev=master;branch=master;destsuffix=openembedded-core'. + Repository at /path/to/bitbake-builds/poky-master/layers/openembedded-core has uncommitted changes, unable to update: + M meta/recipes-devtools/ccache/ccache_4.13.1.bb + + Commit, stash or discard your changes and re-run the update. + Use 'bitbake-setup update --rebase-conflicts-strategy=backup' + to automatically back up the directory and re-clone from upstream, + or use 'bitbake-setup init -L openembedded-core /path/to/local/checkout' + to work with a local checkout instead. + + Stashing the changes and re-running resolves the issue: + + .. code-block:: shell + + $ git -C layers/openembedded-core stash + $ bitbake-setup update + $ git -C layers/openembedded-core stash pop + +- This example shows what happens when a layer directory contains local + commits that conflict with the incoming upstream changes. The failed rebase + is automatically aborted, and the ``dldir`` remote is left in the repository + for manual resolution: + + .. code-block:: shell + + $ bitbake-setup update + ERROR: Repository at layers/openembedded-core has local commits that could + not be rebased onto the new upstream revision: + ... + Note: the 'dldir' remote points to the local download cache and may be + used to resolve the conflict manually. + Once resolved, re-run the update. + + The conflict can be resolved manually using the ``dldir`` remote that + ``bitbake-setup`` adds to the repository: + + .. code-block:: shell + + $ git -C layers/openembedded-core rebase dldir/master + # fix conflicts in an editor, then stage the resolved files: + $ git -C layers/openembedded-core add meta/recipes-core/base-files/base-files.bb + $ git -C layers/openembedded-core rebase --continue + $ bitbake-setup update + +- When manual conflict resolution is not desired, the + ``--rebase-conflicts-strategy=backup`` option can be used instead. It + preserves the conflicting directory under a timestamped backup path and + re-clones the layer cleanly from upstream: + + .. code-block:: shell + + $ bitbake-setup update --rebase-conflicts-strategy=backup + NOTE: Bitbake-setup is using /path/to/bitbake-builds as top directory. + NOTE: Looking up config poky-master in configuration registry + NOTE: Layer repository https://git.openembedded.org/openembedded-core checked + out into /path/to/bitbake-builds/poky-master/layers/openembedded-core + updated revision master from 2ec283e... to b50d6de... + Fetching layer/tool repositories into /path/to/bitbake-builds/poky-master/layers + bitbake + meta-yocto + openembedded-core + WARNING: Unpack failure for URL: + 'git://git.openembedded.org/openembedded-core;protocol=https;rev=master;branch=master;destsuffix=openembedded-core'. + Repository at /path/to/bitbake-builds/poky-master/layers/openembedded-core + has local commits that could not be rebased onto the new upstream revision: + ... + Note: the 'dldir' remote points to the local download cache and may be used to resolve the conflict manually. + Once resolved, re-run the update. + Renaming /path/to/bitbake-builds/poky-master/layers/openembedded-core to + /path/to/bitbake-builds/poky-master/layers/openembedded-core-backup.20260329160426 + to preserve your work, then re-cloning from upstream. + + The backup directory is a complete git repository. Local commits can be + recovered from it after the update by fetching a branch from the backup + into the fresh clone (git accepts local paths as remote URLs) and then + cherry-picking the desired commits. For example, given a ``my-wip`` branch + with two commits existing in the backup repository and not in the fresh clone, + the following commands can be used to apply these commits on top of the new + upstream revision in the fresh clone: + + .. code-block:: shell + + $ git -C layers/openembedded-core-backup.20260329160426 log --oneline my-wip + a1b2c3d u-boot: fix compilation with newer GCC + 2ec283e base-files: update version + ... + + $ cd layers/openembedded-core + $ git checkout -b my-wip + Switched to a new branch 'my-wip' + $ git fetch ../openembedded-core-backup.20260329160426 my-wip + + $ git cherry-pick 2ec283e + # resolve any conflicts, then stage the resolved files: + $ git add meta/recipes-core/base-files/base-files.bb + $ git cherry-pick --continue + + $ git cherry-pick a1b2c3d + # resolve any conflicts, then stage the resolved files: + $ git add meta/recipes-devtools/u-boot/u-boot_2026.04.bb + $ git cherry-pick --continue + + $ cd ../.. + + The sequence above is: + + #. Inspect the backup's branch history to identify the commits to recover. + #. Change into the fresh clone and create a matching branch. + #. Fetch the backup branch so its objects become available locally. Git + accepts filesystem paths as remote URLs, and from inside + ``layers/openembedded-core/``, ``../`` points to ``layers/``, where the + backup directory sits. + #. Cherry-pick the commits in oldest-first order. + + Once all desired commits have been recovered and verified, the backup + directory can be removed: + + .. code-block:: shell + + $ rm -rf layers/openembedded-core-backup.20260329160426 + + If a VSCode workspace is in use, the backup directory will appear as an + additional workspace folder until it is cleaned up. It can be removed from + the workspace via the VS Code UI by right-clicking the folder and selecting + *Remove Folder from Workspace*. + +.. _ref-bbsetup-command-install-buildtools: + +``bitbake-setup install-buildtools`` +------------------------------------ + +The ``bitbake-setup install-buildtools`` sub-command installs and extracts a +buildtools tarball into the specified :term:`Setup`. + +After completion, help is printed to help the user on how to use the extracted +tarball. + +.. note:: + + The purpose of the Buildtools tarball is to provide tools needed to run + BitBake on build machines where such tools cannot be easily obtained from the + host Linux distribution (for example on older distribution versions that do + not contain a recent enough GCC compiler or Python interpreter, or machines + where the user running BitBake cannot easily install additional software into + the system). This command requires that the OpenEmbedded-core layer is + present in the BitBake configuration. + + See https://docs.yoctoproject.org/ref-manual/system-requirements.html#required-git-tar-python-make-and-gcc-versions + for more information. + +In addition, the command can take the following arguments: + +- ``--force``: force the re-installation of the tarball. + +- ``--setup-dir``: path to the :term:`Setup` to check to status for. Not + required if :term:`BBPATH` is already configured. + +.. _ref-bbsetup-command-settings: + +``bitbake-setup settings`` +-------------------------- + +The ``bitbake-setup settings`` sub-command helps modifying the settings of +``bitbake-setup``. This sub-command has sub-commands itself: + +- ``bitbake-setup settings list`` lists the current settings and their value. +- ``bitbake-setup settings set`` sets a setting. +- ``bitbake-setup settings unset`` removes a setting. + +Settings must be set with a section and a value, for example:: + + bitbake-setup setting set default top-dir-name bitbake-builds + +Will set the value of ``top-dir-name`` in the ``default`` section to +"bitbake-builds". + +In addition, the command can take the following arguments: + +- ``--global``: write to the global settings + (``~/.config/bitbake-setup/settings.conf``) instead of the :term:`Top + Directory` settings. + +See the :ref:`bitbake-user-manual/bitbake-user-manual-environment-setup:Settings` +section to see the supported settings. + +.. note:: + + The supported setting listed in the + :ref:`bitbake-user-manual/bitbake-user-manual-environment-setup:Settings` + section are only affected when set in the ``default`` section. + +Settings +======== + +The settings allow configuring ``bitbake-setup``. Settings are stored in a file +named ``settings.conf``, in :wikipedia:`INI ` format. + +There are multiple locations for storing settings. Settings in different +locations can override each other, but the final value of a setting is computed +from reading the files in this order: + +#. Global settings file: ``~/.config/bitbake-setup/settings.conf``. + +#. Local settings file, taken from a ``settings.conf`` file in the :term:`Top + Directory`. + +#. Command-line settings, passed with the ``--setting`` argument. + +A valid settings file would for example be: + +.. code-block:: ini + + [default] + top-dir-prefix = /path/to/workspace + top-dir-name = bitbake-builds + registry = /path/to/bitbake/default-registry + dl-dir = /path/to/bitbake-setup-downloads + use-full-setup-dir-name = yes + common-sstate = yes + +Settings and their values can be listed and modified with the ``bitbake-setup +settings`` command. See the :ref:`ref-bbsetup-command-settings` section for +more information. + +Below are the available settings. + +.. _ref-bbsetup-setting-top-dir-prefix: + +``top-dir-prefix`` +------------------ + +The :ref:`ref-bbsetup-setting-top-dir-prefix` setting helps configuring the +leftmost part of the path to the :term:`Top Directory`. + +For example, with: + +.. code-block:: ini + + [default] + top-dir-prefix = /path/to/workspace + +The :term:`top directory` would be ``/path/to/workspace/`` with +the ```` corresponding to the :ref:`ref-bbsetup-setting-top-dir-name` +setting. + +This is most useful to customize on systems where the default location of the +:term:`Top Directory` (``~/bitbake-builds``) is not suitable, and there is a +dedicated directory for builds somewhere else. + +.. _ref-bbsetup-setting-top-dir-name: + +``top-dir-name`` +---------------- + +The :ref:`ref-bbsetup-setting-top-dir-name` setting helps configuring the +rightmost part of the path to the :term:`Top Directory`. + +For example, with: + +.. code-block:: ini + + [default] + top-dir-name = builds + +The :term:`top directory` would be ``/builds`` with +the ```` corresponding to the :ref:`ref-bbsetup-setting-top-dir-prefix` +setting. + +.. _ref-bbsetup-setting-registry: + +``registry`` +------------ + +The :ref:`ref-bbsetup-setting-registry` setting sets the URI location of the +registry. This URI can be any URI supported by the BitBake fetcher. + +A local registry would be configured as follows: + +.. code-block:: ini + + [default] + registry = /path/to/registry + +When using another fetcher, it must be specified in the URI scheme. For example: + +.. code-block:: ini + + [default] + registry = git://example.com/bitbake-setup-configurations;protocol=https;branch=master;rev=master + +This would fetch the remote configurations from a remote Git remote repository, +on the ``master`` branch. + +See the :doc:`/bitbake-user-manual/bitbake-user-manual-fetching` section for more +information on BitBake fetchers. + +.. _ref-bbsetup-setting-dl-dir: + +``dl-dir`` +---------- + +The :ref:`ref-bbsetup-setting-dl-dir` setting sets the location of the download +cache that ``bitbake-setup`` will configure for the purpose of downloading +configuration repositories, layers and other sources using BitBake fetchers. +Please see :doc:`/bitbake-user-manual/bitbake-user-manual-fetching` and the +:term:`DL_DIR` variable for more information. + +The location can be set such that it is shared with :term:`DL_DIR` specified by +BitBake builds, so that there is a single directory containing a copy of +everything needed to set up and run a BitBake build offline in a reproducible +manner. + +.. _ref-bbsetup-setting-use-full-setup-dir-name: + +``use-full-setup-dir-name`` +--------------------------- + +The :ref:`ref-bbsetup-setting-use-full-setup-dir-name` setting, if set to ``yes`` +will override the suggestions for the :term:`Setup` directory name made by +``setup-dir-name`` entries in :term:`Configuration Template` files. This +will make the directory names longer, but fully specific: they will contain +all selections made during initialization. + +.. _ref-bbsetup-setting-common-sstate: + +``common-sstate`` +----------------- + +When this setting is set to ``yes`` (which is also the default), bitbake-setup will +set up a common sstate directory and common hash equivalency database for all the +:term:`setups ` in a :term:`Top Directory`. This is very beneficial for speeding +up builds as build artefacts will be reused whenever possible between them. + +Set this to ``no`` for advanced use cases, such as placing the sstate directory on a NFS +mount and maintaining a separate hash equivalency server, so that sstate and hash equivalency +data can be shared between several computers. For such use cases the sstate settings need +to be added to a build configuration separately. + +See https://docs.yoctoproject.org/dev-manual/hashequivserver.html for how to share sstate +on the network. + +.. _ref-bbsetup-section-config-reference: + +Configuration Template Files Reference +====================================== + +:term:`Configuration Templates ` are the input files given +to ``bitbake-setup`` to configure :term:`Setups `. + +These files are written in the JSON file format and are stored in a +:term:`Registry`. They can also be standalone files directly passed to the +:ref:`ref-bbsetup-command-init` command: + +.. code-block:: shell + + $ bitbake-setup init /path/to/config.conf.json + +They contain the following sections: + +- ``version`` (**required**): version of the configuration file. + + Example: + + .. code-block:: json + :force: + + { + "version": "1.0" + } + +- ``description`` (**required**): the description of the configuration. + + Example: + + .. code-block:: json + :force: + + { + "description": "OpenEmbedded - 'nodistro' basic configuration" + } + +- ``sources`` (**required**): sources, such as git repositories that should + be provided under ``layers/`` directory of a :term:`Setup`. Although the + list of sources can be very flexible, at least BitBake should be cloned + to generate a :term:`Setup`. + + Example: + + .. code-block:: json + :force: + + { + "sources": { + "bitbake": { + "git-remote": { + "uri": "https://git.openembedded.org/bitbake", + "branch": "master", + "rev": "master" + }, + "path": "bitbake" + }, + "openembedded-core": { + "local": { + "path": "~/path/to/local/openembedded-core" + } + }, + "meta-private": { + "git-remote": { + "remotes": { + "origin": { + "uri": "ssh://git@git.example.com/meta-private" + } + }, + "branch": "master", + "rev": "master" + }, + "path": "meta-private" + } + } + } + + Sources can be specified with the following options: + + - ``path`` (*optional*): where the source is extracted, relative to the + ``layers/`` directory of a :term:`Setup`. If unspecified, the name of the + source is used. + + - ``git-remote`` (*optional*): specifies URI, branch and revision of a git + repository to fetch from. + + ``git-remote`` entries are specified with the following options: + + - ``remotes`` (*optional*): a dictionary of git remote names, each containing a ``uri`` property + with a URI that follows the git URI syntax. See https://git-scm.com/docs/git-clone#_git_urls + and https://git-scm.com/docs/git-remote for more information. + + This structure is useful if you want to provide several remotes. Please + note that each URI will be cloned, but only the the last treated one + will be used in the unpacked source. + + Example: + + .. code-block:: json + :force: + + { + "sources": { + "bitbake": { + "git-remote": { + "remotes": { + "origin": { + "uri": "https://git.openembedded.org/bitbake" + }, + "contrib": { + "uri": "https://git.openembedded.org/bitbake-contrib" + } + }, + "branch": "master", + "rev": "master" + }, + "path": "bitbake" + } + } + } + - ``rev`` (**required**): the revision to checkout. Can be the name of the + branch to checkout on the latest revision of the specified ``branch``. + + If the value is the branch name, ``bitbake-setup`` will check out the + latest revision on that branch, and keep it updated when using the + :ref:`ref-bbsetup-command-update` command. + + - ``branch`` (**required**): the Git branch, used to check that the + specified ``rev`` is indeed on that branch. + + - ``uri`` (*optional*): a URI that follows the git URI syntax. You can replace + the ``remotes`` structure if only one URI is provided. Despite this, ``uri`` + and ``remotes`` can still be used together. + + See https://git-scm.com/docs/git-clone#_git_urls for more information. + + - ``local`` (*optional*): specifies a path on local disk that should be symlinked + to under ``layers/``. This is useful for local development, where some layer + or other component used in a build is managed separately, but should still be + available for bitbake-setup driven builds. + + ``local`` entries are specified with the following options: + + - ``path`` (**required**): the path on local disk where the externally + managed source tree is. ``~`` and ``~user`` are expanded to that user's home + directory. Paths in configuration files must be absolute (after possible + ~ expansion), paths in override files can be relative to the directory where + the override file is. + +- ``expires`` (*optional*): Expiration date of the configuration. This date + should be in :wikipedia:`ISO 8601 ` format (``YYYY-MM-DDTHH:MM:SS``). + +- ``bitbake-setup`` (**required**): contains a list of configurations. + + Example: + + .. code-block:: json + + { + "bitbake-setup": { + "configurations": [ + { + "bb-layers": ["openembedded-core/meta","meta-yocto/meta-yocto-bsp","meta-yocto/meta-poky"], + "bb-env-passthrough-additions": ["DL_DIR","SSTATE_DIR"], + "oe-fragments-one-of": { + "machine": { + "description": "Target machines", + "options" : [ + { "name": "machine/qemux86-64", "description": "Machine configuration for running an x86-64 system on QEMU" }, + { "name": "machine/qemuarm64", "description": "Machine configuration for running an ARMv8 system on QEMU" }, + { "name": "machine/qemuriscv64", "description": "Machine configuration for running an RISC-V system on QEMU" }, + { "name": "machine/genericarm64", "description": "Machine configuration for Arm64 SystemReady IR/ES platforms" }, + { "name": "machine/genericx86-64", "description": "Machine configuration for generic x86_64 (64-bit) PCs and servers" } + ] + }, + "distro": { + "description": "Target Distributions", + "options" : [ + { "name": "distro/poky", "description": "Distro configuration for Poky (Yocto Project Reference Distro)" }, + { "name": "distro/poky-altcfg", "description": "Distro configuration for Poky (systemd init manager)" }, + { "name": "distro/poky-tiny", "description": "Distro configuration for Poky (optimized for size)" } + ] + } + }, + "configurations": [ + { + "name": "poky", + "description": "Poky - The Yocto Project testing distribution" + }, + { + "name": "poky-with-sstate", + "description": "Poky - The Yocto Project testing distribution with internet sstate acceleration. Use with caution as it requires a completely robust local network with sufficient bandwidth.", + "oe-fragments": ["core/yocto/sstate-mirror-cdn"] + } + ] + } + ] + } + } + + Configurations can be specified with the following options: + + - ``name`` (**required**): the name of this configuration snippet. This is + what is prompted during the :ref:`ref-bbsetup-command-init` command + execution. + + - ``description`` (**required**): the description of this configuration + snippet. This is what is prompted during the + :ref:`ref-bbsetup-command-init` command execution. + + - ``configurations``: Configurations can recursively contain as many nested + configurations as needed. This will create more choices when running the + :ref:`ref-bbsetup-command-init` command. + + The purpose of such nesting is to be able to scale the configurations, for + example when there is a need to create multiple configurations that share + some parameters (which are specified in their common parent), but differ + between themselves in other parameters. ``bitbake-setup`` will assemble + configuration choices by putting together information from a leaf + configuration and all of its ancestors. + + - ``bb-env-passthrough-additions`` (*optional*): List of environment + variables to include in :term:`BB_ENV_PASSTHROUGH_ADDITIONS`. + + - ``bb-layers`` (*optional*): List of layers to add to the ``bblayers.conf`` + file. Paths in this list are relative to the ``layers/`` directory of a + :term:`Setup`. + + The ``bb-layers`` keyword cannot be used in conjunction with the + ``oe-template`` option, as the ``bblayers.conf`` file comes from the + template itself. + + - ``bb-layers-file-relative`` (*optional*): List of layers that are not + managed by ``bitbake-setup`` but that need to be included as part of the + ``bblayers.conf`` file. Paths in this list are relative to the + configuration file. + + This is useful when (one or more) configuration files and (one or + more) layers are hosted in the same Git repository, which is cloned + and managed independently from bitbake-setup workflows. For example:: + + ├── meta-myproject/ + └── myproject.conf.json + + Then ``myproject.conf.json`` can contain the following to add + ``meta-myproject`` to ``bblayers.conf``:: + + { + ... + "bb-layers-file-relative": [ + "meta-myproject" + ], + ... + } + + The ``bb-layers-file-relative`` keyword cannot be used in conjunction with the + ``oe-template`` keyword, as the ``bblayers.conf`` file comes from the + template itself. + + - ``oe-template`` (*optional*, OpenEmbedded specific): OpenEmbedded template + to use. This cannot be used in conjunction with the + ``bb-layers`` or ``bb-layers-file-relative`` keywords as it + already provides a ready ``bblayers.conf`` file to use. + + See https://docs.yoctoproject.org/dev-manual/custom-template-configuration-directory.html + for more information of OpenEmbedded templates. + + - ``oe-fragments-one-of`` (*optional*, OpenEmbedded specific): the OpenEmbedded + fragments to select as part of the build. + + This can one of two formats: + + - A list of dictionaries with ``name`` and a ``description``, + corresponding to respectively the fragments names and fragment + descriptions (like in the example above). + + - A list of fragment names. For example: + + .. code-block:: json + + "options" : ["machine/qemux86-64", "machine/qemuarm64", "machine/qemuriscv64", "machine/genericarm64", "machine/genericx86-64"] + + This will trigger choices to make during the + :ref:`ref-bbsetup-command-init` command execution. + + See https://docs.yoctoproject.org/dev/ref-manual/fragments.html for + more information of OpenEmbedded configuration fragments. + + - ``oe-fragments`` (*optional*, OpenEmbedded specific): fragments to select + as part of the build. + + See https://docs.yoctoproject.org/dev/ref-manual/fragments.html for + more information of OpenEmbedded configuration fragments. + + - ``setup-dir-name`` (*optional*): a suggestion for the :term:`Setup` + directory name. Bitbake-setup will use it, unless it already exists, or + the :ref:`ref-bbsetup-setting-use-full-setup-dir-name` setting is set + to ``yes`` (in those cases, it will fall back to the built-in full name, + containing the full set of choices made during initialization). + + This key can have variable expansions in the same format as python + string.Template strings (which matches the shell variable expansion rules): + https://docs.python.org/3/library/string.html#template-strings-strings . + + Variables that can be expanded are any fragment configuration prompted + by the user (e.g. the keys in "oe-fragments-one-of"):: + + { + ... + "oe-fragments-one-of": { + "machine": { + "description": "Target machines", + "options" : ["machine/gadget", "machine/gizmo"] + } + }, + "setup-dir-name": "somebuild-$machine" + ... + } + + would expand to ``somebuild-machine_gadget``. + +Configuration Template Examples +------------------------------- + +OpenEmbedded "nodistro" configuration for master branches: + +.. literalinclude:: ../../default-registry/configurations/oe-nodistro-master.conf.json + :language: json + +Poky distribution configuration for master branches: + +.. literalinclude:: ../../default-registry/configurations/poky-master.conf.json + :language: json + +.. _ref-bbsetup-source-overrides: + +Source Overrides +================ + +See the definition of :term:`Source Override` in the Terminology section. + +These files are written in the JSON file format and are optionally passed to the +``--source-overrides`` argument of the :ref:`ref-bbsetup-command-init` command. +The ``--source-overrides`` option can be passed multiple times, in which case the +overrides are applied in the order specified in the command-line. + +Here is an example file that overrides the branch of the BitBake repository to +"master-next", and provides openembedded-core as a symlink to a path on local disk: + +.. code-block:: json + + { + "description": "Source override file", + "sources": { + "bitbake": { + "git-remote": { + "branch": "master-next", + "remotes": { + "origin": { + "uri": "https://git.openembedded.org/bitbake" + } + }, + "rev": "master-next" + } + }, + "openembedded-core": { + "local": { + "path": "~/path/to/local/openembedded-core" + } + } + }, + "version": "1.0" + } + +- The ``version`` parameter contains the version of the used configuration, and + should match the one of the :term:`Configuration Template` file in use. + +- The ``sources`` section contains the same options as the ``sources`` option + of a :term:`Configuration Template` file. See the + :ref:`ref-bbsetup-section-config-reference` section for more information. + +.. _ref-bbsetup-fixed-revisions: + +Fixed source revisions +====================== + +:term:`Configuration Template` can set source revisions in ``rev`` to a tag or a branch. +Bitbake-setup will make sure the actual revision will match the tag or branch when performing +initializations or updates, and will capture the revisions in a :ref:`ref-bbsetup-source-overrides` +file. + +This file is named ``sources-fixed-revisions.json`` and is available in ``layers/`` +under the :term:`Setup` directory (as well as under ``config/`` for backwards compatibility). +It can be used to keep record of what was checked out when using configurations that do not +specify exact revisions. + +Also, as it is an override file, it can be combined with the original configurations +to initialize a setup in a reproducible way that guarantees an exact, never-changing +set of revisions, by using the ``--source-overrides`` option of the +:ref:`ref-bbsetup-command-init` command. + +For example if the original configuration had specified only a master branch +for a source:: + + "bitbake": { + "git-remote": { + "remotes": { + "origin": { + "uri": "https://git.openembedded.org/bitbake" + } + }, + "branch": "master", + "rev": "master" + } + + +the override will contain the exact revisions:: + + "bitbake": { + "git-remote": { + "remotes": { + "origin": { + "uri": "https://git.openembedded.org/bitbake" + } + }, + "branch": "master", + "rev": "720df1a53452983c1c832f624490e255cf389204" + } diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst index d58fbb32..1638a8f5 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst @@ -64,11 +64,11 @@ data itself is of various types: together. The ``layer.conf`` files are used to construct key variables such as -:term:`BBPATH` and :term:`BBFILES`. -:term:`BBPATH` is used to search for configuration and class files under the -``conf`` and ``classes`` directories, respectively. :term:`BBFILES` is used -to locate both recipe and recipe append files (``.bb`` and -``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the +:term:`BBPATH` and :term:`BBFILES`. :term:`BBPATH` is used to search for +configuration files under the ``conf`` directory and class files under the +``classes-global``, ``classes-recipe`` and ``classes`` directories. +:term:`BBFILES` is used to locate both recipe and recipe append files (``.bb`` +and ``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the user has set the :term:`BBPATH` and :term:`BBFILES` directly in the environment. Next, the ``bitbake.conf`` file is located using the :term:`BBPATH` variable @@ -337,21 +337,22 @@ file that BitBake parses, you can change that preference:: .. note:: - It is common for a recipe to provide two versions -- a stable, - numbered (and preferred) version, and a version that is automatically - checked out from a source code repository that is considered more - "bleeding edge" but can be selected only explicitly. + It is not uncommon for a given recipe to provide more than one + version -- a stable (and preferred) version that should be selected + by default, and a newer, development or "bleeding edge" version that + will be selected only explicitly by the developer. - For example, in the OpenEmbedded codebase, there is a standard, - versioned recipe file for BusyBox, ``busybox_1.22.1.bb``, but there - is also a Git-based version, ``busybox_git.bb``, which explicitly - contains the line :: + For example, at the moment, in the OpenEmbedded codebase, there are + two versions of the ``nginx`` webserver: a stable "1.28.0" version + that would be selected by default, and a newer "1.29.1" development + version that a developer would need to choose explicitly. - DEFAULT_PREFERENCE = "-1" + This can be seen based on this snippet from the ``nginx`` 1.29.1 + recipe file:: - to ensure that the - numbered, stable version is always preferred unless the developer - selects otherwise. + # 1.28.x branch is the current stable branch, the recommended default + # 1.29.x is the current mainline branches containing all new features + DEFAULT_PREFERENCE = "-1" .. _bb-bitbake-dependencies: diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst index fb4f0a23..c2747c40 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst @@ -39,10 +39,10 @@ variable and then calls the ``download`` method to download the files. The instantiation of the fetch class is usually followed by:: - rootdir = l.getVar('WORKDIR') + rootdir = l.getVar('UNPACKDIR') fetcher.unpack(rootdir) -This code unpacks the downloaded files to the specified by ``WORKDIR``. +This code unpacks the downloaded files to the specified by ``UNPACKDIR``. .. note:: @@ -51,7 +51,7 @@ This code unpacks the downloaded files to the specified by ``WORKDIR``. examine the OpenEmbedded class file ``base.bbclass`` . -The :term:`SRC_URI` and ``WORKDIR`` variables are not hardcoded into the +The :term:`SRC_URI` and ``UNPACKDIR`` variables are not hardcoded into the fetcher, since those fetcher methods can be (and are) called with different variable names. In OpenEmbedded for example, the shared state (sstate) code uses the fetch module to fetch the sstate files. @@ -84,11 +84,8 @@ fetcher does know how to use HTTP as a transport. Here are some examples that show commonly used mirror definitions:: PREMIRRORS ?= "\ - bzr://.*/.\* http://somemirror.org/sources/ \ - cvs://.*/.\* http://somemirror.org/sources/ \ git://.*/.\* http://somemirror.org/sources/ \ hg://.*/.\* http://somemirror.org/sources/ \ - osc://.*/.\* http://somemirror.org/sources/ \ p4://.*/.\* http://somemirror.org/sources/ \ svn://.*/.\* http://somemirror.org/sources/" @@ -159,19 +156,25 @@ URLs except Git URLs, BitBake uses the common ``unpack`` method. A number of parameters exist that you can specify within the URL to govern the behavior of the unpack stage: -- *unpack:* Controls whether the URL components are unpacked. If set to +- *"unpack":* Controls whether the URL components are unpacked. If set to "1", which is the default, the components are unpacked. If set to "0", the unpack stage leaves the file alone. This parameter is useful when you want an archive to be copied in and not be unpacked. -- *dos:* Applies to ``.zip`` and ``.jar`` files and specifies whether +- *"dos":* Applies to ``.zip`` and ``.jar`` files and specifies whether to use DOS line ending conversion on text files. -- *striplevel:* Strip specified number of leading components (levels) - from file names on extraction +- *"striplevel":* Strip specified number of leading components (levels) + from file names on extraction. -- *subdir:* Unpacks the specific URL to the specified subdirectory - within the root directory. +- *"subdir":* Unpacks the specific URL to the specified subdirectory + within the specified root directory. This path can be further modified + by fetcher specific parameters. + +- *"name":* Assigns a name to a given component of the :term:`SRC_URI`. + This component is later referenced by this name when specifying its + :term:`SRCREV` or :term:`SRC_URI` checksum, or to correctly place its + revision in the package version string with aid of :term:`SRCREV_FORMAT`. The unpack call automatically decompresses and extracts files with ".Z", ".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". ".srpm", ".deb" and @@ -183,6 +186,51 @@ cloning the tree into the final directory. The process is completed using references so that there is only one central copy of the Git metadata needed. +.. _bb-the-unpack-update: + +The Non-Destructive Update (``unpack_update``) +============================================== + +.. note:: + + This is a specialised method intended for tools that manage persistent + layer checkouts on behalf of the user, such as + :ref:`bitbake-setup update `. It is not part of the + normal recipe fetch/unpack flow. + +For Git URLs, an alternative ``unpack_update()`` method is available +that updates an existing checkout *in place* rather than removing and +re-cloning it. This is useful when the target directory may contain +local commits that should be preserved across updates. + +The code to call the non-destructive update looks like the following:: + + rootdir = l.getVar('UNPACKDIR') + fetcher.unpack_update(rootdir) + +``unpack_update()`` performs the following steps: + +1. A ``dldir`` git remote is added (or updated) in the existing + checkout, pointing at the local download cache. This remote may + also be useful for manually resolving conflicts outside the fetcher. +2. The new upstream revision is fetched from the ``dldir`` remote into + the existing repository. +3. Any local commits are rebased on top of the new upstream revision, + preserving local work. + +The method raises an error if: + +- The working tree contains staged or unstaged changes to tracked + files (``LocalModificationsError``). +- Local commits cannot be cleanly rebased onto the new upstream + revision (``RebaseError``). A failed rebase is automatically aborted + before the exception is raised. +- The download cache does not contain a sufficiently recent clone + of the repository, or the checkout is a shallow clone. + +Currently only the Git fetcher supports ``unpack_update()``. All other +fetcher types raise ``RuntimeError`` if it is called. + .. _bb-fetchers: Fetchers @@ -246,91 +294,19 @@ Some example URLs are as follows:: introduce ambiguity when parsing URLs that also contain semi-colons, for example:: - SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47" - + SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47" Such URLs should should be modified by replacing semi-colons with '&' characters:: - SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47" - + SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47" In most cases this should work. Treating semi-colons and '&' in queries identically is recommended by the World Wide Web Consortium (W3C). Note that due to the nature of the URL, you may have to specify the name of the downloaded file as well:: - SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2" - - -.. _cvs-fetcher: - -CVS fetcher (``(cvs://``) -------------------------- - -This submodule handles checking out files from the CVS version control -system. You can configure it using a number of different variables: - -- :term:`FETCHCMD_cvs `: The name of the executable to use when running - the ``cvs`` command. This name is usually "cvs". - -- :term:`SRCDATE`: The date to use when fetching the CVS source code. A - special value of "now" causes the checkout to be updated on every - build. - -- :term:`CVSDIR`: Specifies where a temporary - checkout is saved. The location is often ``DL_DIR/cvs``. - -- CVS_PROXY_HOST: The name to use as a "proxy=" parameter to the - ``cvs`` command. - -- CVS_PROXY_PORT: The port number to use as a "proxyport=" - parameter to the ``cvs`` command. - -As well as the standard username and password URL syntax, you can also -configure the fetcher with various URL parameters: - -The supported parameters are as follows: - -- *"method":* The protocol over which to communicate with the CVS - server. By default, this protocol is "pserver". If "method" is set to - "ext", BitBake examines the "rsh" parameter and sets ``CVS_RSH``. You - can use "dir" for local directories. - -- *"module":* Specifies the module to check out. You must supply this - parameter. - -- *"tag":* Describes which CVS TAG should be used for the checkout. By - default, the TAG is empty. - -- *"date":* Specifies a date. If no "date" is specified, the - :term:`SRCDATE` of the configuration is used to - checkout a specific date. The special value of "now" causes the - checkout to be updated on every build. - -- *"localdir":* Used to rename the module. Effectively, you are - renaming the output directory to which the module is unpacked. You - are forcing the module into a special directory relative to - :term:`CVSDIR`. - -- *"rsh":* Used in conjunction with the "method" parameter. - -- *"scmdata":* Causes the CVS metadata to be maintained in the tarball - the fetcher creates when set to "keep". The tarball is expanded into - the work directory. By default, the CVS metadata is removed. - -- *"fullpath":* Controls whether the resulting checkout is at the - module level, which is the default, or is at deeper paths. - -- *"norecurse":* Causes the fetcher to only checkout the specified - directory with no recurse into any subdirectories. - -- *"port":* The port to which the CVS server connects. - -Some example URLs are as follows:: - - SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext" - SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat" + SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2" .. _svn-fetcher: @@ -398,16 +374,16 @@ This fetcher supports the following parameters: .. note:: - When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` differs - from the one that is typically passed to ``git clone`` command and provided - by the Git server to fetch from. For example, the URL returned by GitLab - server for ``mesa`` when cloning over SSH is - ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the expected URL in - :term:`SRC_URI` is the following:: + When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` differs + from the one that is typically passed to ``git clone`` command and provided + by the Git server to fetch from. For example, the URL returned by GitLab + server for ``mesa`` when cloning over SSH is + ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the expected URL in + :term:`SRC_URI` is the following:: - SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." + SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." - Note the ``:`` character changed for a ``/`` before the path to the project. + Note the ``:`` character changed for a ``/`` before the path to the project. - *"nocheckout":* Tells the fetcher to not checkout source code when unpacking when set to "1". Set this option for the URL where there is @@ -436,13 +412,15 @@ This fetcher supports the following parameters: "nobranch" is set to "1", this is a mandatory parameter. The number of branch parameters must match the number of name parameters. -- *"rev":* The revision to use for the checkout. The default is - "master". +- *"rev":* The revision to use for the checkout. If :term:`SRCREV` is also set, + this parameter must match its value. -- *"tag":* Specifies a tag to use for the checkout. To correctly - resolve tags, BitBake must access the network. For that reason, tags - are often not used. As far as Git is concerned, the "tag" parameter - behaves effectively the same as the "rev" parameter. +- *"tag":* Specifies a tag to use when fetching. To correctly resolve + tags, BitBake must access the network. If a ``rev`` parameter or + :term:`SRCREV` is also specified, network access is not necessary to resolve + the tag and instead, it is verified that they both resolve to the same commit + SHA at unpack time. The ``tag`` parameter is optional, but strongly + recommended if the checked out revision is a tag. - *"subpath":* Limits the checkout to a specific subpath of the tree. By default, the whole tree is checked out. @@ -451,7 +429,7 @@ This fetcher supports the following parameters: By default, the path is ``git/``. - *"usehead":* Enables local ``git://`` URLs to use the current branch - HEAD as the revision for use with ``AUTOREV``. The "usehead" + HEAD as the revision for use with :term:`AUTOREV`. The "usehead" parameter implies no branch and only works when the transfer protocol is ``file://``. @@ -461,13 +439,6 @@ Here are some example URLs:: SRC_URI = "git://github.com/asciidoc/asciidoc-py;protocol=https;branch=main" SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." -.. note:: - - When using ``git`` as the fetcher of the main source code of your software, - ``S`` should be set accordingly:: - - S = "${WORKDIR}/git" - .. note:: Specifying passwords directly in ``git://`` urls is not supported. @@ -476,7 +447,7 @@ Here are some example URLs:: easy to share metadata without removing passwords. SSH keys, ``~/.netrc`` and ``~/.ssh/config`` files can be used as alternatives. -Using tags with the git fetcher may cause surprising behaviour. Bitbake needs to +Using tags with the git fetcher may cause surprising behaviour. BitBake needs to resolve the tag to a specific revision and to do that, it has to connect to and use the upstream repository. This is because the revision the tags point at can change and we've seen cases of this happening in well known public repositories. This can mean @@ -530,10 +501,10 @@ The fetcher uses the ``rcleartool`` or Following are options for the :term:`SRC_URI` statement: -- *vob*: The name, which must include the prepending "/" character, +- *"vob":* The name, which must include the prepending "/" character, of the ClearCase VOB. This option is required. -- *module*: The module, which must include the prepending "/" +- *"module":* The module, which must include the prepending "/" character, in the selected VOB. .. note:: @@ -545,7 +516,7 @@ Following are options for the :term:`SRC_URI` statement: load /example_vob/example_module -- *proto*: The protocol, which can be either ``http`` or ``https``. +- *"proto":* The protocol, which can be either ``http`` or ``https``. By default, the fetcher creates a configuration specification. If you want this specification written to an area other than the default, use @@ -554,9 +525,9 @@ the specification is written. .. note:: - the SRCREV loses its functionality if you specify this variable. However, - SRCREV is still used to label the archive after a fetch even though it does - not define what is fetched. + the :term:`SRCREV` loses its functionality if you specify this variable. + However, :term:`SRCREV` is still used to label the archive after a fetch even + though it does not define what is fetched. Here are a couple of other behaviors worth mentioning: @@ -598,7 +569,7 @@ and port, username, and password, and fetches the Head Revision:: SRC_URI = "p4://example-depot/main/source/..." SRCREV = "${AUTOREV}" PV = "p4-${SRCPV}" - S = "${WORKDIR}/p4" + S = "${UNPACKDIR}/p4" Here is an example that specifies the server URL and port, username, and password, and fetches a Revision based on a Label:: @@ -607,25 +578,25 @@ password, and fetches a Revision based on a Label:: SRC_URI = "p4://user:passwd@example-depot/main/source/..." SRCREV = "release-1.0" PV = "p4-${SRCPV}" - S = "${WORKDIR}/p4" + S = "${UNPACKDIR}/p4" .. note:: - You should always set S to "${WORKDIR}/p4" in your recipe. + You should always set S to "${UNPACKDIR}/p4" in your recipe. By default, the fetcher strips the depot location from the local file paths. In the above example, the content of ``example-depot/main/source/`` will be placed -in ``${WORKDIR}/p4``. For situations where preserving parts of the remote depot +in ``${UNPACKDIR}/p4``. For situations where preserving parts of the remote depot paths locally is desirable, the fetcher supports two parameters: -- *"module":* - The top-level depot location or directory to fetch. The value of this - parameter can also point to a single file within the depot, in which case - the local file path will include the module path. -- *"remotepath":* - When used with the value "``keep``", the fetcher will mirror the full depot - paths locally for the specified location, even in combination with the - ``module`` parameter. +- *"module":* + The top-level depot location or directory to fetch. The value of this + parameter can also point to a single file within the depot, in which case + the local file path will include the module path. +- *"remotepath":* + When used with the value "``keep``", the fetcher will mirror the full depot + paths locally for the specified location, even in combination with the + ``module`` parameter. Here is an example use of the the ``module`` parameter:: @@ -686,9 +657,9 @@ Such functionality is set by the variable: delegate access to resources, if this variable is set, the Az Fetcher will use it when fetching artifacts from the cloud. -You can specify the AZ_SAS variable as shown below:: +You can specify the AZ_SAS variable prefixed with a ? as shown below:: - AZ_SAS = "se=2021-01-01&sp=r&sv=2018-11-09&sr=c&skoid=&sig=" + AZ_SAS = "?se=2021-01-01&sp=r&sv=2018-11-09&sr=c&skoid=&sig=" Here is an example URL:: @@ -754,6 +725,12 @@ Here is an example URL:: NPM Fetcher (``npm://``) ------------------------ +.. warning:: + + The NPM fetcher is currently disabled due to security concerns. See + `355cd226e0720a9ed7683bb01c8c0a58eee03664 `__ + for more information. + This submodule fetches source code from an `NPM `__ Javascript package registry. @@ -793,6 +770,12 @@ to automatically create a recipe from an NPM URL. NPM shrinkwrap Fetcher (``npmsw://``) ------------------------------------- +.. warning:: + + The NPM fetcher is currently disabled due to security concerns. See + `355cd226e0720a9ed7683bb01c8c0a58eee03664 `__ + for more information. + This submodule fetches source code from an `NPM shrinkwrap `__ description file, which lists the dependencies @@ -828,12 +811,8 @@ Other Fetchers Fetch submodules also exist for the following: -- Bazaar (``bzr://``) - - Mercurial (``hg://``) -- OSC (``osc://``) - - S3 (``s3://``) - Secure FTP (``sftp://``) @@ -848,4 +827,42 @@ submodules. However, you might find the code helpful and readable. Auto Revisions ============== -We need to document ``AUTOREV`` and :term:`SRCREV_FORMAT` here. +For recipes which need to use the latest revision of their source code, +the way to achieve it is to use :term:`AUTOREV` as the value of the +source code repository's :term:`SRCREV`:: + + SRCREV = "${AUTOREV}" + +.. note:: + + With :term:`AUTOREV`, BitBake will always need to take the additional step of + querying the remote repository to retrieve the latest available revision. + + Also, recipes using it are not part of the parsing-time cache, + and hence are parsed every time. + +Multiple Source Control Repositories +==================================== + +For some recipes it is necessary to make use of more than one +version controlled source code repository. In such case, the recipe +must provide BitBake with information about how it should include +the different SCM revisions in its package version string, instead of its +usual approach with a single :term:`SRCREV`. + +For this purpose, the recipe must set the :term:`SRCREV_FORMAT` +variable. Consider the following example:: + + SRC_URI = " \ + git://git.some.example.com/source-tree.git;name=machine \ + git://git.some.example.com/metadata.git;name=meta \ + " + SRCREV_machine = "3f9db490a81eeb0077be3c5a5aa1388a2372232f" + SRCREV_meta = "1ac1d0ff730fe1dd1371823d562db8126750a98c" + SRCREV_FORMAT ?= "meta_machine" + +The value given to :term:`SRCREV_FORMAT` references names, which were +assigned using the ``name`` parameter in the :term:`SRC_URI` definition, +and which represent the version controlled source code repositories. +In the above example, the :term:`SRC_URI` contained two URLs named +"meta" and "machine". diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst index 35ffb88b..6801323e 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst @@ -198,7 +198,7 @@ Class files, which are denoted by the ``.bbclass`` extension, contain information that is useful to share between metadata files. The BitBake source tree currently comes with one class metadata file called ``base.bbclass``. You can find this file in the ``classes`` directory. -The ``base.bbclass`` class files is special since it is always included +The ``base.bbclass`` class file is special since it is always included automatically for all recipes and classes. This class contains definitions for standard basic tasks such as fetching, unpacking, configuring (empty by default), compiling (runs any Makefile present), @@ -206,6 +206,35 @@ installing (empty by default) and packaging (empty by default). These tasks are often overridden or extended by other classes added during the project development process. +Class Types +~~~~~~~~~~~ + +BitBake supports class files installed in three different directories: + +- ``classes-global/``: these classes must be inherited globally through the + :term:`INHERIT` variable in a :ref:`configuration file + `. These + classes are included for every recipe being built. For example, you would use + the global class named ``myclass`` like so:: + + INHERIT += "myclass" + +- ``classes-recipe/``: these classes must be inherited from a recipe using the + :ref:`inherit ` or + :ref:`inherit_defer ` + directive. They do not support being inherited globally. For example, you + would use the recipe class named ``myclass`` like so:: + + inherit myclass + +- ``classes/``: this final directory is meant for classes that can be used in + the two contexts explained above. In other words, they can be inherited either + globally or in a recipe. + +For details on how BitBake locates class files, see the +:ref:`bitbake-user-manual/bitbake-user-manual-metadata:Locating Class Files` +section of the BitBake User Manual. + Layers ------ @@ -242,12 +271,9 @@ same root filename. The filenames can differ only in the file type suffix used (e.g. ``formfactor_0.0.bb`` and ``formfactor_0.0.bbappend``). -Information in append files extends or overrides the information in the -underlying, similarly-named recipe files. - When you name an append file, you can use the "``%``" wildcard character -to allow for matching recipe names. For example, suppose you have an -append file named as follows:: +to allow for a less strict recipe name matching. For example, suppose you have +an append file named as follows:: busybox_1.21.%.bbappend @@ -258,12 +284,8 @@ the append file would match the following recipe names:: busybox_1.21.1.bb busybox_1.21.2.bb busybox_1.21.3.bb - -.. note:: - - The use of the " % " character is limited in that it only works directly in - front of the .bbappend portion of the append file's name. You cannot use the - wildcard character in any other location of the name. + busybox_1.21.10.bb + busybox_1.21.11.bb If the ``busybox`` recipe was updated to ``busybox_1.3.0.bb``, the append name would not match. However, if you named the append file @@ -272,6 +294,47 @@ append name would not match. However, if you named the append file In the most general case, you could name the append file something as simple as ``busybox_%.bbappend`` to be entirely version independent. +A recipe named without a version may only be appended by an append file named +exactly the same. That is, a recipe named:: + + mesa.bb + +can only be appended by an append file named:: + + mesa.bbappend + +That is, ``mesa_%.bbappend`` append file will not match a ``mesa.bb`` recipe. + +.. note:: + + The "``%``" character is a marker for BitBake, and everything after that + marker will not be used for matching recipe names. That is, an append file + named:: + + busybox_1.%.11.bbappend + + will match recipes named:: + + busybox_1.0.11.bb + busybox_1.2.bb + busybox_1.3.48.bb + + Note that the "``%``" character is not specific to the version part of the + recipe name. That is, an append file named:: + + linux-yocto%.bbappend + + will match recipes named:: + + linux-yocto.bb + linux-yocto_6.18.bb + linux-yocto-tiny_6.12.bb + linux-yocto-rt_6.16.bb + + While it can be useful to append to similar recipes with slightly different + names, it is highly discouraged to do so as recipes from other layers may be + named similarly enough that the append file would unwittingly match them. + Obtaining BitBake ================= @@ -349,40 +412,84 @@ Usage and syntax Following is the usage and syntax for BitBake:: $ bitbake -h - Usage: bitbake [options] [recipename/target recipe:do_task ...] - - Executes the specified task (default is 'build') for a given set of target recipes (.bb files). - It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which - will provide the layer, BBFILES and other configuration information. + usage: bitbake [-s] [-e] [-g] [-u UI] [--version] [-h] [-f] [-c CMD] + [-C INVALIDATE_STAMP] [--runall RUNALL] [--runonly RUNONLY] + [--no-setscene] [--skip-setscene] [--setscene-only] [-n] [-p] + [-k] [-P] [-S SIGNATURE_HANDLER] [--revisions-changed] + [-b BUILDFILE] [-D] [-l DEBUG_DOMAINS] [-v] [-q] + [-w WRITEEVENTLOG] [-B BIND] [-T SERVER_TIMEOUT] + [--remote-server REMOTE_SERVER] [-m] [--token XMLRPCTOKEN] + [--observe-only] [--status-only] [--server-only] [-r PREFILE] + [-R POSTFILE] [-I EXTRA_ASSUME_PROVIDED] + [recipename/target ...] + + It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH + which will provide the layer, BBFILES and other configuration information. + + General options: + recipename/target Execute the specified task (default is 'build') for + these target recipes (.bb files). + -s, --show-versions Show current and preferred versions of all recipes. + -e, --environment Show the global or per-recipe environment complete + with information about where variables were + set/changed. + -g, --graphviz Save dependency tree information for the specified + targets in the dot syntax. + -u UI, --ui UI The user interface to use (knotty, ncurses, taskexp, + taskexp_ncurses or teamcity - default knotty). + --version Show programs version and exit. + -h, --help Show this help message and exit. - Options: - --version show program's version number and exit - -h, --help show this help message and exit - -b BUILDFILE, --buildfile=BUILDFILE - Execute tasks from a specific .bb recipe directly. - WARNING: Does not handle any dependencies from other - recipes. - -k, --continue Continue as much as possible after an error. While the - target that failed and anything depending on it cannot - be built, as much as possible will be built before - stopping. + Task control options: -f, --force Force the specified targets/task to run (invalidating any existing stamp file). - -c CMD, --cmd=CMD Specify the task to execute. The exact options + -c CMD, --cmd CMD Specify the task to execute. The exact options available depend on the metadata. Some examples might be 'compile' or 'populate_sysroot' or 'listtasks' may give a list of the tasks available. - -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP + -C INVALIDATE_STAMP, --clear-stamp INVALIDATE_STAMP Invalidate the stamp for the specified task such as 'compile' and then run the default task for the specified target(s). - -r PREFILE, --read=PREFILE - Read the specified file before bitbake.conf. - -R POSTFILE, --postread=POSTFILE - Read the specified file after bitbake.conf. - -v, --verbose Enable tracing of shell tasks (with 'set -x'). Also - print bb.note(...) messages to stdout (in addition to - writing them to ${T}/log.do_<task>). + --runall RUNALL Run the specified task for any recipe in the taskgraph + of the specified target (even if it wouldn't otherwise + have run). + --runonly RUNONLY Run only the specified task within the taskgraph of + the specified targets (and any task dependencies those + tasks may have). + --no-setscene Do not run any setscene tasks. sstate will be ignored + and everything needed, built. + --skip-setscene Skip setscene tasks if they would be executed. Tasks + previously restored from sstate will be kept, unlike + --no-setscene. + --setscene-only Only run setscene tasks, don't run any real tasks. + + Execution control options: + -n, --dry-run Don't execute, just go through the motions. + -p, --parse-only Quit after parsing the BB recipes. + -k, --continue Continue as much as possible after an error. While the + target that failed and anything depending on it cannot + be built, as much as possible will be built before + stopping. + -P, --profile Profile the command and save reports. + -S SIGNATURE_HANDLER, --dump-signatures SIGNATURE_HANDLER + Dump out the signature construction information, with + no task execution. The SIGNATURE_HANDLER parameter is + passed to the handler. Two common values are none and + printdiff but the handler may define more/less. none + means only dump the signature, printdiff means + recursively compare the dumped signature with the most + recent one in a local build or sstate cache (can be + used to find out why tasks re-run when that is not + expected) + --revisions-changed Set the exit code depending on whether upstream + floating revisions have changed or not. + -b BUILDFILE, --buildfile BUILDFILE + Execute tasks from a specific .bb recipe directly. + WARNING: Does not handle any dependencies from other + recipes. + + Logging/output control options: -D, --debug Increase the debug level. You can specify this more than once. -D sets the debug level to 1, where only bb.debug(1, ...) messages are printed to stdout; -DD @@ -392,65 +499,47 @@ Following is the usage and syntax for BitBake:: -D only affects output to stdout. All debug messages are written to ${T}/log.do_taskname, regardless of the debug level. + -l DEBUG_DOMAINS, --log-domains DEBUG_DOMAINS + Show debug logging for the specified logging domains. + -v, --verbose Enable tracing of shell tasks (with 'set -x'). Also + print bb.note(...) messages to stdout (in addition to + writing them to ${T}/log.do_). -q, --quiet Output less log message data to the terminal. You can specify this more than once. - -n, --dry-run Don't execute, just go through the motions. - -S SIGNATURE_HANDLER, --dump-signatures=SIGNATURE_HANDLER - Dump out the signature construction information, with - no task execution. The SIGNATURE_HANDLER parameter is - passed to the handler. Two common values are none and - printdiff but the handler may define more/less. none - means only dump the signature, printdiff means compare - the dumped signature with the cached one. - -p, --parse-only Quit after parsing the BB recipes. - -s, --show-versions Show current and preferred versions of all recipes. - -e, --environment Show the global or per-recipe environment complete - with information about where variables were - set/changed. - -g, --graphviz Save dependency tree information for the specified - targets in the dot syntax. - -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED - Assume these dependencies don't exist and are already - provided (equivalent to ASSUME_PROVIDED). Useful to - make dependency graphs more appealing - -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS - Show debug logging for the specified logging domains - -P, --profile Profile the command and save reports. - -u UI, --ui=UI The user interface to use (knotty, ncurses, taskexp or - teamcity - default knotty). - --token=XMLRPCTOKEN Specify the connection token to be used when - connecting to a remote server. - --revisions-changed Set the exit code depending on whether upstream - floating revisions have changed or not. - --server-only Run bitbake without a UI, only starting a server - (cooker) process. - -B BIND, --bind=BIND The name/address for the bitbake xmlrpc server to bind + -w WRITEEVENTLOG, --write-log WRITEEVENTLOG + Writes the event log of the build to a bitbake event + json file. Use '' (empty string) to assign the name + automatically. + + Server options: + -B BIND, --bind BIND The name/address for the bitbake xmlrpc server to bind to. - -T SERVER_TIMEOUT, --idle-timeout=SERVER_TIMEOUT + -T SERVER_TIMEOUT, --idle-timeout SERVER_TIMEOUT Set timeout to unload bitbake server due to inactivity, set to -1 means no unload, default: Environment variable BB_SERVER_TIMEOUT. - --no-setscene Do not run any setscene tasks. sstate will be ignored - and everything needed, built. - --skip-setscene Skip setscene tasks if they would be executed. Tasks - previously restored from sstate will be kept, unlike - --no-setscene - --setscene-only Only run setscene tasks, don't run any real tasks. - --remote-server=REMOTE_SERVER + --remote-server REMOTE_SERVER Connect to the specified server. -m, --kill-server Terminate any running bitbake server. + --token XMLRPCTOKEN Specify the connection token to be used when + connecting to a remote server. --observe-only Connect to a server as an observing-only client. --status-only Check the status of the remote bitbake server. - -w WRITEEVENTLOG, --write-log=WRITEEVENTLOG - Writes the event log of the build to a bitbake event - json file. Use '' (empty string) to assign the name - automatically. - --runall=RUNALL Run the specified task for any recipe in the taskgraph - of the specified target (even if it wouldn't otherwise - have run). - --runonly=RUNONLY Run only the specified task within the taskgraph of - the specified targets (and any task dependencies those - tasks may have). + --server-only Run bitbake without a UI, only starting a server + (cooker) process. + + Configuration options: + -r PREFILE, --read PREFILE + Read the specified file before bitbake.conf. + -R POSTFILE, --postread POSTFILE + Read the specified file after bitbake.conf. + -I EXTRA_ASSUME_PROVIDED, --ignore-deps EXTRA_ASSUME_PROVIDED + Assume these dependencies don't exist and are already + provided (equivalent to ASSUME_PROVIDED). Useful to + make dependency graphs more appealing. + +.. + BitBake help output generated with "stty columns 80; bin/bitbake -h" .. _bitbake-examples: @@ -481,7 +570,7 @@ The following command runs the clean task on the ``foo.bb`` recipe file:: .. note:: - The "-b" option explicitly does not handle recipe dependencies. Other + The ``-b`` option explicitly does not handle recipe dependencies. Other than for debugging purposes, it is instead recommended that you use the syntax presented in the next section. @@ -496,7 +585,7 @@ dependencies, both for build-time and runtime. There must be a way for you to express recipe preferences when multiple recipes provide the same functionality, or when there are multiple versions of a recipe. -The ``bitbake`` command, when not using "--buildfile" or "-b" only +The ``bitbake`` command, when not using ``--buildfile`` or ``-b`` only accepts a "PROVIDES". You cannot provide anything else. By default, a recipe file generally "PROVIDES" its "packagename" as shown in the following example:: @@ -504,7 +593,7 @@ following example:: $ bitbake foo This next example "PROVIDES" the -package name and also uses the "-c" option to tell BitBake to just +package name and also uses the ``-c`` option to tell BitBake to just execute the ``do_clean`` task:: $ bitbake -c clean foo diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-library-functions.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-library-functions.rst new file mode 100644 index 00000000..09e35394 --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-library-functions.rst @@ -0,0 +1,59 @@ +.. SPDX-License-Identifier: CC-BY-2.5 + +================= +Library Functions +================= + +| + +This chapter lists common library functions available under the ``lib/`` +directory in BitBake. + +These functions can be used in recipes or configuration files with +:ref:`inline-Python ` or :ref:`Python +` functions. + +Logging utilities +================= + +Different logging utilities can be used from Python code in recipes or +configuration files. + +The strings passed below can be formatted with ``str.format()``, for example:: + + bb.warn("Houston, we have a %s", "bit of a problem") + +Formatted string can also be used directly:: + + bb.error("%s, we have a %s" % ("Houston", "big problem")) + +Python f-strings may also be used:: + + h = "Houston" + bb.fatal(f"{h}, we have a critical problem") + +.. automodule:: bb + :members: + debug, + error, + erroronce, + fatal, + note, + plain, + verbnote, + warn, + warnonce, + +``bb.utils`` +============ + +.. automodule:: bb.utils + :members: + :exclude-members: + LogCatcher, + PrCtlError, + VersionStringException, + better_compile, + better_exec, diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst index 58975f4c..40cae6b0 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst @@ -753,23 +753,17 @@ share the task. This section presents the mechanisms BitBake provides to allow you to share functionality between recipes. Specifically, the mechanisms -include ``include``, ``inherit``, :term:`INHERIT`, and ``require`` -directives. - -Locating Include and Class Files --------------------------------- - -BitBake uses the :term:`BBPATH` variable to locate -needed include and class files. Additionally, BitBake searches the -current directory for ``include`` and ``require`` directives. - -.. note:: - - The BBPATH variable is analogous to the environment variable PATH . - -In order for include and class files to be found by BitBake, they need -to be located in a "classes" subdirectory that can be found in -:term:`BBPATH`. +include :ref:`inherit`, +:ref:`inherit_defer`, +:ref:`include`, +:ref:`include_all`, +:ref:`require` and +:ref:`INHERIT` +directives. There is also a higher-level abstraction called +`configuration fragments` that is enabled with the +:ref:`bitbake-user-manual/bitbake-user-manual-metadata:\`\`addfragments\`\` directive`. + +.. _ref-bitbake-user-manual-metadata-inherit: ``inherit`` Directive --------------------- @@ -809,19 +803,43 @@ An advantage with the inherit directive as compared to both the :ref:`include ` and :ref:`require ` directives is that you can inherit class files conditionally. You can accomplish this by using a variable expression after the ``inherit`` -statement. Here is an example:: +statement. + +For inheriting classes conditionally, using the :ref:`inherit_defer +` directive is advised as +:ref:`inherit_defer ` is +evaluated at the end of parsing. + +.. _ref-bitbake-user-manual-metadata-inherit-defer: + +``inherit_defer`` Directive +--------------------------- + +The :ref:`inherit_defer ` +directive works like the :ref:`inherit +` directive, except that it is only +evaluated at the end of parsing. Its usage is recommended when a conditional +expression is used. + +This allows conditional expressions to be evaluated "late", meaning changes to +the variable after the line is parsed will take effect. With the :ref:`inherit +` directive this is not the case. - inherit ${VARNAME} +Here is an example:: + + inherit_defer ${VARNAME} If ``VARNAME`` is -going to be set, it needs to be set before the ``inherit`` statement is +going to be set, it needs to be set before the ``inherit_defer`` statement is parsed. One way to achieve a conditional inherit in this case is to use overrides:: VARIABLE = "" VARIABLE:someoverride = "myclass" -Another method is by using anonymous Python. Here is an example:: +Another method is by using :ref:`anonymous Python +`. +Here is an example:: python () { if condition == value: @@ -830,44 +848,94 @@ Another method is by using anonymous Python. Here is an example:: d.setVar('VARIABLE', '') } -Alternatively, you could use an in-line Python expression in the +Alternatively, you could use an inline Python expression in the following form:: - inherit ${@'classname' if condition else ''} - inherit ${@functionname(params)} + inherit_defer ${@'classname' if condition else ''} + +Or:: + + inherit_defer ${@bb.utils.contains('VARIABLE', 'something', 'classname', '', d)} In all cases, if the expression evaluates to an empty string, the statement does not trigger a syntax error because it becomes a no-op. +See also :term:`BB_DEFER_BBCLASSES` for automatically promoting classes +``inherit`` calls to ``inherit_defer``. + +.. _ref-include-directive: + ``include`` Directive --------------------- -BitBake understands the ``include`` directive. This directive causes -BitBake to parse whatever file you specify, and to insert that file at -that location. The directive is much like its equivalent in Make except -that if the path specified on the include line is a relative path, -BitBake locates the first file it can find within :term:`BBPATH`. +The ``include`` directive causes BitBake to parse a given file, +and to include that file's contents at the location of the +``include`` statement. This directive is similar to its equivalent +in Make, except that if the path specified on the BitBake ``include`` +line is a relative path, BitBake will search for it on the path designated +by :term:`BBPATH` and will include *only the first matching file*. -The include directive is a more generic method of including +The ``include`` directive is a more generic method of including functionality as compared to the :ref:`inherit ` directive, which is restricted to class (i.e. ``.bbclass``) files. The -include directive is applicable for any other kind of shared or +``include`` directive is applicable for any other kind of shared or encapsulated functionality or configuration that does not suit a ``.bbclass`` file. -As an example, suppose you needed a recipe to include some self-test -definitions:: +For example, if you needed a recipe to include some self-test definitions, +you might write:: include test_defs.inc +The ``include`` directive does not produce an error if the specified file +cannot be found. If the included file *must* exist, then you should use +use :ref:`require ` instead, which will generate an error +if the file cannot be found. + +.. note:: + + Note well that the ``include`` directive will include the first matching + file and nothing further (which is almost always the behaviour you want). + If you need to include all matching files, you need to use the + ``include_all`` directive, explained below. + +.. _ref-include-all-directive: + +``include_all`` Directive +------------------------- + +The ``include_all`` directive works like the :ref:`include +` +directive but will include *all* of the files that match the specified path in +the enabled layers (layers part of :term:`BBLAYERS`). + +Note that only :term:`BBPATH` will be searched, the parent directory of the file +with the ``include_all`` directive will not be searched (unlike for the +``include`` directive). + .. note:: - The include directive does not produce an error when the file cannot be - found. Consequently, it is recommended that if the file you are including is - expected to exist, you should use :ref:`require ` instead - of include . Doing so makes sure that an error is produced if the file cannot - be found. + This behaviour is rarely what you want in normal operation, and should + be reserved for only those situations when you explicitly want to parse + and include all matching files found across all layers, as the following + example shows. + +As a realistic example of this directive, imagine that all of your active +layers contain a file ``conf/distro/include/maintainers.inc``, containing +maintainer information for the recipes in that layer, and you wanted to +collect all of the content from all of those files across all of those layers. +You could use the statement:: + + include_all conf/distro/include/maintainers.inc + +In this case, BitBake will iterate through all of the directories in +the colon-separated :term:`BBPATH` (from left to right) and collect the +contents of all matching files, so you end up with the maintainer +information of all of your active layers, not just the first one. + +As the ``include_all`` directive uses the ``include`` directive in the +background, as with ``include``, no error is produced if no files are matched. .. _require-inclusion: @@ -933,6 +1001,197 @@ the ``autotools`` and ``pkgconfig`` classes:: INHERIT += "autotools pkgconfig" +``addfragments`` Directive +-------------------------- + +This directive allows fine-tuning local configurations with configuration +snippets contained in layers in a structured, controlled way. Typically it would +go into ``bitbake.conf``, for example:: + + addfragments conf/fragments OE_FRAGMENTS OE_FRAGMENTS_METADATA_VARS OE_FRAGMENTS_BUILTIN + +``addfragments`` takes four parameters: + +- path prefix for fragment files inside the layer file tree that bitbake + uses to construct full paths to the fragment files + +- name of variable that holds the list of enabled fragments in an + active build + +- name of variable that contains a list of variable names containing + fragment-specific metadata (such as descriptions) + +- name of variable that contains definitions for built-in fragments + +This allows listing enabled configuration fragments in ``OE_FRAGMENTS`` +variable like this:: + + OE_FRAGMENTS = "core/domain/somefragment core/someotherfragment anotherlayer/anotherdomain/anotherfragment" + +Fragment names listed in this variable must be prefixed by the layer name +where a fragment file is located, defined by :term:`BBFILE_COLLECTIONS` in ``layer.conf``. + +The implementation then expands this list into +:ref:`require ` +directives with full paths to respective layers:: + + require /path/to/core-layer/conf/fragments/domain/somefragment.conf + require /path/to/core-layer/conf/fragments/someotherfragment.conf + require /path/to/another-layer/conf/fragments/anotherdomain/anotherfragment.conf + +The variable containing a list of fragment metadata variables could look like this:: + + OE_FRAGMENTS_METADATA_VARS = "BB_CONF_FRAGMENT_SUMMARY BB_CONF_FRAGMENT_DESCRIPTION" + +The implementation will add a flag containing the fragment name to each of those variables +when parsing fragments, so that the variables are namespaced by fragment name, and do not override +each other when several fragments are enabled. + +The variable containing a built-in fragment definitions could look like this:: + + OE_FRAGMENTS_BUILTIN = "someprefix:SOMEVARIABLE anotherprefix:ANOTHERVARIABLE" + +and then if 'someprefix/somevalue' is added to the variable that holds the list +of enabled fragments:: + + OE_FRAGMENTS = "... someprefix/somevalue" + +bitbake will treat that as direct value assignment in its configuration:: + + SOMEVARIABLE = "somevalue" + +Locating Include Files +---------------------- + +BitBake first searches the current directory for needed include files for +:ref:`include ` and :ref:`require ` +directives. +BitBake also uses the :term:`BBPATH` variable. + +.. note:: + + The :term:`BBPATH` variable is analogous to the environment variable ``PATH``. + +For these two directives, BitBake includes the first file it finds. + +Let's consider the following statement called from a recipe file located in +``/layers/meta-custom2/recipes-example/example/example_0.1.bb``:: + + require myfile.inc + +Where ``myfile.inc`` is located in ``/layers/meta-custom2/recipes-example/example``. + +And let's assume that the value of :term:`BBPATH` is +``/layers/meta-custom1:/layers/meta-custom2``. Then BitBake will try to find +``myfile.inc`` in this order:: + + /layers/meta-custom2/recipes-example/example/myfile.inc + /layers/meta-custom1/myfile.inc + /layers/meta-custom2/myfile.inc + +In this case the first path of the list matches and BitBake includes this file +in ``example_0.1.bb``. + +Another common example would be:: + + require recipes-other/other/otherfile.inc + +Where ``otherfile.inc`` is located in +``/layers/meta-custom1/recipes-other/other/``. + +In this case, the following paths would be searched:: + + /layers/meta-custom2/recipes-example/example/recipes-other/other/otherfile.inc + /layers/meta-custom1/recipes-other/other/otherfile.inc + /layers/meta-custom2/recipes-other/other/otherfile.inc + +This time, the second item of this list would be matched. + +Note that the first path is based on the location of the file with the +``require`` (or ``include``) directive. Imagine there's a +``/layers/meta-custom2/recipes-bbappend/example/example_0.1.bbappend`` with:: + + require myappend.inc + +In this case, the following paths would be searched:: + + /layers/meta-custom2/recipes-bbappend/example/myappend.inc + /layers/meta-custom1/myappend.inc + /layers/meta-custom2/myappend.inc + +.. note:: + + In the above examples, the exact same search order applies for the + :ref:`include ` directive. + +It is also possible to include *all* occurences of a file with the same name +with the :ref:`include_all ` directive. + +Let's consider the following statement called from a recipe file located in +``/layers/meta-custom2/recipes-example/example/exampleall_0.1.bb``:: + + include_all all.inc + +Where multiple ``all.inc`` are in located in ``/layers/meta-custom2`` and +``/layers/meta-custom1``. + +And let's assume that the value of :term:`BBPATH` is +``/layers/meta-custom1:/layers/meta-custom2``. Then BitBake will try to include +all ``all.inc`` in this order:: + + /layers/meta-custom1/all.inc + /layers/meta-custom2/all.inc + +In this case the first ``/layers/meta-custom1/all.inc`` would be included, and +then ``/layers/meta-custom2/all.inc`` if both are found. + +.. note:: + + The same logic as for the :ref:`ref-include-directive` applies, except that + the path relative to the file where the directive is specified is not + searched when using the ``include_all`` directive. + +Locating Class Files +-------------------- + +Like include files, class files are located using the :term:`BBPATH` variable. +The classes can be included in the ``classes-recipe``, ``classes-global`` and +``classes`` directories, as explained in the +:ref:`bitbake-user-manual/bitbake-user-manual-intro:Class types` section of the +BitBake User Manual. Like for the :ref:`include ` and +:ref:`require ` directives, BitBake stops and inherits the +first class that it finds. + +For classes inherited with the :ref:`inherit +` directive, BitBake will try to +locate the class under each ``classes-recipe`` directory for each path in +:term:`BBPATH`, and then do the same for each ``classes`` directory for each +path in :term:`BBPATH`. + +For example, if the value :term:`BBPATH` is +``/layers/meta-custom1:/layers/meta-custom2`` then the ``hello`` class +would be searched in this order:: + + /layers/meta-custom1/classes-recipe/hello.bbclass + /layers/meta-custom2/classes-recipe/hello.bbclass + /layers/meta-custom1/classes/hello.bbclass + /layers/meta-custom2/classes/hello.bbclass + +.. note:: + + Note that the order of the list above does not depend on where the class is + inherited from. + +Likewise, for classes inherited with the :term:`INHERIT` variable, the +``classes-global`` directory is searched first, and the ``classes`` directory is +searched second. Taking the above example, this would result in the following +list:: + + /layers/meta-custom1/classes-global/hello.bbclass + /layers/meta-custom2/classes-global/hello.bbclass + /layers/meta-custom1/classes/hello.bbclass + /layers/meta-custom2/classes/hello.bbclass + Functions ========= @@ -972,7 +1231,9 @@ When you create these types of functions in your recipe or class files, you need to follow the shell programming rules. The scripts are executed by ``/bin/sh``, which may not be a bash shell but might be something such as ``dash``. You should not use -Bash-specific script (bashisms). +Bash-specific script (bashisms), such as ``[[`` (use ``[`` instead). The BitBake +shell parser also has further +:ref:`limitations `. Overrides and override-style operators like ``:append`` and ``:prepend`` can also be applied to shell functions. Most commonly, this application @@ -1013,6 +1274,20 @@ Running ``do_foo`` prints the following:: You can use the ``bitbake -e recipename`` command to view the final assembled function after all overrides have been applied. +Limitations +^^^^^^^^^^^ + +Shell functions must be parsable by BitBake, which cannot handle all +syntax supported by POSIX shells. In particular: + +#. Arithmetic expansion is not supported. This means you cannot do: + + .. code-block:: shell + + VAR=$(( i + 1 )) + + See https://bugzilla.yoctoproject.org/show_bug.cgi?id=11314. + BitBake-Style Python Functions ------------------------------ @@ -1303,8 +1578,8 @@ the task and other tasks. Here is an example that shows how to define a task and declare some dependencies:: python do_printdate () { - import time - print time.strftime('%Y%m%d', time.gmtime()) + import datetime + bb.plain('Date: %s' % (datetime.date.today())) } addtask printdate after do_fetch before do_build @@ -1496,22 +1771,22 @@ functionality of the task: directory listed is used as the current working directory for the task. -- ``[file-checksums]``: Controls the file dependencies for a task. The - baseline file list is the set of files associated with - :term:`SRC_URI`. May be used to set additional dependencies on - files not associated with :term:`SRC_URI`. +- ``[file-checksums]``: Controls the file dependencies for a task. The + baseline file list is the set of files associated with + :term:`SRC_URI`. May be used to set additional dependencies on + files not associated with :term:`SRC_URI`. - The value set to the list is a file-boolean pair where the first - value is the file name and the second is whether or not it - physically exists on the filesystem. :: + The value set to the list is a file-boolean pair where the first + value is the file name and the second is whether or not it + physically exists on the filesystem. :: - do_configure[file-checksums] += "${MY_DIRPATH}/my-file.txt:True" + do_configure[file-checksums] += "${MY_DIRPATH}/my-file.txt:True" - It is important to record any paths which the task looked at and - which didn't exist. This means that if these do exist at a later - time, the task can be rerun with the new additional files. The - "exists" True or False value after the path allows this to be - handled. + It is important to record any paths which the task looked at and + which didn't exist. This means that if these do exist at a later + time, the task can be rerun with the new additional files. The + "exists" True or False value after the path allows this to be + handled. - ``[lockfiles]``: Specifies one or more lockfiles to lock while the task executes. Only one task may hold a lockfile, and any task that @@ -1972,11 +2247,8 @@ access. Here is a list of available operations: Other Functions --------------- -You can find many other functions that can be called from Python by -looking at the source code of the ``bb`` module, which is in -``bitbake/lib/bb``. For example, ``bitbake/lib/bb/utils.py`` includes -the commonly used functions ``bb.utils.contains()`` and -``bb.utils.mkdirhier()``, which come with docstrings. +Other functions are documented in the +:doc:`/bitbake-user-manual/bitbake-user-manual-library-functions` document. Extending Python Library Code ----------------------------- @@ -2016,12 +2288,12 @@ OpenEmbedded metadata-based example. These checksums are stored in :term:`STAMP`. You can examine the checksums using the following BitBake command:: - $ bitbake-dumpsigs + $ bitbake-dumpsig This command returns the signature data in a readable format that allows you to examine the inputs used when the OpenEmbedded build system generates signatures. For example, using -``bitbake-dumpsigs`` allows you to examine the ``do_compile`` task's +``bitbake-dumpsig`` allows you to examine the ``do_compile`` task's "sigdata" for a C application (e.g. ``bash``). Running the command also reveals that the "CC" variable is part of the inputs that are hashed. Any changes to this variable would invalidate the stamp and cause the diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst index 899e584f..8d8e8b8b 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst @@ -36,6 +36,14 @@ overview of their function and contents. when specified allows for the Git binary from the host to be used rather than building ``git-native``. + :term:`AUTOREV` + This is a special variable used during fetching. When :term:`SRCREV` is + set to the value of this variable, the latest revision from the version + controlled source code repository is used. + It should be set as follows:: + + SRCREV = "${AUTOREV}" + :term:`AZ_SAS` Azure Storage Shared Access Signature, when using the :ref:`Azure Storage fetcher ` @@ -108,6 +116,10 @@ overview of their function and contents. Specifies if SSL certificates should be checked when fetching. The default value is ``1`` and certificates are not checked if the value is set to ``0``. + :term:`BB_CMDLINE` + :term:`BB_CMDLINE` is an internal variable and is read-only. It captures + the exact command line used for the current invocation of BitBake. + :term:`BB_HASH_CODEPARSER_VALS` Specifies values for variables to use when populating the codeparser cache. This can be used selectively to set dummy values for variables to avoid @@ -127,17 +139,10 @@ overview of their function and contents. Contains the name of the currently running task. The name does not include the ``do_`` prefix. - :term:`BB_DANGLINGAPPENDS_WARNONLY` - Defines how BitBake handles situations where an append file - (``.bbappend``) has no corresponding recipe file (``.bb``). This - condition often occurs when layers get out of sync (e.g. ``oe-core`` - bumps a recipe version and the old recipe no longer exists and the - other layer has not been updated to the new version of the recipe - yet). - - The default fatal behavior is safest because it is the sane reaction - given something is out of sync. It is important to realize when your - changes are no longer being applied. + :term:`BB_CURRENT_MC` + Contains the name of the current multiconfig a task is being run under. + The name is taken from the multiconfig configuration file (a file + ``mc1.conf`` would make this variable equal to ``mc1``). :term:`BB_DEFAULT_TASK` The default task to use when none is specified (e.g. with the ``-c`` @@ -148,6 +153,25 @@ overview of their function and contents. The default umask to apply to tasks if specified and no task specific umask flag is set. + :term:`BB_DEFER_BBCLASSES` + The classes listed in this variable have their :ref:`inherit + ` calls automatically promoted + to deferred inherits. See :ref:`inherit_defer + ` for more information on + deferred inherits. + + This means that if :term:`BB_DEFER_BBCLASSES` is set as follows:: + + BB_DEFER_BBCLASSES = "foo" + + The following statement:: + + inherit foo + + Will automatically be equal to calling:: + + inherit_defer foo + :term:`BB_DISKMON_DIRS` Monitors disk space and available inodes during the build and allows you to control the build based on these parameters. @@ -262,8 +286,8 @@ overview of their function and contents. Specifies the internal list of variables to allow through from the external environment into BitBake's datastore. If the value of this variable is not specified (which is the default), the following - list is used: :term:`BBPATH`, :term:`BB_PRESERVE_ENV`, - :term:`BB_ENV_PASSTHROUGH`, and :term:`BB_ENV_PASSTHROUGH_ADDITIONS`. + list is used: :term:`BBPATH`, :term:`BB_PRESERVE_ENV`, and + :term:`BB_ENV_PASSTHROUGH_ADDITIONS`. .. note:: @@ -317,6 +341,11 @@ overview of their function and contents. For example usage, see :term:`BB_GIT_SHALLOW`. + :term:`BB_GIT_DEFAULT_DESTSUFFIX` + The default destination directory where the :ref:`Git fetcher + ` unpacks the source code. If this variable is not set, the + source code is unpacked in a directory named "git". + :term:`BB_GIT_SHALLOW` Setting this variable to "1" enables the support for fetching, using and generating mirror tarballs of `shallow git repositories `_. @@ -327,11 +356,26 @@ overview of their function and contents. mirror tarball. If the shallow mirror tarball cannot be fetched, it will try to fetch the full mirror tarball and use that. - When a mirror tarball is not available, a full git clone will be performed - regardless of whether this variable is set or not. Support for shallow - clones is not currently implemented as git does not directly support - shallow cloning a particular git commit hash (it only supports cloning - from a tag or branch reference). + This setting causes an initial shallow clone instead of an initial full bare clone. + The amount of data transferred during the initial clone will be significantly reduced. + + However, every time the source revision (referenced in :term:`SRCREV`) + changes, regardless of whether the cache within the download directory + (defined by :term:`DL_DIR`) has been cleaned up or not, + the data transfer may be significantly higher because entirely + new shallow clones are required for each source revision change. + + Over time, numerous shallow clones may cumulatively transfer + the same amount of data as an initial full bare clone. + This is especially the case with very large repositories. + + Existing initial full bare clones, created without this setting, + will still be utilized. + + If the Git error "Server does not allow request for unadvertised object" + occurs, an initial full bare clone is fetched automatically. + This may happen if the Git server does not allow the request + or if the Git client has issues with this functionality. See also :term:`BB_GIT_SHALLOW_DEPTH` and :term:`BB_GENERATE_SHALLOW_TARBALLS`. @@ -359,6 +403,23 @@ overview of their function and contents. For example usage, see :term:`BB_GIT_SHALLOW`. + :term:`BB_GIT_SHALLOW_EXTRA_REFS` + With :term:`BB_GIT_SHALLOW` enabled, by default, all unused refs (branches + and tags) are removed from the repository, as shallow processing scales + with the number of refs it has to process. + + :term:`BB_GIT_SHALLOW_EXTRA_REFS` allows to explicitly specify additional + refs to keep. This is particularly useful for recipes with custom checkout + processes, or whose Git-based versioning requires a tag be available (i.e. + for ``git describe --tags``). The :term:`BB_GIT_SHALLOW_EXTRA_REFS` + variable is a space-separated list of refs, fully specified, and supports + wildcards. + + Example usage:: + + BB_GIT_SHALLOW_EXTRA_REFS = "refs/tags/v1.0" + BB_GIT_SHALLOW_EXTRA_REFS += "refs/heads/*" + :term:`BB_GLOBAL_PYMODULES` Specifies the list of Python modules to place in the global namespace. It is intended that only the core layer should set this and it is meant @@ -414,6 +475,29 @@ overview of their function and contents. equivalences that correspond to Share State caches that are only available on specific clients. + :term:`BB_HASHSERVE_DB_DIR` + When :term:`BB_HASHSERVE` is set to ``auto``, bitbake will use + a private location inside a particular build directory for the sqlite + database file that holds hash equivalency data. + + This variable allows using a different path, which can be shared + between multiple build directories and bitbake instances + that operate on them. This enables using a common ``${SSTATE_DIR}`` + together with common hash equivalency data for local builds, without having to + separately manage a hash equivalency server. + + .. note:: + + This variable cannot be set to a NFS mount and bitbake will error out then. + The reason is that NFS implementations can have file locking issues, which + can cause data loss and corruption when there are multiple writers operating + on a file at the same time as explained in https://sqlite.org/faq.html#q5 + + If you'd like to share hash equivalency data between multiple computers, you + need to set up a hash equivalency server separately and point :term:`BB_HASHSERVE` + to it. See https://docs.yoctoproject.org/dev-manual/hashequivserver.html for + additional information. + :term:`BB_HASHSERVE_UPSTREAM` Specifies an upstream Hash Equivalence server. @@ -424,7 +508,7 @@ overview of their function and contents. Example usage:: - BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687" + BB_HASHSERVE_UPSTREAM = "hashserv.yoctoproject.org:8686" :term:`BB_INVALIDCONF` Used in combination with the ``ConfigParsed`` event to trigger @@ -525,11 +609,32 @@ overview of their function and contents. version 4.20 expose under ``/proc/pressure``. The threshold represents the difference in "total" pressure from the previous second. The minimum value is 1.0 (extremely slow builds) and the maximum is - 1000000 (a pressure value unlikely to ever be reached). + 1000000 (a pressure value unlikely to ever be reached). See + https://docs.kernel.org/accounting/psi.html for more information. + + A default value to limit the CPU pressure to be set in ``conf/local.conf`` + could be:: + + BB_PRESSURE_MAX_CPU = "15000" + + Multiple values should be tested on the build host to determine what suits + best, depending on the need for performance versus load average during + the build. - This threshold can be set in ``conf/local.conf`` as:: + .. note:: + + You can track the pressure information while a build is running with: + + .. code-block:: console - BB_PRESSURE_MAX_CPU = "500" + $ tail -F tmp/log/cooker//console-latest.log | grep "Pressure status changed" + NOTE: Pressure status changed to CPU: True, IO: False, Mem: False (CPU: 1105.9/2.0, IO: 0.0/2.0, Mem: 0.0/2.0) - using 1/64 bitbake threads + ... + + If these messages are printed a lot, it means that the :term:`BB_PRESSURE_MAX_CPU` + should be increased to a reasonable value for limiting the CPU pressure + on the system. Monitor the varying value after ``CPU:`` above to set a + sensible value. :term:`BB_PRESSURE_MAX_IO` Specifies a maximum I/O pressure threshold, above which BitBake's @@ -541,14 +646,38 @@ overview of their function and contents. version 4.20 expose under ``/proc/pressure``. The threshold represents the difference in "total" pressure from the previous second. The minimum value is 1.0 (extremely slow builds) and the maximum is - 1000000 (a pressure value unlikely to ever be reached). + 1000000 (a pressure value unlikely to ever be reached). See + https://docs.kernel.org/accounting/psi.html for more information. At this point in time, experiments show that IO pressure tends to be short-lived and regulating just the CPU with :term:`BB_PRESSURE_MAX_CPU` can help to reduce it. - :term:`BB_PRESSURE_MAX_MEMORY` + A default value to limit the IO pressure to be set in ``conf/local.conf`` + could be:: + + BB_PRESSURE_MAX_IO = "15000" + Multiple values should be tested on the build host to determine what suits + best, depending on the need for performance versus I/O usage during the + build. + + .. note:: + + You can track the pressure information while a build is running with: + + .. code-block:: console + + $ tail -F tmp/log/cooker//console-latest.log | grep "Pressure status changed" + NOTE: Pressure status changed to CPU: None, IO: True, Mem: False (CPU: 2236.0/None, IO: 153.6/2.0, Mem: 0.0/2.0) - using 19/64 bitbake threads + ... + + If these messages are printed a lot, it means that the :term:`BB_PRESSURE_MAX_IO` + should be increased to a reasonable value for limiting the I/O pressure + on the system. Monitor the varying value after ``IO:`` above to set a + sensible value. + + :term:`BB_PRESSURE_MAX_MEMORY` Specifies a maximum memory pressure threshold, above which BitBake's scheduler will not start new tasks (providing there is at least one active task). If no value is set, memory pressure is not @@ -558,7 +687,8 @@ overview of their function and contents. version 4.20 expose under ``/proc/pressure``. The threshold represents the difference in "total" pressure from the previous second. The minimum value is 1.0 (extremely slow builds) and the maximum is - 1000000 (a pressure value unlikely to ever be reached). + 1000000 (a pressure value unlikely to ever be reached). See + https://docs.kernel.org/accounting/psi.html for more information. Memory pressure is experienced when time is spent swapping, refaulting pages from the page cache or performing direct reclaim. @@ -566,6 +696,30 @@ overview of their function and contents. might be useful as a last resort to prevent OOM errors if they are occurring during builds. + A default value to limit the memory pressure to be set in + ``conf/local.conf`` could be:: + + BB_PRESSURE_MAX_MEMORY = "15000" + + Multiple values should be tested on the build host to determine what suits + best, depending on the need for performance versus memory consumption + during the build. + + .. note:: + + You can track the pressure information while a build is running with: + + .. code-block:: console + + $ tail -F tmp/log/cooker//console-latest.log | grep "Pressure status changed" + NOTE: Pressure status changed to CPU: None, IO: False, Mem: True (CPU: 29.5/None, IO: 0.0/2.0, Mem: 2553.3/2.0) - using 17/64 bitbake threads + ... + + If these messages are printed a lot, it means that the :term:`BB_PRESSURE_MAX_MEMORY` + should be increased to a reasonable value for limiting the memory + pressure on the system. Monitor the varying value after ``Mem:`` above + to set a sensible value. + :term:`BB_RUNFMT` Specifies the name of the executable script files (i.e. run files) saved into ``${``\ :term:`T`\ ``}``. By default, the @@ -680,11 +834,11 @@ overview of their function and contents. .. note:: In order for your I/O priority settings to take effect, you need the - Completely Fair Queuing (CFQ) Scheduler selected for the backing block + Budget Fair Queuing (BFQ) Scheduler selected for the backing block device. To select the scheduler, use the following command form where device is the device (e.g. sda, sdb, and so forth):: - $ sudo sh -c "echo cfq > /sys/block/device/queu/scheduler" + $ sudo sh -c "echo bfq > /sys/block/device/queue/scheduler" :term:`BB_TASK_NICE_LEVEL` Allows specific tasks to change their priority (i.e. nice level). @@ -699,6 +853,12 @@ overview of their function and contents. Within an executing task, this variable holds the hash of the task as returned by the currently enabled signature generator. + :term:`BB_USE_HOME_NPMRC` + Controls whether or not BitBake uses the user's .npmrc file within their + home directory within the npm fetcher. This can be used for authentication + of private NPM registries, among other uses. This is turned off by default + and requires the user to explicitly set it to "1" to enable. + :term:`BB_VERBOSE_LOGS` Controls how verbose BitBake is during builds. If set, shell scripts echo commands and shell script output appears on standard out @@ -766,6 +926,10 @@ overview of their function and contents. :term:`BBFILE_PRIORITY` Assigns the priority for recipe files in each layer. + This variable is used in the ``conf/layer.conf`` file and must be + suffixed with a `_` followed by the name of the specific layer (e.g. + ``BBFILE_PRIORITY_emenlow``). Colon as separator is not supported. + This variable is useful in situations where the same recipe appears in more than one layer. Setting this variable allows you to prioritize a layer against other layers that contain the same recipe @@ -780,7 +944,7 @@ overview of their function and contents. higher precedence. For example, the value 6 has a higher precedence than the value 5. If not specified, the :term:`BBFILE_PRIORITY` variable is set based on layer dependencies (see the :term:`LAYERDEPENDS` variable - for more information. The default priority, if unspecified for a + for more information). The default priority, if unspecified for a layer with no dependencies, is the lowest defined priority + 1 (or 1 if no priorities are defined). @@ -884,28 +1048,101 @@ overview of their function and contents. compiler. Consequently, the syntax follows Python's Regular Expression (re) syntax. The expressions are compared against the full paths to the files. For complete syntax information, see Python's - documentation at http://docs.python.org/3/library/re.html. + documentation at https://docs.python.org/3/library/re.html. The following example uses a complete regular expression to tell - BitBake to ignore all recipe and recipe append files in the - ``meta-ti/recipes-misc/`` directory:: + BitBake to ignore all recipe and recipe append files in + ``recipes-bsp`` directory (recursively) of ``meta-ti-bsp``:: - BBMASK = "meta-ti/recipes-misc/" + BBMASK = "${BBFILE_PATTERN_meta-ti-bsp}/recipes-bsp/" If you want to mask out multiple directories or recipes, you can specify multiple regular expression fragments. This next example masks out multiple directories and individual recipes:: - BBMASK += "/meta-ti/recipes-misc/ meta-ti/recipes-ti/packagegroup/" - BBMASK += "/meta-oe/recipes-support/" - BBMASK += "/meta-foo/.*/openldap" - BBMASK += "opencv.*\.bbappend" - BBMASK += "lzma" + BBMASK += "${BBFILE_PATTERN_meta-ti-bsp}/recipes-graphics/libgal/" + BBMASK += "${BBFILE_PATTERN_openembedded-layer}/recipes-support/" + BBMASK += "${BBFILE_PATTERN_openembedded-layer}/.*/openldap" + BBMASK += "${BBFILE_PATTERN_meta-ti-bsp}/.*/optee.*\.bbappend" + + This masks: + + - everything under the ``recipes-graphics/libgal/`` directory from + ``meta-ti-bsp``, + - everything under the ``recipes-support/`` directory in ``meta-oe``, + - everything under a directory whose name starts with ``openldap``, and + every file with the same naming scheme, in ``meta-oe`` at any directory + depth > 1 (e.g. in ``meta-oe``, ``recipes-foo/openldap-stuff/`` or + ``recipes-bar/baz/openldap_0.1.bb`` but not ``openldap/``), + - every append file whose name starts with ``optee`` in ``meta-ti-bsp`` at + any directory depth > 1 (e.g. ``optee/optee-examples_%.bbappend`` and + ``recipes-security/optee/optee-client_%.bbappend``). + + .. note:: + + Because these are complete regular expressions, if you want to match a + directory and not a file, you must end the expression with a trailing + slash. That is:: + + BBMASK += "${BBFILE_PATTERN_meta-ti-bsp}/recipes-graphics/libgal/" + + Will match anything under ``recipes-graphics/ligbal/`` directory of + ``meta-ti-bsp``. And:: + + BBMASK += "${BBFILE_PATTERN_meta-ti-bsp}/recipes-graphics/libgal" + + Will match in ``meta-ti-bsp`` any file prefixed with ``libgal`` in + ``recipes-graphics/`` and any directory (recursively; and its + recipes and recipe append files regardless how they are named) prefixed + with ``libgal`` in ``recipes-graphics/``. That is, provided your layers + are available at ``/bitbake-builds/poky-master/layers/``, it'll match:: + + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal.bb + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal_%.bbappend + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal-foo/foo.bb + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal-foo/foo/bz.bbappend + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal/bar.bb + + .. note:: + + Because these are complete regular expressions, failing to start the + pattern with a ``^`` sign (which is usually the first character in + :term:`BBFILE_PATTERN`) means it can match *any* portion of a path. + Take the following as an example:: + + BBMASK = "recipes-graphics/libgal/" + + This will match subdirectories and files in any path containing + ``recipes-graphics/libgal/``, meaning (considering your layers are + available at ``/bitbake-builds/poky-master/layers/``):: + + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal/ + /bitbake-builds/poky-master/layers/my-layer/foo-recipes-graphics/libgal/ + + will be both matched. This may be a more relaxed way of matching + directories, recipes and recipe append files from any third party layer + but is generally discouraged as it may be casting too wide of a net. .. note:: - When specifying a directory name, use the trailing slash character - to ensure you match just that directory name. + Because these are complete regular expressions, a leading slash does + not mean the path is absolute. It simply forces the directory to be + named exactly that. Take:: + + BBMASK = "recipes-graphics/libgal/" + + If you happen to have a directory ``foo-recipes-graphics/libgal/``, it + will be matched. + + Leading with a slash:: + + BBMASK = "/recipes-graphics/libgal/" + + makes sure that doesn't happen. However, this doesn't prevent matching + a directory ``recipes-graphics/libgal/`` from another layer. + + Again, we highly recommend using :term:`BBFILE_PATTERN` in the + expressions to specify absolute paths specific to one layer. :term:`BBMULTICONFIG` Enables BitBake to perform multiple configuration builds and lists @@ -963,18 +1200,10 @@ overview of their function and contents. A name assigned to the build. The name defaults to a datetime stamp of when the build was started but can be defined by the metadata. - :term:`BZRDIR` - The directory in which files checked out of a Bazaar system are - stored. - :term:`CACHE` Specifies the directory BitBake uses to store a cache of the metadata so it does not need to be parsed every time BitBake is started. - :term:`CVSDIR` - The directory in which files checked out under the CVS system are - stored. - :term:`DEFAULT_PREFERENCE` Specifies a weak bias for recipe selection priority. @@ -1100,9 +1329,17 @@ overview of their function and contents. :term:`INHERIT` Causes the named class or classes to be inherited globally. Anonymous - functions in the class or classes are not executed for the base - configuration and in each individual recipe. The OpenEmbedded build - system ignores changes to :term:`INHERIT` in individual recipes. + functions in the class or classes are executed in two disjoint situations: + + - When only the + :ref:`base configuration ` + is parsed. For example as a result of the following BitBake invocation:: + + $ bitbake -e + + - When recipes are parsed - then for each parsed recipe. + + BitBake ignores changes to :term:`INHERIT` in individual recipes. For more information on :term:`INHERIT`, see the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:\`\`inherit\`\` configuration directive`" @@ -1213,8 +1450,13 @@ overview of their function and contents. :term:`PERSISTENT_DIR` Specifies the directory BitBake uses to store data that should be preserved between builds. In particular, the data stored is the data - that uses BitBake's persistent data API and the data used by the PR - Server and PR Service. + that uses BitBake's persistent data API, the data used by the PR + Server and PR Service, and the default location of the Hash Equivalence + database (when :term:`BB_HASHSERVE` is set to ``auto``). + + This directory should not be shared between different builds. If you need + to share the Hash Equivalence database, you should setup a Hash + Equivalence server instead. :term:`PF` Specifies the recipe or package name and includes all version and @@ -1460,14 +1702,8 @@ overview of their function and contents. - ``az://``: Fetches files from an Azure Storage account using HTTPS. - - ``bzr://``: Fetches files from a Bazaar revision control - repository. - - ``ccrc://``: Fetches files from a ClearCase repository. - - ``cvs://``: Fetches files from a CVS revision control - repository. - - ``file://``: Fetches files, which are usually files shipped with the Metadata, from the local machine. The path is relative to the :term:`FILESPATH` @@ -1502,8 +1738,11 @@ overview of their function and contents. - ``npm://``: Fetches JavaScript modules from a registry. - - ``osc://``: Fetches files from an OSC (OpenSUSE Build service) - revision control repository. + .. warning:: + + The NPM fetcher is currently disabled due to security concerns. See + `355cd226e0720a9ed7683bb01c8c0a58eee03664 `__ + for more information. - ``p4://``: Fetches files from a Perforce (``p4``) revision control repository. @@ -1522,7 +1761,8 @@ overview of their function and contents. - ``name``: Specifies a name to be used for association with :term:`SRC_URI` checksums or :term:`SRCREV` when you have more than one - file or git repository specified in :term:`SRC_URI`. For example:: + file or source control repository specified in :term:`SRC_URI`. + For example:: SRC_URI = "git://example.com/foo.git;branch=main;name=first \ git://example.com/bar.git;branch=main;name=second \ @@ -1535,7 +1775,8 @@ overview of their function and contents. - ``subdir``: Places the file (or extracts its contents) into the specified subdirectory. This option is useful for unusual tarballs or other archives that do not have their files already in a - subdirectory within the archive. + subdirectory within the archive. This path can be further modified + by fetcher specific parameters. - ``subpath``: Limits the checkout to a specific subpath of the tree when using the Git fetcher is used. @@ -1550,18 +1791,18 @@ overview of their function and contents. :term:`SRCREV` The revision of the source code used to build the package. This - variable applies only when using Subversion, Git, Mercurial and - Bazaar. If you want to build a fixed revision and you want to avoid + variable applies only when using Subversion, Git and Mercurial. + If you want to build a fixed revision and you want to avoid performing a query on the remote repository every time BitBake parses your recipe, you should specify a :term:`SRCREV` that is a full revision identifier and not just a tag. :term:`SRCREV_FORMAT` - Helps construct valid :term:`SRCREV` values when + Helps construct a valid package version string when multiple source controlled URLs are used in :term:`SRC_URI`. - The system needs help constructing these values under these + The system needs help constructing this value under these circumstances. Each component in the :term:`SRC_URI` is assigned a name and these are referenced in the :term:`SRCREV_FORMAT` variable. Consider an example with URLs named "machine" and "meta". In this case, @@ -1570,6 +1811,12 @@ overview of their function and contents. ``AUTOINC`` placeholder is added and if needed. And, this placeholder is placed at the start of the returned string. + The :term:`SRCREV_FORMAT` can also take the form "_component2". + This assumes that there is a component in the :term:`SRC_URI` that does not + have a name assigned. While this is not considered good practice, it can be + usefull if a ``.bbappend`` file needs to extend the :term:`SRC_URI` with + an additional repository. + :term:`STAMP` Specifies the base path used to create recipe stamp files. The path to an actual stamp file is constructed by evaluating this string and diff --git a/bitbake/doc/conf.py b/bitbake/doc/conf.py index fc2ee081..9a53e55c 100644 --- a/bitbake/doc/conf.py +++ b/bitbake/doc/conf.py @@ -6,18 +6,27 @@ # -- Path setup -------------------------------------------------------------- -# If extensions (or modules to document with autodoc) are in another directory, -# add these directories to sys.path here. If the directory is relative to the -# documentation root, use os.path.abspath to make it absolute, like shown here. -# -# import os -# import sys -# sys.path.insert(0, os.path.abspath('.')) - -import sys import datetime +import os +import subprocess +import sys + +from pathlib import Path -current_version = "dev" +# Test that we are building from a Git repository +try: + subprocess.run(["git", "rev-parse", "--is-inside-work-tree"], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True) +except subprocess.CalledProcessError: + sys.exit("Building bitbake's documentation must be done from its Git repository.\n" + "Clone the repository with the following command:\n" + "git clone https://git.openembedded.org/bitbake ") + +sys.path.insert(0, os.path.abspath('.')) +import setversions + +current_version = setversions.get_current_version() +print(f"Version calculated to be {current_version}") # String used in sidebar version = 'Version: ' + current_version @@ -26,6 +35,12 @@ if current_version == 'dev': # Version seen in documentation_options.js and hence in js switchers code release = current_version +setversions.write_switchers_js("sphinx-static/switchers.js.in", + "sphinx-static/switchers.js", + current_version) + +setversions.write_releases_rst("releases.rst") + # -- Project information ----------------------------------------------------- project = 'Bitbake' @@ -37,6 +52,7 @@ author = 'Richard Purdie, Chris Larson, and Phil Blundell' extlinks = { 'yocto_docs': ('https://docs.yoctoproject.org%s', None), 'oe_lists': ('https://lists.openembedded.org%s', None), + 'wikipedia': ('https://en.wikipedia.org/wiki/%s', None), } # -- General configuration --------------------------------------------------- @@ -47,6 +63,7 @@ extlinks = { extensions = [ 'sphinx.ext.autosectionlabel', 'sphinx.ext.extlinks', + 'sphinx.ext.autodoc', ] autosectionlabel_prefix_document = True @@ -99,3 +116,7 @@ html_last_updated_fmt = '%b %d, %Y' # Remove the trailing 'dot' in section numbers html_secnumber_suffix = " " + +# autoconf needs the modules available to auto-generate documentation from the +# code +sys.path.insert(0, str(Path('..', 'lib').resolve())) diff --git a/bitbake/doc/index.rst b/bitbake/doc/index.rst index ee1660ac..9f2a9067 100644 --- a/bitbake/doc/index.rst +++ b/bitbake/doc/index.rst @@ -12,10 +12,12 @@ BitBake User Manual bitbake-user-manual/bitbake-user-manual-intro bitbake-user-manual/bitbake-user-manual-execution + bitbake-user-manual/bitbake-user-manual-environment-setup bitbake-user-manual/bitbake-user-manual-metadata bitbake-user-manual/bitbake-user-manual-ref-variables-context bitbake-user-manual/bitbake-user-manual-fetching bitbake-user-manual/bitbake-user-manual-ref-variables + bitbake-user-manual/bitbake-user-manual-library-functions bitbake-user-manual/bitbake-user-manual-hello .. toctree:: diff --git a/bitbake/doc/releases.rst b/bitbake/doc/releases.rst deleted file mode 100644 index b38b1c06..00000000 --- a/bitbake/doc/releases.rst +++ /dev/null @@ -1,174 +0,0 @@ -.. SPDX-License-Identifier: CC-BY-2.5 - -================================= -BitBake Supported Release Manuals -================================= - -******************************* -Release Series 4.2 (mickledore) -******************************* - -- :yocto_docs:`BitBake 2.4 User Manual ` - -****************************** -Release Series 4.0 (kirkstone) -****************************** - -- :yocto_docs:`BitBake 2.0 User Manual ` - -**************************** -Release Series 3.1 (dunfell) -**************************** - -- :yocto_docs:`BitBake 1.46 User Manual ` - -================================ -BitBake Outdated Release Manuals -================================ - -***************************** -Release Series 4.1 (langdale) -***************************** - -- :yocto_docs:`BitBake 2.2 User Manual ` - -****************************** -Release Series 3.4 (honister) -****************************** - -- :yocto_docs:`BitBake 1.52 User Manual ` - -****************************** -Release Series 3.3 (hardknott) -****************************** - -- :yocto_docs:`BitBake 1.50 User Manual ` - -******************************* -Release Series 3.2 (gatesgarth) -******************************* - -- :yocto_docs:`BitBake 1.48 User Manual ` - -******************************************* -Release Series 3.1 (dunfell first versions) -******************************************* - -- :yocto_docs:`3.1 BitBake User Manual ` -- :yocto_docs:`3.1.1 BitBake User Manual ` -- :yocto_docs:`3.1.2 BitBake User Manual ` -- :yocto_docs:`3.1.3 BitBake User Manual ` - -************************* -Release Series 3.0 (zeus) -************************* - -- :yocto_docs:`3.0 BitBake User Manual ` -- :yocto_docs:`3.0.1 BitBake User Manual ` -- :yocto_docs:`3.0.2 BitBake User Manual ` -- :yocto_docs:`3.0.3 BitBake User Manual ` -- :yocto_docs:`3.0.4 BitBake User Manual ` - -**************************** -Release Series 2.7 (warrior) -**************************** - -- :yocto_docs:`2.7 BitBake User Manual ` -- :yocto_docs:`2.7.1 BitBake User Manual ` -- :yocto_docs:`2.7.2 BitBake User Manual ` -- :yocto_docs:`2.7.3 BitBake User Manual ` -- :yocto_docs:`2.7.4 BitBake User Manual ` - -************************* -Release Series 2.6 (thud) -************************* - -- :yocto_docs:`2.6 BitBake User Manual ` -- :yocto_docs:`2.6.1 BitBake User Manual ` -- :yocto_docs:`2.6.2 BitBake User Manual ` -- :yocto_docs:`2.6.3 BitBake User Manual ` -- :yocto_docs:`2.6.4 BitBake User Manual ` - -************************* -Release Series 2.5 (sumo) -************************* - -- :yocto_docs:`2.5 Documentation ` -- :yocto_docs:`2.5.1 Documentation ` -- :yocto_docs:`2.5.2 Documentation ` -- :yocto_docs:`2.5.3 Documentation ` - -************************** -Release Series 2.4 (rocko) -************************** - -- :yocto_docs:`2.4 BitBake User Manual ` -- :yocto_docs:`2.4.1 BitBake User Manual ` -- :yocto_docs:`2.4.2 BitBake User Manual ` -- :yocto_docs:`2.4.3 BitBake User Manual ` -- :yocto_docs:`2.4.4 BitBake User Manual ` - -************************* -Release Series 2.3 (pyro) -************************* - -- :yocto_docs:`2.3 BitBake User Manual ` -- :yocto_docs:`2.3.1 BitBake User Manual ` -- :yocto_docs:`2.3.2 BitBake User Manual ` -- :yocto_docs:`2.3.3 BitBake User Manual ` -- :yocto_docs:`2.3.4 BitBake User Manual ` - -************************** -Release Series 2.2 (morty) -************************** - -- :yocto_docs:`2.2 BitBake User Manual ` -- :yocto_docs:`2.2.1 BitBake User Manual ` -- :yocto_docs:`2.2.2 BitBake User Manual ` -- :yocto_docs:`2.2.3 BitBake User Manual ` - -**************************** -Release Series 2.1 (krogoth) -**************************** - -- :yocto_docs:`2.1 BitBake User Manual ` -- :yocto_docs:`2.1.1 BitBake User Manual ` -- :yocto_docs:`2.1.2 BitBake User Manual ` -- :yocto_docs:`2.1.3 BitBake User Manual ` - -*************************** -Release Series 2.0 (jethro) -*************************** - -- :yocto_docs:`1.9 BitBake User Manual ` -- :yocto_docs:`2.0 BitBake User Manual ` -- :yocto_docs:`2.0.1 BitBake User Manual ` -- :yocto_docs:`2.0.2 BitBake User Manual ` -- :yocto_docs:`2.0.3 BitBake User Manual ` - -************************* -Release Series 1.8 (fido) -************************* - -- :yocto_docs:`1.8 BitBake User Manual ` -- :yocto_docs:`1.8.1 BitBake User Manual ` -- :yocto_docs:`1.8.2 BitBake User Manual ` - -************************** -Release Series 1.7 (dizzy) -************************** - -- :yocto_docs:`1.7 BitBake User Manual ` -- :yocto_docs:`1.7.1 BitBake User Manual ` -- :yocto_docs:`1.7.2 BitBake User Manual ` -- :yocto_docs:`1.7.3 BitBake User Manual ` - -************************** -Release Series 1.6 (daisy) -************************** - -- :yocto_docs:`1.6 BitBake User Manual ` -- :yocto_docs:`1.6.1 BitBake User Manual ` -- :yocto_docs:`1.6.2 BitBake User Manual ` -- :yocto_docs:`1.6.3 BitBake User Manual ` - diff --git a/bitbake/doc/setversions.py b/bitbake/doc/setversions.py new file mode 100755 index 00000000..f1aea0a4 --- /dev/null +++ b/bitbake/doc/setversions.py @@ -0,0 +1,356 @@ +#!/usr/bin/env python3 +# +# This file defines is used in doc/conf.py to setup the version information for +# the documentation: +# - get_current_version() used in doc/conf.py computes the current version by +# trying to guess the approximate versions we're at using git tags and +# branches from the repository. +# - write_switchers_js() write the switchers.js file used for switching between +# versions of the documentation. +# +# Copyright (c) 2026 Antonin Godard +# +# SPDX-License-Identifier: MIT +# + +import itertools +import json +import os +import re +import subprocess +import sys +import textwrap + +from urllib.request import urlopen, URLError + +# NOTE: the following variables contain default values in case we are not able to fetch +# the releases.json file from https://dashboard.yoctoproject.org/releases.json +DEVBRANCH = "2.18" +LTSSERIES = ["2.8", "2.0"] +ACTIVERELEASES = ["2.16"] + LTSSERIES +YOCTO_MAPPING = { + "2.18": "wrynose", + "2.16": "whinlatter", + "2.8": "scarthgap", + "2.0": "kirkstone", +} + +RELEASES_FROM_JSON = {} + +# Use the local releases.json file if found, fetch it from the dashboard otherwise +releases_json_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "releases.json") +try: + with open(releases_json_path, "r") as f: + RELEASES_FROM_JSON = json.load(f) +except FileNotFoundError: + print("Fetching releases.json from https://dashboard.yoctoproject.org/releases.json...", + file=sys.stderr) + try: + with urlopen("https://dashboard.yoctoproject.org/releases.json") as r, \ + open(releases_json_path, "w") as f: + RELEASES_FROM_JSON = json.load(r) + json.dump(RELEASES_FROM_JSON, f) + except URLError: + print("WARNING: tried to fetch https://dashboard.yoctoproject.org/releases.json " + "but failed, using default values for active releases", file=sys.stderr) + pass + +if RELEASES_FROM_JSON: + ACTIVERELEASES = [] + DEVBRANCH = "" + LTSSERIES = [] + YOCTO_MAPPING = {} + + for release in RELEASES_FROM_JSON: + bb_ver = release["bitbake_version"] + if release["status"] == "Active Development": + DEVBRANCH = bb_ver + if "LTS until" in release["status"]: + LTSSERIES.append(bb_ver) + if release["bitbake_version"]: + YOCTO_MAPPING[bb_ver] = release["release_codename"] + + # Find the first non-dev release, which should be displayed as the default + # page on the docs website. + current_branch = "" + for release in RELEASES_FROM_JSON: + if release["status"] != "Active Development": + current_branch = release["bitbake_version"] + break + + if not current_branch: + sys.exit("Unable to find a current release! Exiting...") + + # make the list of releases unique, there can be duplication when the + # current releases is also an LTS + ACTIVERELEASES = list(dict.fromkeys([current_branch] + LTSSERIES)) + +print(f"ACTIVERELEASES calculated to be {ACTIVERELEASES}", file=sys.stderr) +print(f"DEVBRANCH calculated to be {DEVBRANCH}", file=sys.stderr) +print(f"LTSSERIES calculated to be {LTSSERIES}", file=sys.stderr) + +BB_RELEASE_TAG_RE = re.compile(r"^[0-9]+\.[0-9]+\.[0-9]+$") + +def get_current_version(): + # Test tags exist and inform the user to fetch if not + try: + subprocess.run(["git", "show", f"{LTSSERIES[0]}.0"], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True) + except subprocess.CalledProcessError: + sys.exit("Please run 'git fetch --tags' before building the documentation") + + # Try and figure out what we are + tags = subprocess.run(["git", "tag", "--points-at", "HEAD"], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + universal_newlines=True).stdout + for t in tags.split(): + if re.match(BB_RELEASE_TAG_RE, t): + return t + + # We're floating on a branch + branch = subprocess.run(["git", "branch", "--show-current"], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + universal_newlines=True).stdout.strip() + + if branch == "" or branch not in list(YOCTO_MAPPING.keys()) + ["master", "master-next"]: + # We're not on a known release branch so we have to guess. Compare the + # numbers of commits from each release branch and assume the smallest + # number of commits is the one we're based off + possible_branch = None + branch_count = 0 + for b in itertools.chain(YOCTO_MAPPING.keys(), ["master"]): + result = subprocess.run(["git", "log", "--format=oneline", "HEAD..origin/" + b], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + universal_newlines=True) + if result.returncode == 0: + count = result.stdout.count('\n') + if not possible_branch or count < branch_count: + print("Branch %s has count %s" % (b, count)) + possible_branch = b + branch_count = count + if possible_branch: + branch = possible_branch + else: + branch = "master" + print("Nearest release branch estimated to be %s" % branch) + + if branch == "master": + return "dev" + + if branch == "master-next": + return "next" + + ourversion = branch + head_commit = subprocess.run(["git", "rev-parse", "--short", "HEAD"], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + universal_newlines=True).stdout.strip() + branch_commit = subprocess.run(["git", "rev-parse", "--short", branch], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + universal_newlines=True).stdout.strip() + if head_commit != branch_commit: + ourversion += f" ({head_commit})" + + return ourversion + +def write_switchers_js(js_in, js_out, current_version): + with open(js_in, "r") as r, open(js_out, "w") as w: + lines = r.readlines() + for line in lines: + if "VERSIONS_PLACEHOLDER" in line: + if current_version != "dev": + w.write(" 'dev': 'Unstable (dev)',\n") + for series in ACTIVERELEASES: + w.write(f" '{series}': '{series} ({YOCTO_MAPPING[series]})',\n") + else: + w.write(line) + print("switchers.js generated from switchers.js.in") + +def _release_section(series_version: str, codename: str, bitbake_version: str) -> str: + """ + Helper function to generate a release section, as: + + ******************** + Release Series xxxxx + ******************** + + - + """ + section_length = len(series_version) + len(codename) + 18 + return textwrap.dedent( + f"""\ + {'*' * section_length} + Release Series {series_version} ({codename}) + {'*' * section_length} + + - :yocto_docs:`BitBake {bitbake_version} User Manual ` + + """) + +def write_releases_rst(releases_rst_out: str): + """ + Generates the releases.rst file automatically, based on what is found + in the releases.json file. + """ + with open(releases_rst_out, "w") as f: + f.write(textwrap.dedent("""\ + .. SPDX-License-Identifier: CC-BY-2.5 + + ================================= + BitBake Supported Release Manuals + ================================= + + """)) + + for release in RELEASES_FROM_JSON: + if release["status"] == "Active Development": + continue + + if not release["bitbake_version"]: + continue + + if release["series"] == "current": + f.write(_release_section( + release["series_version"], + release["release_codename"], + release["bitbake_version"])) + + f.write(textwrap.dedent("""\ + ================================ + BitBake Outdated Release Manuals + ================================ + + """)) + + for release in RELEASES_FROM_JSON: + if not release["series"] == "previous": + continue + + if not release["bitbake_version"]: + continue + + f.write(_release_section( + release["series_version"], + release["release_codename"], + release["bitbake_version"])) + + # old legacy links, which cannot be auto-generated + f.write(textwrap.dedent( + """\ + - :yocto_docs:`3.1.2 BitBake User Manual ` + - :yocto_docs:`3.1 BitBake User Manual ` + - :yocto_docs:`3.1.1 BitBake User Manual ` + - :yocto_docs:`3.1.3 BitBake User Manual ` + + ************************* + Release Series 3.0 (Zeus) + ************************* + + - :yocto_docs:`3.0 BitBake User Manual ` + - :yocto_docs:`3.0.1 BitBake User Manual ` + - :yocto_docs:`3.0.2 BitBake User Manual ` + - :yocto_docs:`3.0.3 BitBake User Manual ` + - :yocto_docs:`3.0.4 BitBake User Manual ` + + **************************** + Release Series 2.7 (Warrior) + **************************** + + - :yocto_docs:`2.7 BitBake User Manual ` + - :yocto_docs:`2.7.1 BitBake User Manual ` + - :yocto_docs:`2.7.2 BitBake User Manual ` + - :yocto_docs:`2.7.3 BitBake User Manual ` + - :yocto_docs:`2.7.4 BitBake User Manual ` + + ************************* + Release Series 2.6 (Thud) + ************************* + + - :yocto_docs:`2.6 BitBake User Manual ` + - :yocto_docs:`2.6.1 BitBake User Manual ` + - :yocto_docs:`2.6.2 BitBake User Manual ` + - :yocto_docs:`2.6.3 BitBake User Manual ` + - :yocto_docs:`2.6.4 BitBake User Manual ` + + ************************* + Release Series 2.5 (Sumo) + ************************* + + - :yocto_docs:`2.5 Documentation ` + - :yocto_docs:`2.5.1 Documentation ` + - :yocto_docs:`2.5.2 Documentation ` + - :yocto_docs:`2.5.3 Documentation ` + + ************************** + Release Series 2.4 (Rocko) + ************************** + + - :yocto_docs:`2.4 BitBake User Manual ` + - :yocto_docs:`2.4.1 BitBake User Manual ` + - :yocto_docs:`2.4.2 BitBake User Manual ` + - :yocto_docs:`2.4.3 BitBake User Manual ` + - :yocto_docs:`2.4.4 BitBake User Manual ` + + ************************* + Release Series 2.3 (Pyro) + ************************* + + - :yocto_docs:`2.3 BitBake User Manual ` + - :yocto_docs:`2.3.1 BitBake User Manual ` + - :yocto_docs:`2.3.2 BitBake User Manual ` + - :yocto_docs:`2.3.3 BitBake User Manual ` + - :yocto_docs:`2.3.4 BitBake User Manual ` + + ************************** + Release Series 2.2 (Morty) + ************************** + + - :yocto_docs:`2.2 BitBake User Manual ` + - :yocto_docs:`2.2.1 BitBake User Manual ` + - :yocto_docs:`2.2.2 BitBake User Manual ` + - :yocto_docs:`2.2.3 BitBake User Manual ` + + **************************** + Release Series 2.1 (Krogoth) + **************************** + + - :yocto_docs:`2.1 BitBake User Manual ` + - :yocto_docs:`2.1.1 BitBake User Manual ` + - :yocto_docs:`2.1.2 BitBake User Manual ` + - :yocto_docs:`2.1.3 BitBake User Manual ` + + *************************** + Release Series 2.0 (Jethro) + *************************** + + - :yocto_docs:`1.9 BitBake User Manual ` + - :yocto_docs:`2.0 BitBake User Manual ` + - :yocto_docs:`2.0.1 BitBake User Manual ` + - :yocto_docs:`2.0.2 BitBake User Manual ` + - :yocto_docs:`2.0.3 BitBake User Manual ` + + ************************* + Release Series 1.8 (Fido) + ************************* + + - :yocto_docs:`1.8 BitBake User Manual ` + - :yocto_docs:`1.8.1 BitBake User Manual ` + - :yocto_docs:`1.8.2 BitBake User Manual ` + + ************************** + Release Series 1.7 (Dizzy) + ************************** + + - :yocto_docs:`1.7 BitBake User Manual ` + - :yocto_docs:`1.7.1 BitBake User Manual ` + - :yocto_docs:`1.7.2 BitBake User Manual ` + - :yocto_docs:`1.7.3 BitBake User Manual ` + + ************************** + Release Series 1.6 (Daisy) + ************************** + + - :yocto_docs:`1.6 BitBake User Manual ` + - :yocto_docs:`1.6.1 BitBake User Manual ` + - :yocto_docs:`1.6.2 BitBake User Manual ` + - :yocto_docs:`1.6.3 BitBake User Manual ` + """)) diff --git a/bitbake/doc/sphinx-static/switchers.js b/bitbake/doc/sphinx-static/switchers.js deleted file mode 100644 index 32113cfa..00000000 --- a/bitbake/doc/sphinx-static/switchers.js +++ /dev/null @@ -1,233 +0,0 @@ -(function() { - 'use strict'; - - var all_versions = { - 'dev': 'dev (3.2)', - '3.1.2': '3.1.2', - '3.0.3': '3.0.3', - '2.7.4': '2.7.4', - }; - - var all_doctypes = { - 'single': 'Individual Webpages', - 'mega': "All-in-one 'Mega' Manual", - }; - - // Simple version comparision - // Return 1 if a > b - // Return -1 if a < b - // Return 0 if a == b - function ver_compare(a, b) { - if (a == "dev") { - return 1; - } - - if (a === b) { - return 0; - } - - var a_components = a.split("."); - var b_components = b.split("."); - - var len = Math.min(a_components.length, b_components.length); - - // loop while the components are equal - for (var i = 0; i < len; i++) { - // A bigger than B - if (parseInt(a_components[i]) > parseInt(b_components[i])) { - return 1; - } - - // B bigger than A - if (parseInt(a_components[i]) < parseInt(b_components[i])) { - return -1; - } - } - - // If one's a prefix of the other, the longer one is greater. - if (a_components.length > b_components.length) { - return 1; - } - - if (a_components.length < b_components.length) { - return -1; - } - - // Otherwise they are the same. - return 0; - } - - function build_version_select(current_series, current_version) { - var buf = [''); - return buf.join(''); - } - - function build_doctype_select(current_doctype) { - var buf = [''); - return buf.join(''); - } - - function navigate_to_first_existing(urls) { - // Navigate to the first existing URL in urls. - var url = urls.shift(); - - // Web browsers won't redirect file:// urls to file urls using ajax but - // its useful for local testing - if (url.startsWith("file://")) { - window.location.href = url; - return; - } - - if (urls.length == 0) { - window.location.href = url; - return; - } - $.ajax({ - url: url, - success: function() { - window.location.href = url; - }, - error: function() { - navigate_to_first_existing(urls); - } - }); - } - - function get_docroot_url() { - var url = window.location.href; - var root = DOCUMENTATION_OPTIONS.URL_ROOT; - - var urlarray = url.split('/'); - // Trim off anything after '/' - urlarray.pop(); - var depth = (root.match(/\.\.\//g) || []).length; - for (var i = 0; i < depth; i++) { - urlarray.pop(); - } - - return urlarray.join('/') + '/'; - } - - function on_version_switch() { - var selected_version = $(this).children('option:selected').attr('value'); - var url = window.location.href; - var current_version = DOCUMENTATION_OPTIONS.VERSION; - var docroot = get_docroot_url() - - var new_versionpath = selected_version + '/'; - if (selected_version == "dev") - new_versionpath = ''; - - // dev versions have no version prefix - if (current_version == "dev") { - var new_url = docroot + new_versionpath + url.replace(docroot, ""); - var fallback_url = docroot + new_versionpath; - } else { - var new_url = url.replace('/' + current_version + '/', '/' + new_versionpath); - var fallback_url = new_url.replace(url.replace(docroot, ""), ""); - } - - console.log(get_docroot_url()) - console.log(url + " to url " + new_url); - console.log(url + " to fallback " + fallback_url); - - if (new_url != url) { - navigate_to_first_existing([ - new_url, - fallback_url, - 'https://www.yoctoproject.org/docs/', - ]); - } - } - - function on_doctype_switch() { - var selected_doctype = $(this).children('option:selected').attr('value'); - var url = window.location.href; - if (selected_doctype == 'mega') { - var docroot = get_docroot_url() - var current_version = DOCUMENTATION_OPTIONS.VERSION; - // Assume manuals before 3.2 are using old docbook mega-manual - if (ver_compare(current_version, "3.2") < 0) { - var new_url = docroot + "mega-manual/mega-manual.html"; - } else { - var new_url = docroot + "singleindex.html"; - } - } else { - var new_url = url.replace("singleindex.html", "index.html") - } - - if (new_url != url) { - navigate_to_first_existing([ - new_url, - 'https://www.yoctoproject.org/docs/', - ]); - } - } - - // Returns the current doctype based upon the url - function doctype_segment_from_url(url) { - if (url.includes("singleindex") || url.includes("mega-manual")) - return "mega"; - return "single"; - } - - $(document).ready(function() { - var release = DOCUMENTATION_OPTIONS.VERSION; - var current_doctype = doctype_segment_from_url(window.location.href); - var current_series = release.substr(0, 3); - var version_select = build_version_select(current_series, release); - - $('.version_switcher_placeholder').html(version_select); - $('.version_switcher_placeholder select').bind('change', on_version_switch); - - var doctype_select = build_doctype_select(current_doctype); - - $('.doctype_switcher_placeholder').html(doctype_select); - $('.doctype_switcher_placeholder select').bind('change', on_doctype_switch); - - if (ver_compare(release, "3.1") < 0) { - $('#outdated-warning').html('Version ' + release + ' of the project is now considered obsolete, please select and use a more recent version'); - $('#outdated-warning').css('padding', '.5em'); - } else if (release != "dev") { - $.each(all_versions, function(version, title) { - var series = version.substr(0, 3); - if (series == current_series && version != release) { - $('#outdated-warning').html('This document is for outdated version ' + release + ', you should select the latest release version in this series, ' + version + '.'); - $('#outdated-warning').css('padding', '.5em'); - } - }); - } - }); -})(); diff --git a/bitbake/doc/sphinx-static/switchers.js.in b/bitbake/doc/sphinx-static/switchers.js.in new file mode 100644 index 00000000..0b209e95 --- /dev/null +++ b/bitbake/doc/sphinx-static/switchers.js.in @@ -0,0 +1,249 @@ +// SPDX-License-Identifier: MIT +(function () { + "use strict"; + + var all_versions = { + VERSIONS_PLACEHOLDER, + }; + + var all_doctypes = { + single: "Individual Webpages", + mega: "All-in-one 'Mega' Manual", + }; + + // Simple version comparision + // Return 1 if a > b + // Return -1 if a < b + // Return 0 if a == b + function ver_compare(a, b) { + if (a == "dev") { + return 1; + } + + if (a === b) { + return 0; + } + + var a_components = a.split("."); + var b_components = b.split("."); + + var len = Math.min(a_components.length, b_components.length); + + // loop while the components are equal + for (var i = 0; i < len; i++) { + // A bigger than B + if (parseInt(a_components[i]) > parseInt(b_components[i])) { + return 1; + } + + // B bigger than A + if (parseInt(a_components[i]) < parseInt(b_components[i])) { + return -1; + } + } + + // If one's a prefix of the other, the longer one is greater. + if (a_components.length > b_components.length) { + return 1; + } + + if (a_components.length < b_components.length) { + return -1; + } + + // Otherwise they are the same. + return 0; + } + + function build_version_select(current_version) { + var buf = [""); + return buf.join(""); + } + + function build_doctype_select(current_doctype) { + var buf = [""); + return buf.join(""); + } + + function navigate_to_first_existing(urls) { + // Navigate to the first existing URL in urls. + var url = urls.shift(); + + // Web browsers won't redirect file:// urls to file urls using ajax but + // its useful for local testing + if (url.startsWith("file://")) { + window.location.href = url; + return; + } + + if (urls.length == 0) { + window.location.href = url; + return; + } + $.ajax({ + url: url, + success: function () { + window.location.href = url; + }, + error: function () { + navigate_to_first_existing(urls); + }, + }); + } + + function get_docroot_url() { + var url = window.location.href; + // Try to get the variable from documentation_options.js + var root = DOCUMENTATION_OPTIONS.URL_ROOT; + if (root == null) { + // In recent versions of Sphinx, URL_ROOT was removed from + // documentation_options.js, so get it like searchtools.js does. + root = document.documentElement.dataset.content_root; + } + + var urlarray = url.split("/"); + // Trim off anything after '/' + urlarray.pop(); + var depth = (root.match(/\.\.\//g) || []).length; + for (var i = 0; i < depth; i++) { + urlarray.pop(); + } + + return urlarray.join("/") + "/"; + } + + function on_version_switch() { + var selected_version = $(this).children("option:selected").attr("value"); + var url = window.location.href; + var current_version = DOCUMENTATION_OPTIONS.VERSION; + var docroot = get_docroot_url(); + + var new_versionpath = selected_version + "/"; + + // latest tag is also the default page (without version information) + if (docroot.endsWith("dev/")) { + var new_url = url.replace("/dev/", "/" + new_versionpath); + var fallback_url = new_url.replace(url.replace(docroot, ""), ""); + } else if (docroot.endsWith(current_version + "/") == false) { + var new_url = docroot + new_versionpath + url.replace(docroot, ""); + var fallback_url = docroot + new_versionpath; + } else { + var new_url = url.replace( + "/" + current_version + "/", + "/" + new_versionpath, + ); + var fallback_url = new_url.replace(url.replace(docroot, ""), ""); + } + + console.log(url + " to url " + new_url); + console.log(url + " to fallback " + fallback_url); + + if (new_url != url) { + navigate_to_first_existing([ + new_url, + fallback_url, + "https://www.yoctoproject.org/bitbake/", + ]); + } + } + + function on_doctype_switch() { + var selected_doctype = $(this).children("option:selected").attr("value"); + var url = window.location.href; + if (selected_doctype == "mega") { + var docroot = get_docroot_url(); + var current_version = DOCUMENTATION_OPTIONS.VERSION; + var new_url = docroot + "singleindex.html"; + } else { + var new_url = url.replace("singleindex.html", "index.html"); + } + + if (new_url != url) { + navigate_to_first_existing([ + new_url, + "https://www.yoctoproject.org/docs/", + ]); + } + } + + // Returns the current doctype based upon the url + function doctype_segment_from_url(url) { + if (url.includes("singleindex") || url.includes("mega-manual")) + return "mega"; + return "single"; + } + + $(document).ready(function () { + var release = DOCUMENTATION_OPTIONS.VERSION; + var current_doctype = doctype_segment_from_url(window.location.href); + var current_series = release.substr(0, 3); + var version_select = build_version_select(release); + + $(".version_switcher_placeholder").html(version_select); + $(".version_switcher_placeholder select").bind("change", on_version_switch); + + var doctype_select = build_doctype_select(current_doctype); + + $(".doctype_switcher_placeholder").html(doctype_select); + $(".doctype_switcher_placeholder select").bind("change", on_doctype_switch); + + // if release = "X.Y ()", remove the "()" so that only X.Y is compared + release = release.split(" ")[0]; + if (!(["dev", "next"].includes(release)) && !(release in all_versions)) { + $("#outdated-warning").html( + "Version " + release + " of the project is now considered obsolete, please select and use a more recent version", + ); + $("#outdated-warning").css("padding", ".5em"); + } + }); +})(); diff --git a/bitbake/lib/bb/COW.py b/bitbake/lib/bb/COW.py index 76bc08a3..4af03c54 100644 --- a/bitbake/lib/bb/COW.py +++ b/bitbake/lib/bb/COW.py @@ -36,8 +36,9 @@ class COWDictMeta(COWMeta): __marker__ = tuple() def __str__(cls): - # FIXME: I have magic numbers! - return "" % (cls.__count__, len(cls.__dict__) - 3) + ignored_keys = set(["__count__", "__doc__", "__module__", "__firstlineno__", "__static_attributes__"]) + keys = set(cls.__dict__.keys()) - ignored_keys + return "" % (cls.__count__, len(keys)) __repr__ = __str__ @@ -161,8 +162,9 @@ class COWDictMeta(COWMeta): class COWSetMeta(COWDictMeta): def __str__(cls): - # FIXME: I have magic numbers! - return "" % (cls.__count__, len(cls.__dict__) - 3) + ignored_keys = set(["__count__", "__doc__", "__module__", "__firstlineno__", "__static_attributes__"]) + keys = set(cls.__dict__.keys()) - ignored_keys + return "" % (cls.__count__, len(keys)) __repr__ = __str__ diff --git a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py index 7d9bc147..77edfa2b 100644 --- a/bitbake/lib/bb/__init__.py +++ b/bitbake/lib/bb/__init__.py @@ -9,12 +9,11 @@ # SPDX-License-Identifier: GPL-2.0-only # -__version__ = "2.8.0" +__version__ = "2.18.0" import sys -# It was 3.8.0 originally but set to 3.7.3 for Debian Buster -if sys.version_info < (3, 7, 3): - raise RuntimeError("Sorry, python 3.7.3 or later is required for this version of bitbake") +if sys.version_info < (3, 9, 0): + raise RuntimeError("Sorry, python 3.9.0 or later is required for this version of bitbake") if sys.version_info < (3, 10, 0): # With python 3.8 and 3.9, we see errors of "libgcc_s.so.1 must be installed for pthread_cancel to work" @@ -37,6 +36,35 @@ class BBHandledException(Exception): import os import logging +from collections import namedtuple +import multiprocessing as mp + +# Python 3.14 changes the default multiprocessing context from "fork" to +# "forkserver". However, bitbake heavily relies on "fork" behavior to +# efficiently pass data to the child processes. Places that need this should do: +# from bb import multiprocessing +# in place of +# import multiprocessing + +class MultiprocessingContext(object): + """ + Multiprocessing proxy object that uses the "fork" context for a property if + available, otherwise goes to the main multiprocessing module. This allows + it to be a drop-in replacement for the multiprocessing module, but use the + fork context + """ + def __init__(self): + super().__setattr__("_ctx", mp.get_context("fork")) + + def __getattr__(self, name): + if hasattr(self._ctx, name): + return getattr(self._ctx, name) + return getattr(mp, name) + + def __setattr__(self, name, value): + raise AttributeError(f"Unable to set attribute {name}") + +multiprocessing = MultiprocessingContext() class NullHandler(logging.Handler): @@ -104,26 +132,6 @@ class BBLoggerAdapter(logging.LoggerAdapter, BBLoggerMixin): self.setup_bblogger(logger.name) super().__init__(logger, *args, **kwargs) - if sys.version_info < (3, 6): - # These properties were added in Python 3.6. Add them in older versions - # for compatibility - @property - def manager(self): - return self.logger.manager - - @manager.setter - def manager(self, value): - self.logger.manager = value - - @property - def name(self): - return self.logger.name - - def __repr__(self): - logger = self.logger - level = logger.getLevelName(logger.getEffectiveLevel()) - return '<%s %s (%s)>' % (self.__class__.__name__, logger.name, level) - logging.LoggerAdapter = BBLoggerAdapter logger = logging.getLogger("BitBake") @@ -149,9 +157,25 @@ sys.modules['bb.fetch'] = sys.modules['bb.fetch2'] # Messaging convenience functions def plain(*args): + """ + Prints a message at "plain" level (higher level than a ``bb.note()``). + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.plain(''.join(args)) def debug(lvl, *args): + """ + Prints a debug message. + + Arguments: + + - ``lvl``: debug level. Higher value increases the debug level + (determined by ``bitbake -D``). + - ``args``: one or more strings to print. + """ if isinstance(lvl, str): mainlogger.warning("Passed invalid debug level '%s' to bb.debug", lvl) args = (lvl,) + args @@ -159,33 +183,81 @@ def debug(lvl, *args): mainlogger.bbdebug(lvl, ''.join(args)) def note(*args): + """ + Prints a message at "note" level. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.info(''.join(args)) -# -# A higher prioity note which will show on the console but isn't a warning -# -# Something is happening the user should be aware of but they probably did -# something to make it happen -# def verbnote(*args): + """ + A higher priority note which will show on the console but isn't a warning. + + Use in contexts when something is happening the user should be aware of but + they probably did something to make it happen. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.verbnote(''.join(args)) # # Warnings - things the user likely needs to pay attention to and fix # def warn(*args): + """ + Prints a warning message. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.warning(''.join(args)) def warnonce(*args): + """ + Prints a warning message like ``bb.warn()``, but only prints the message + once. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.warnonce(''.join(args)) def error(*args, **kwargs): + """ + Prints an error message. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.error(''.join(args), extra=kwargs) def erroronce(*args): + """ + Prints an error message like ``bb.error()``, but only prints the message + once. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.erroronce(''.join(args)) def fatal(*args, **kwargs): + """ + Prints an error message and stops the BitBake execution. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.critical(''.join(args), extra=kwargs) raise BBHandledException() @@ -214,7 +286,6 @@ def deprecated(func, name=None, advice=""): # For compatibility def deprecate_import(current, modulename, fromlist, renames = None): """Import objects from one module into another, wrapping them with a DeprecationWarning""" - import sys module = __import__(modulename, fromlist = fromlist) for position, objname in enumerate(fromlist): @@ -228,3 +299,14 @@ def deprecate_import(current, modulename, fromlist, renames = None): setattr(sys.modules[current], newname, newobj) +TaskData = namedtuple("TaskData", [ + "pn", + "taskname", + "fn", + "deps", + "provides", + "taskhash", + "unihash", + "hashfn", + "taskhash_deps", +]) diff --git a/bitbake/lib/bb/acl.py b/bitbake/lib/bb/acl.py index 0f41b275..e9dbdb61 100755 --- a/bitbake/lib/bb/acl.py +++ b/bitbake/lib/bb/acl.py @@ -195,8 +195,6 @@ class ACL(object): def main(): import argparse - import pwd - import grp from pathlib import Path parser = argparse.ArgumentParser() diff --git a/bitbake/lib/bb/asyncrpc/__init__.py b/bitbake/lib/bb/asyncrpc/__init__.py index 639e1607..a4371643 100644 --- a/bitbake/lib/bb/asyncrpc/__init__.py +++ b/bitbake/lib/bb/asyncrpc/__init__.py @@ -5,7 +5,7 @@ # -from .client import AsyncClient, Client, ClientPool +from .client import AsyncClient, Client from .serv import AsyncServer, AsyncServerConnection from .connection import DEFAULT_MAX_CHUNK from .exceptions import ( diff --git a/bitbake/lib/bb/asyncrpc/client.py b/bitbake/lib/bb/asyncrpc/client.py index a350b4fb..dd28fa73 100644 --- a/bitbake/lib/bb/asyncrpc/client.py +++ b/bitbake/lib/bb/asyncrpc/client.py @@ -7,15 +7,19 @@ import abc import asyncio import json +import logging import os import socket import sys import re import contextlib +import weakref from threading import Thread from .connection import StreamConnection, WebsocketConnection, DEFAULT_MAX_CHUNK from .exceptions import ConnectionClosedError, InvokeError +logger = logging.getLogger("bb.asyncrpc.client") + UNIX_PREFIX = "unix://" WS_PREFIX = "ws://" WSS_PREFIX = "wss://" @@ -24,6 +28,12 @@ ADDR_TYPE_UNIX = 0 ADDR_TYPE_TCP = 1 ADDR_TYPE_WS = 2 +WEBSOCKETS_MIN_VERSION = (9, 1) +# Need websockets 10 with python 3.10+ +if sys.version_info >= (3, 10, 0): + WEBSOCKETS_MIN_VERSION = (10, 0) + + def parse_address(addr): if addr.startswith(UNIX_PREFIX): return (ADDR_TYPE_UNIX, (addr[len(UNIX_PREFIX) :],)) @@ -39,6 +49,7 @@ def parse_address(addr): return (ADDR_TYPE_TCP, (host, int(port))) + class AsyncClient(object): def __init__( self, @@ -86,8 +97,35 @@ class AsyncClient(object): async def connect_websocket(self, uri): import websockets + try: + version = tuple( + int(v) + for v in websockets.__version__.split(".")[ + 0 : len(WEBSOCKETS_MIN_VERSION) + ] + ) + except ValueError: + raise ImportError( + f"Unable to parse websockets version '{websockets.__version__}'" + ) + + if version < WEBSOCKETS_MIN_VERSION: + min_ver_str = ".".join(str(v) for v in WEBSOCKETS_MIN_VERSION) + raise ImportError( + f"Websockets version {websockets.__version__} is less than minimum required version {min_ver_str}" + ) + async def connect_sock(): - websocket = await websockets.connect(uri, ping_interval=None) + try: + websocket = await websockets.connect( + uri, + ping_interval=None, + open_timeout=self.timeout, + ) + except asyncio.exceptions.TimeoutError: + raise ConnectionError("Timeout while connecting to websocket") + except (OSError, websockets.InvalidHandshake, websockets.InvalidURI) as exc: + raise ConnectionError(f"Could not connect to websocket: {exc}") from exc return WebsocketConnection(websocket, self.timeout) self._connect_sock = connect_sock @@ -190,8 +228,49 @@ class Client(object): # required (but harmless) with it. asyncio.set_event_loop(self.loop) + # The loop and its transports must be closed even if the caller never + # calls close(). Since set_event_loop() above replaces the reference + # held by the previous client, an unclosed loop only becomes reachable + # for collection once another client is created, at which point + # asyncio reports "unclosed transport" and "unclosed event loop" + # ResourceWarnings. The finalizer is detached by close() so that the + # normal path is unaffected. + self._finalizer = weakref.finalize( + self, self._close_loop, self.loop, self.client + ) + self._add_methods("connect_tcp", "ping") + @staticmethod + def _close_loop(loop, client): + if loop.is_closed(): + return + try: + loop.run_until_complete(client.close()) + loop.run_until_complete(loop.shutdown_asyncgens()) + # This can be called from the finalizer, where an exception would be + # discarded by the interpreter and reported only as "Exception ignored + # in", so report the error here instead. The loop is closed below in + # every case. + except (OSError, ConnectionClosedError, asyncio.TimeoutError) as exc: + # The peer has gone away or is not responding. That is expected + # for a client that was never closed, so it is not worth warning + # about. asyncio.TimeoutError is only distinct from OSError on + # python older than 3.11. + logger.debug("Client connection already closed or unreachable: %s" % exc) + except RuntimeError as exc: + # The loop could not be run, for example because another loop is + # already running in this thread. Unlike the above, this means + # something is wrong with how the client is being used. + logger.warning("Could not shut down client event loop: %s" % exc) + except Exception as exc: + # Also covers the exceptions raised by websockets, which cannot be + # named here as it is imported lazily in connect_websocket(), and + # whatever may be raised while the interpreter is shutting down. + logger.warning("Error shutting down client connection: %s" % exc) + finally: + loop.close() + @abc.abstractmethod def _get_async_client(self): pass @@ -224,86 +303,9 @@ class Client(object): def close(self): if self.loop: - self.loop.run_until_complete(self.client.close()) - if sys.version_info >= (3, 6): - self.loop.run_until_complete(self.loop.shutdown_asyncgens()) - self.loop.close() - self.loop = None - - def __enter__(self): - return self - - def __exit__(self, exc_type, exc_value, traceback): - self.close() - return False - - -class ClientPool(object): - def __init__(self, max_clients): - self.avail_clients = [] - self.num_clients = 0 - self.max_clients = max_clients + self._finalizer.detach() + self._close_loop(self.loop, self.client) self.loop = None - self.client_condition = None - - @abc.abstractmethod - async def _new_client(self): - raise NotImplementedError("Must be implemented in derived class") - - def close(self): - if self.client_condition: - self.client_condition = None - - if self.loop: - self.loop.run_until_complete(self.__close_clients()) - self.loop.run_until_complete(self.loop.shutdown_asyncgens()) - self.loop.close() - self.loop = None - - def run_tasks(self, tasks): - if not self.loop: - self.loop = asyncio.new_event_loop() - - thread = Thread(target=self.__thread_main, args=(tasks,)) - thread.start() - thread.join() - - @contextlib.asynccontextmanager - async def get_client(self): - async with self.client_condition: - if self.avail_clients: - client = self.avail_clients.pop() - elif self.num_clients < self.max_clients: - self.num_clients += 1 - client = await self._new_client() - else: - while not self.avail_clients: - await self.client_condition.wait() - client = self.avail_clients.pop() - - try: - yield client - finally: - async with self.client_condition: - self.avail_clients.append(client) - self.client_condition.notify() - - def __thread_main(self, tasks): - async def process_task(task): - async with self.get_client() as client: - await task(client) - - asyncio.set_event_loop(self.loop) - if not self.client_condition: - self.client_condition = asyncio.Condition() - tasks = [process_task(t) for t in tasks] - self.loop.run_until_complete(asyncio.gather(*tasks)) - - async def __close_clients(self): - for c in self.avail_clients: - await c.close() - self.avail_clients = [] - self.num_clients = 0 def __enter__(self): return self diff --git a/bitbake/lib/bb/asyncrpc/serv.py b/bitbake/lib/bb/asyncrpc/serv.py index a66117ac..bd1aded8 100644 --- a/bitbake/lib/bb/asyncrpc/serv.py +++ b/bitbake/lib/bb/asyncrpc/serv.py @@ -11,7 +11,7 @@ import os import signal import socket import sys -import multiprocessing +from bb import multiprocessing import logging from .connection import StreamConnection, WebsocketConnection from .exceptions import ClientError, ServerError, ConnectionClosedError, InvokeError @@ -138,14 +138,20 @@ class StreamServer(object): class TCPStreamServer(StreamServer): - def __init__(self, host, port, handler, logger): + def __init__(self, host, port, handler, logger, *, reuseport=False): super().__init__(handler, logger) self.host = host self.port = port + self.reuseport = reuseport def start(self, loop): self.server = loop.run_until_complete( - asyncio.start_server(self.handle_stream_client, self.host, self.port) + asyncio.start_server( + self.handle_stream_client, + self.host, + self.port, + reuse_port=self.reuseport, + ) ) for s in self.server.sockets: @@ -205,15 +211,19 @@ class UnixStreamServer(StreamServer): self.server.close() def cleanup(self): - os.unlink(self.path) + try: + os.unlink(self.path) + except FileNotFoundError: + pass class WebsocketsServer(object): - def __init__(self, host, port, handler, logger): + def __init__(self, host, port, handler, logger, *, reuseport=False): self.host = host self.port = port self.handler = handler self.logger = logger + self.reuseport = reuseport def start(self, loop): import websockets.server @@ -224,6 +234,7 @@ class WebsocketsServer(object): self.host, self.port, ping_interval=None, + reuse_port=self.reuseport, ) ) @@ -262,14 +273,26 @@ class AsyncServer(object): self.loop = None self.run_tasks = [] - def start_tcp_server(self, host, port): - self.server = TCPStreamServer(host, port, self._client_handler, self.logger) + def start_tcp_server(self, host, port, *, reuseport=False): + self.server = TCPStreamServer( + host, + port, + self._client_handler, + self.logger, + reuseport=reuseport, + ) def start_unix_server(self, path): self.server = UnixStreamServer(path, self._client_handler, self.logger) - def start_websocket_server(self, host, port): - self.server = WebsocketsServer(host, port, self._client_handler, self.logger) + def start_websocket_server(self, host, port, reuseport=False): + self.server = WebsocketsServer( + host, + port, + self._client_handler, + self.logger, + reuseport=reuseport, + ) async def _client_handler(self, socket): address = socket.address @@ -368,8 +391,7 @@ class AsyncServer(object): self._serve_forever(tasks) - if sys.version_info >= (3, 6): - self.loop.run_until_complete(self.loop.shutdown_asyncgens()) + self.loop.run_until_complete(self.loop.shutdown_asyncgens()) self.loop.close() queue = multiprocessing.Queue() diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py index 44d08f5c..40839a81 100644 --- a/bitbake/lib/bb/build.py +++ b/bitbake/lib/bb/build.py @@ -197,6 +197,8 @@ def exec_func(func, d, dirs = None): for cdir in d.expand(cleandirs).split(): bb.utils.remove(cdir, True) bb.utils.mkdirhier(cdir) + if cdir == oldcwd: + os.chdir(cdir) if flags and dirs is None: dirs = flags.get('dirs') @@ -395,7 +397,7 @@ def create_progress_handler(func, progress, logfile, d): # Use specified regex return bb.progress.OutOfProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile) elif progress.startswith("custom:"): - # Use a custom progress handler that was injected via OE_EXTRA_IMPORTS or __builtins__ + # Use a custom progress handler that was injected via other means import functools from types import ModuleType @@ -741,7 +743,7 @@ def _exec_task(fn, task, d, quieterr): if quieterr: if not handled: - logger.warning(repr(exc)) + logger.warning(str(exc)) event.fire(TaskFailedSilent(task, fn, logfn, localdata), localdata) else: errprinted = errchk.triggered @@ -750,7 +752,7 @@ def _exec_task(fn, task, d, quieterr): if verboseStdoutLogging or handled: errprinted = True if not handled: - logger.error(repr(exc)) + logger.error(str(exc)) event.fire(TaskFailed(task, fn, logfn, localdata, errprinted), localdata) return 1 @@ -930,9 +932,13 @@ def add_tasks(tasklist, d): # don't assume holding a reference d.setVar('_task_deps', task_deps) +def ensure_task_prefix(name): + if name[:3] != "do_": + name = "do_" + name + return name + def addtask(task, before, after, d): - if task[:3] != "do_": - task = "do_" + task + task = ensure_task_prefix(task) d.setVarFlag(task, "task", 1) bbtasks = d.getVar('__BBTASKS', False) or [] @@ -944,19 +950,20 @@ def addtask(task, before, after, d): if after is not None: # set up deps for function for entry in after.split(): + entry = ensure_task_prefix(entry) if entry not in existing: existing.append(entry) d.setVarFlag(task, "deps", existing) if before is not None: # set up things that depend on this func for entry in before.split(): + entry = ensure_task_prefix(entry) existing = d.getVarFlag(entry, "deps", False) or [] if task not in existing: d.setVarFlag(entry, "deps", [task] + existing) def deltask(task, d): - if task[:3] != "do_": - task = "do_" + task + task = ensure_task_prefix(task) bbtasks = d.getVar('__BBTASKS', False) or [] if task in bbtasks: @@ -1021,3 +1028,9 @@ def tasksbetween(task_start, task_end, d): chain.pop() follow_chain(task_start, task_end) return outtasks + +def listtasks(d): + """ + Return the list of tasks in the current recipe. + """ + return tuple(d.getVar('__BBTASKS', False) or ()) diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py index 18d5574a..2361c568 100644 --- a/bitbake/lib/bb/cache.py +++ b/bitbake/lib/bb/cache.py @@ -28,7 +28,7 @@ import shutil logger = logging.getLogger("BitBake.Cache") -__cache_version__ = "155" +__cache_version__ = "156" def getCacheFile(path, filename, mc, data_hash): mcspec = '' @@ -395,7 +395,7 @@ class Cache(object): # It will be used later for deciding whether we # need extra cache file dump/load support self.mc = mc - self.logger = PrefixLoggerAdapter("Cache: %s: " % (mc if mc else "default"), logger) + self.logger = PrefixLoggerAdapter("Cache: %s: " % (mc if mc else ''), logger) self.caches_array = caches_array self.cachedir = self.data.getVar("CACHE") self.clean = set() @@ -441,7 +441,7 @@ class Cache(object): else: symlink = os.path.join(self.cachedir, "bb_cache.dat") - if os.path.exists(symlink): + if os.path.exists(symlink) or os.path.islink(symlink): bb.utils.remove(symlink) try: os.symlink(os.path.basename(self.cachefile), symlink) @@ -779,25 +779,6 @@ class MulticonfigCache(Mapping): for k in self.__caches: yield k -def init(cooker): - """ - The Objective: Cache the minimum amount of data possible yet get to the - stage of building packages (i.e. tryBuild) without reparsing any .bb files. - - To do this, we intercept getVar calls and only cache the variables we see - being accessed. We rely on the cache getVar calls being made for all - variables bitbake might need to use to reach this stage. For each cached - file we need to track: - - * Its mtime - * The mtimes of all its dependencies - * Whether it caused a parse.SkipRecipe exception - - Files causing parsing errors are evicted from the cache. - - """ - return Cache(cooker.configuration.data, cooker.configuration.data_hash) - class CacheData(object): """ @@ -866,6 +847,16 @@ class MultiProcessCache(object): data = [{}] return data + def clear_cache(self): + if not self.cachefile: + bb.fatal("Can't clear invalid cachefile") + + self.cachedata = self.create_cachedata() + self.cachedata_extras = self.create_cachedata() + with bb.utils.fileslocked([self.cachefile + ".lock"]): + bb.utils.remove(self.cachefile) + bb.utils.remove(self.cachefile + "-*") + def save_extras(self): if not self.cachefile: return diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py index 557793d3..3fb39a30 100644 --- a/bitbake/lib/bb/checksum.py +++ b/bitbake/lib/bb/checksum.py @@ -142,3 +142,28 @@ class FileChecksumCache(MultiProcessCache): checksums.sort(key=operator.itemgetter(1)) return checksums + +class RevisionsCache(MultiProcessCache): + cache_file_name = "local_srcrevisions.dat" + CACHE_VERSION = 1 + + def __init__(self): + MultiProcessCache.__init__(self) + + def get_revs(self): + return self.cachedata[0] + + def get_rev(self, k): + if k in self.cachedata_extras[0]: + return self.cachedata_extras[0][k] + if k in self.cachedata[0]: + return self.cachedata[0][k] + return None + + def set_rev(self, k, v): + self.cachedata[0][k] = v + self.cachedata_extras[0][k] = v + + def merge_data(self, source, dest): + for h in source[0]: + dest[0][h] = source[0][h] diff --git a/bitbake/lib/bb/codeparser.py b/bitbake/lib/bb/codeparser.py index 2e8b7ced..4f70cf7f 100644 --- a/bitbake/lib/bb/codeparser.py +++ b/bitbake/lib/bb/codeparser.py @@ -69,12 +69,25 @@ def add_module_functions(fn, functions, namespace): name = "%s.%s" % (namespace, f) parser = PythonParser(name, logger) try: - parser.parse_python(None, filename=fn, lineno=1, fixedhash=fixedhash+f) + parser.parse_python(None, filename=fn, lineno=1, fixedhash=fixedhash+f, func=functions[f]) #bb.warn("Cached %s" % f) except KeyError: - lines, lineno = inspect.getsourcelines(functions[f]) + try: + targetfn = inspect.getsourcefile(functions[f]) + except TypeError: + # Builtin + continue + if fn != targetfn: + # Skip references to other modules outside this file + #bb.warn("Skipping %s" % name) + continue + try: + lines, lineno = inspect.getsourcelines(functions[f]) + except TypeError: + # Builtin + continue src = "".join(lines) - parser.parse_python(src, filename=fn, lineno=lineno, fixedhash=fixedhash+f) + parser.parse_python(src, filename=fn, lineno=lineno, fixedhash=fixedhash+f, func=functions[f]) #bb.warn("Not cached %s" % f) execs = parser.execs.copy() # Expand internal module exec references @@ -82,14 +95,17 @@ def add_module_functions(fn, functions, namespace): if e in functions: execs.remove(e) execs.add(namespace + "." + e) - modulecode_deps[name] = [parser.references.copy(), execs, parser.var_execs.copy(), parser.contains.copy()] + visitorcode = None + if hasattr(functions[f], 'visitorcode'): + visitorcode = getattr(functions[f], "visitorcode") + modulecode_deps[name] = [parser.references.copy(), execs, parser.var_execs.copy(), parser.contains.copy(), parser.extra, visitorcode] #bb.warn("%s: %s\nRefs:%s Execs: %s %s %s" % (name, fn, parser.references, parser.execs, parser.var_execs, parser.contains)) def update_module_dependencies(d): for mod in modulecode_deps: excludes = set((d.getVarFlag(mod, "vardepsexclude") or "").split()) if excludes: - modulecode_deps[mod] = [modulecode_deps[mod][0] - excludes, modulecode_deps[mod][1] - excludes, modulecode_deps[mod][2] - excludes, modulecode_deps[mod][3]] + modulecode_deps[mod] = [modulecode_deps[mod][0] - excludes, modulecode_deps[mod][1] - excludes, modulecode_deps[mod][2] - excludes, modulecode_deps[mod][3], modulecode_deps[mod][4], modulecode_deps[mod][5]] # A custom getstate/setstate using tuples is actually worth 15% cachesize by # avoiding duplication of the attribute names! @@ -112,21 +128,22 @@ class SetCache(object): codecache = SetCache() class pythonCacheLine(object): - def __init__(self, refs, execs, contains): + def __init__(self, refs, execs, contains, extra): self.refs = codecache.internSet(refs) self.execs = codecache.internSet(execs) self.contains = {} for c in contains: self.contains[c] = codecache.internSet(contains[c]) + self.extra = extra def __getstate__(self): - return (self.refs, self.execs, self.contains) + return (self.refs, self.execs, self.contains, self.extra) def __setstate__(self, state): - (refs, execs, contains) = state - self.__init__(refs, execs, contains) + (refs, execs, contains, extra) = state + self.__init__(refs, execs, contains, extra) def __hash__(self): - l = (hash(self.refs), hash(self.execs)) + l = (hash(self.refs), hash(self.execs), hash(self.extra)) for c in sorted(self.contains.keys()): l = l + (c, hash(self.contains[c])) return hash(l) @@ -155,7 +172,7 @@ class CodeParserCache(MultiProcessCache): # so that an existing cache gets invalidated. Additionally you'll need # to increment __cache_version__ in cache.py in order to ensure that old # recipe caches don't trigger "Taskhash mismatch" errors. - CACHE_VERSION = 11 + CACHE_VERSION = 14 def __init__(self): MultiProcessCache.__init__(self) @@ -169,8 +186,8 @@ class CodeParserCache(MultiProcessCache): self.pythoncachelines = {} self.shellcachelines = {} - def newPythonCacheLine(self, refs, execs, contains): - cacheline = pythonCacheLine(refs, execs, contains) + def newPythonCacheLine(self, refs, execs, contains, extra): + cacheline = pythonCacheLine(refs, execs, contains, extra) h = hash(cacheline) if h in self.pythoncachelines: return self.pythoncachelines[h] @@ -255,7 +272,15 @@ class PythonParser(): def visit_Call(self, node): name = self.called_node_name(node.func) - if name and (name.endswith(self.getvars) or name.endswith(self.getvarflags) or name in self.containsfuncs or name in self.containsanyfuncs): + if name and name in modulecode_deps and modulecode_deps[name][5]: + visitorcode = modulecode_deps[name][5] + contains, execs, warn = visitorcode(name, node.args) + for i in contains: + self.contains[i] = contains[i] + self.execs |= execs + if warn: + self.warn(node.func, warn) + elif name and (name.endswith(self.getvars) or name.endswith(self.getvarflags) or name in self.containsfuncs or name in self.containsanyfuncs): if isinstance(node.args[0], ast.Constant) and isinstance(node.args[0].value, str): varname = node.args[0].value if name in self.containsfuncs and isinstance(node.args[1], ast.Constant): @@ -323,7 +348,7 @@ class PythonParser(): # For the python module code it is expensive to have the function text so it is # uses a different fixedhash to cache against. We can take the hit on obtaining the # text if it isn't in the cache. - def parse_python(self, node, lineno=0, filename="", fixedhash=None): + def parse_python(self, node, lineno=0, filename="", fixedhash=None, func=None): if not fixedhash and (not node or not node.strip()): return @@ -338,6 +363,7 @@ class PythonParser(): self.contains = {} for i in codeparsercache.pythoncache[h].contains: self.contains[i] = set(codeparsercache.pythoncache[h].contains[i]) + self.extra = codeparsercache.pythoncache[h].extra return if h in codeparsercache.pythoncacheextras: @@ -346,6 +372,7 @@ class PythonParser(): self.contains = {} for i in codeparsercache.pythoncacheextras[h].contains: self.contains[i] = set(codeparsercache.pythoncacheextras[h].contains[i]) + self.extra = codeparsercache.pythoncacheextras[h].extra return if fixedhash and not node: @@ -363,9 +390,16 @@ class PythonParser(): if n.__class__.__name__ == "Call": self.visit_Call(n) + if func is not None: + self.references |= getattr(func, "bb_vardeps", set()) + self.references -= getattr(func, "bb_vardepsexclude", set()) + self.execs.update(self.var_execs) + self.extra = None + if fixedhash: + self.extra = bbhash(str(node)) - codeparsercache.pythoncacheextras[h] = codeparsercache.newPythonCacheLine(self.references, self.execs, self.contains) + codeparsercache.pythoncacheextras[h] = codeparsercache.newPythonCacheLine(self.references, self.execs, self.contains, self.extra) class ShellParser(): def __init__(self, name, log): @@ -484,19 +518,34 @@ class ShellParser(): """ words = list(words) - for word in list(words): + for word in words: wtree = pyshlex.make_wordtree(word[1]) for part in wtree: if not isinstance(part, list): continue - if part[0] in ('`', '$('): - command = pyshlex.wordtree_as_string(part[1:-1]) - self._parse_shell(command) - - if word[0] in ("cmd_name", "cmd_word"): - if word in words: - words.remove(word) + candidates = [part] + + # If command is of type: + # + # var="... $(cmd [...]) ..." + # + # Then iterate on what's between the quotes and if we find a + # list, make that what we check for below. + if len(part) >= 3 and part[0] == '"': + for p in part[1:-1]: + if isinstance(p, list): + candidates.append(p) + + for candidate in candidates: + if len(candidate) >= 2: + if candidate[0] in ('`', '$('): + command = pyshlex.wordtree_as_string(candidate[1:-1]) + self._parse_shell(command) + + if word[0] in ("cmd_name", "cmd_word"): + if word in words: + words.remove(word) usetoken = False for word in words: diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py index 1fcb9bf1..59a979ee 100644 --- a/bitbake/lib/bb/command.py +++ b/bitbake/lib/bb/command.py @@ -24,6 +24,7 @@ import io import bb.event import bb.cooker import bb.remotedata +import bb.parse class DataStoreConnectionHandle(object): def __init__(self, dsindex=0): @@ -108,7 +109,7 @@ class Command: def runAsyncCommand(self, _, process_server, halt): try: - if self.cooker.state in (bb.cooker.state.error, bb.cooker.state.shutdown, bb.cooker.state.forceshutdown): + if self.cooker.state in (bb.cooker.State.ERROR, bb.cooker.State.SHUTDOWN, bb.cooker.State.FORCE_SHUTDOWN): # updateCache will trigger a shutdown of the parser # and then raise BBHandledException triggering an exit self.cooker.updateCache() @@ -118,7 +119,7 @@ class Command: (command, options) = cmd commandmethod = getattr(CommandsAsync, command) needcache = getattr( commandmethod, "needcache" ) - if needcache and self.cooker.state != bb.cooker.state.running: + if needcache and self.cooker.state != bb.cooker.State.RUNNING: self.cooker.updateCache() return True else: @@ -142,14 +143,14 @@ class Command: return bb.server.process.idleFinish(traceback.format_exc()) def finishAsyncCommand(self, msg=None, code=None): + self.cooker.finishcommand() + self.process_server.clear_async_cmd() if msg or msg == "": bb.event.fire(CommandFailed(msg), self.cooker.data) elif code: bb.event.fire(CommandExit(code), self.cooker.data) else: bb.event.fire(CommandCompleted(), self.cooker.data) - self.cooker.finishcommand() - self.process_server.clear_async_cmd() def reset(self): if self.remotedatastores: @@ -310,7 +311,7 @@ class CommandsSync: def revalidateCaches(self, command, params): """Called by UI clients when metadata may have changed""" command.cooker.revalidateCaches() - parseConfiguration.needconfig = False + revalidateCaches.needconfig = False def getRecipes(self, command, params): try: @@ -420,15 +421,30 @@ class CommandsSync: return command.cooker.recipecaches[mc].pkg_dp getDefaultPreference.readonly = True + def getSkippedRecipes(self, command, params): + """ + Get the map of skipped recipes for the specified multiconfig/mc name (`params[0]`). + + Invoked by `bb.tinfoil.Tinfoil.get_skipped_recipes` + + :param command: Internally used parameter. + :param params: Parameter array. params[0] is multiconfig/mc name. If not given, then default mc '' is assumed. + :return: Dict whose keys are virtualfns and values are `bb.cooker.SkippedPackage` + """ + try: + mc = params[0] + except IndexError: + mc = '' + # Return list sorted by reverse priority order import bb.cache def sortkey(x): vfn, _ = x - realfn, _, mc = bb.cache.virtualfn2realfn(vfn) - return (-command.cooker.collections[mc].calc_bbfile_priority(realfn)[0], vfn) + realfn, _, item_mc = bb.cache.virtualfn2realfn(vfn) + return -command.cooker.collections[item_mc].calc_bbfile_priority(realfn)[0], vfn - skipdict = OrderedDict(sorted(command.cooker.skiplist.items(), key=sortkey)) + skipdict = OrderedDict(sorted(command.cooker.skiplist_by_mc[mc].items(), key=sortkey)) return list(skipdict.items()) getSkippedRecipes.readonly = True @@ -582,6 +598,13 @@ class CommandsSync: return DataStoreConnectionHandle(idx) parseRecipeFile.readonly = True + def finalizeData(self, command, params): + newdata = command.cooker.data.createCopy() + bb.data.expandKeys(newdata) + bb.parse.ast.runAnonFuncs(newdata) + idx = command.remotedatastores.store(newdata) + return DataStoreConnectionHandle(idx) + class CommandsAsync: """ A class of asynchronous commands diff --git a/bitbake/lib/bb/compress/lz4.py b/bitbake/lib/bb/compress/lz4.py index 88b09893..2a64681c 100644 --- a/bitbake/lib/bb/compress/lz4.py +++ b/bitbake/lib/bb/compress/lz4.py @@ -13,7 +13,7 @@ def open(*args, **kwargs): class LZ4File(bb.compress._pipecompress.PipeFile): def get_compress(self): - return ["lz4c", "-z", "-c"] + return ["lz4", "-z", "-c"] def get_decompress(self): - return ["lz4c", "-d", "-c"] + return ["lz4", "-d", "-c"] diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py index 701cf51b..4b6ba319 100644 --- a/bitbake/lib/bb/cooker.py +++ b/bitbake/lib/bb/cooker.py @@ -8,11 +8,11 @@ # # SPDX-License-Identifier: GPL-2.0-only # - +import enum import sys, os, glob, os.path, re, time import itertools import logging -import multiprocessing +from bb import multiprocessing import threading from io import StringIO, UnsupportedOperation from contextlib import closing @@ -26,6 +26,7 @@ import json import pickle import codecs import hashserv +import ctypes logger = logging.getLogger("BitBake") collectlog = logging.getLogger("BitBake.Collection") @@ -48,16 +49,15 @@ class CollectionError(bb.BBHandledException): Exception raised when layer configuration is incorrect """ -class state: - initial, parsing, running, shutdown, forceshutdown, stopped, error = list(range(7)) - @classmethod - def get_name(cls, code): - for name in dir(cls): - value = getattr(cls, name) - if type(value) == type(cls.initial) and value == code: - return name - raise ValueError("Invalid status code: %s" % code) +class State(enum.Enum): + INITIAL = 0, + PARSING = 1, + RUNNING = 2, + SHUTDOWN = 3, + FORCE_SHUTDOWN = 4, + STOPPED = 5, + ERROR = 6 class SkippedPackage: @@ -79,7 +79,7 @@ class SkippedPackage: class CookerFeatures(object): - _feature_list = [HOB_EXTRA_CACHES, BASEDATASTORE_TRACKING, SEND_SANITYEVENTS, RECIPE_SIGGEN_INFO] = list(range(4)) + _feature_list = [HOB_EXTRA_CACHES, BASEDATASTORE_TRACKING, SEND_SANITYEVENTS, RECIPE_SIGGEN_INFO, SKIP_FRAGMENTS] = list(range(5)) def __init__(self): self._features=set() @@ -90,6 +90,11 @@ class CookerFeatures(object): return self._features.add(f) + def delFeature(self, f): + if f not in self._features: + return + self._features.remove(f) + def __contains__(self, f): return f in self._features @@ -134,7 +139,8 @@ class BBCooker: self.baseconfig_valid = False self.parsecache_valid = False self.eventlog = None - self.skiplist = {} + # The skiplists, one per multiconfig + self.skiplist_by_mc = defaultdict(dict) self.featureset = CookerFeatures() if featureSet: for f in featureSet: @@ -180,7 +186,7 @@ class BBCooker: pass self.command = bb.command.Command(self, self.process_server) - self.state = state.initial + self.state = State.INITIAL self.parser = None @@ -226,23 +232,26 @@ class BBCooker: bb.warn("Cooker received SIGTERM, shutting down...") elif signum == signal.SIGHUP: bb.warn("Cooker received SIGHUP, shutting down...") - self.state = state.forceshutdown + self.state = State.FORCE_SHUTDOWN bb.event._should_exit.set() def setFeatures(self, features): # we only accept a new feature set if we're in state initial, so we can reset without problems - if not self.state in [state.initial, state.shutdown, state.forceshutdown, state.stopped, state.error]: + if not self.state in [State.INITIAL, State.SHUTDOWN, State.FORCE_SHUTDOWN, State.STOPPED, State.ERROR]: raise Exception("Illegal state for feature set change") original_featureset = list(self.featureset) for feature in features: self.featureset.setFeature(feature) - bb.debug(1, "Features set %s (was %s)" % (original_featureset, list(self.featureset))) - if (original_featureset != list(self.featureset)) and self.state != state.error and hasattr(self, "data"): + for orig_feature in original_featureset: + if orig_feature not in features: + self.featureset.delFeature(orig_feature) + bb.debug(1, "Features set %s (was %s)" % (list(self.featureset), original_featureset)) + if (original_featureset != list(self.featureset)) and self.state != State.ERROR and hasattr(self, "data"): + bb.server.process.serverlog("Reseting due to feature chanages") self.reset() def initConfigurationData(self): - - self.state = state.initial + self.state = State.INITIAL self.caches_array = [] sys.path = self.orig_syspath.copy() @@ -250,7 +259,11 @@ class BBCooker: if mod not in self.orig_sysmodules: del sys.modules[mod] + self._baseconfig_set(False) + self._parsecache_set(False) + self.configwatched = {} + self.revalidateCaches() # Need to preserve BB_CONSOLELOG over resets consolelog = None @@ -278,10 +291,14 @@ class BBCooker: logger.critical("Unable to import extra RecipeInfo '%s' from '%s': %s" % (cache_name, module_name, exc)) raise bb.BBHandledException() + if CookerFeatures.SKIP_FRAGMENTS in self.featureset: + self.enableSkipFragments() + else: + self.disableSkipFragments() + self.databuilder = bb.cookerdata.CookerDataBuilder(self.configuration, False) self.databuilder.parseBaseConfiguration() self.data = self.databuilder.data - self.data_hash = self.databuilder.data_hash self.extraconfigdata = {} eventlog = self.data.getVar("BB_DEFAULT_EVENTLOG") @@ -300,7 +317,6 @@ class BBCooker: self.add_filewatch(mc.getVar("__base_depends", False), configwatcher=True) self._baseconfig_set(True) - self._parsecache_set(False) def handlePRServ(self): # Setup a PR Server based on the new configuration @@ -312,16 +328,34 @@ class BBCooker: if self.data.getVar("BB_HASHSERVE") == "auto": # Create a new hash server bound to a unix domain socket if not self.hashserv: - dbfile = (self.data.getVar("PERSISTENT_DIR") or self.data.getVar("CACHE")) + "/hashserv.db" + bb_hashserve_db_dir = self.data.getVar("BB_HASHSERVE_DB_DIR") + if bb_hashserve_db_dir and utils.is_path_on_nfs(bb_hashserve_db_dir): + bb.fatal("""Hash equivalency database location (set via BB_HASHSERVE_DB_DIR to {}) +cannot be on a NFS mount due to potential NFS locking issues between sqlite clients, per https://sqlite.org/faq.html#q5 + +If you need to share the database between several computers, set up a permanently running hash equivalency server +according to https://docs.yoctoproject.org/dev-manual/hashequivserver.html""".format(bb_hashserve_db_dir)) + dbdir = bb_hashserve_db_dir or self.data.getVar("PERSISTENT_DIR") or self.data.getVar("CACHE") + os.makedirs(dbdir, exist_ok=True) + dbfile = dbdir + "/hashserv.db" upstream = self.data.getVar("BB_HASHSERVE_UPSTREAM") or None if upstream: - import socket try: - sock = socket.create_connection(upstream.split(":"), 5) - sock.close() - except socket.error as e: - bb.warn("BB_HASHSERVE_UPSTREAM is not valid, unable to connect hash equivalence server at '%s': %s" + with hashserv.create_client(upstream) as client: + client.ping() + except ImportError as e: + bb.fatal(""""Unable to use hash equivalence server at '%s' due to missing or incorrect python module: +%s +Please install the needed module on the build host, or use an environment containing it: + - if you are using bitbake-setup, run 'bitbake-setup install-buildtools' + - openembedded-core layer contains 'scripts/install-buildtools' that can also be used + - or set up pip venv +You can also remove the BB_HASHSERVE_UPSTREAM setting, but this may result in significantly longer build times as bitbake will be unable to reuse prebuilt sstate artefacts.""" + % (upstream, repr(e))) + except ConnectionError as e: + bb.warn("Unable to connect to hash equivalence server at '%s', please correct or remove BB_HASHSERVE_UPSTREAM:\n%s" % (upstream, repr(e))) + upstream = None self.hashservaddr = "unix://%s/hashserve.sock" % self.data.getVar("TOPDIR") self.hashserv = hashserv.create_server( @@ -347,17 +381,34 @@ class BBCooker: if hasattr(self, "data"): self.data.disableTracking() + def enableSkipFragments(self): + self.configuration.skip_fragments = True + + def disableSkipFragments(self): + self.configuration.skip_fragments = False + def revalidateCaches(self): bb.parse.clear_cache() + if not self.baseconfig_valid: + bb.parse.BBHandler.cached_statements = {} + if hasattr(self, "databuilder"): + self.databuilder.calc_datastore_hashes(clean=False) + return + clean = True for f in self.configwatched: if not bb.parse.check_mtime(f, self.configwatched[f]): bb.server.process.serverlog("Found %s changed, invalid cache" % f) + if self.state not in (State.SHUTDOWN, State.FORCE_SHUTDOWN, State.ERROR): + self.state = State.INITIAL self._baseconfig_set(False) self._parsecache_set(False) clean = False - break + bb.parse.BBHandler.cached_statements = {} + if hasattr(self, "databuilder"): + self.databuilder.calc_datastore_hashes(clean=False) + return if clean: for f in self.parsewatched: @@ -365,10 +416,13 @@ class BBCooker: bb.server.process.serverlog("Found %s changed, invalid cache" % f) self._parsecache_set(False) clean = False + bb.parse.BBHandler.cached_statements = {} break - if not clean: - bb.parse.BBHandler.cached_statements = {} + # If writes were made to any of the data stores, we need to recalculate the data + # store cache + if hasattr(self, "databuilder"): + self.databuilder.calc_datastore_hashes(clean=True) def parseConfiguration(self): self.updateCacheSync() @@ -612,8 +666,8 @@ class BBCooker: localdata = {} for mc in self.multiconfigs: - taskdata[mc] = bb.taskdata.TaskData(halt, skiplist=self.skiplist, allowincomplete=allowincomplete) - localdata[mc] = data.createCopy(self.databuilder.mcdata[mc]) + taskdata[mc] = bb.taskdata.TaskData(halt, skiplist=self.skiplist_by_mc[mc], allowincomplete=allowincomplete) + localdata[mc] = bb.data.createCopy(self.databuilder.mcdata[mc]) bb.data.expandKeys(localdata[mc]) current = 0 @@ -680,14 +734,14 @@ class BBCooker: bb.event.fire(bb.event.TreeDataPreparationCompleted(len(fulltargetlist)), self.data) return taskdata, runlist - def prepareTreeData(self, pkgs_to_build, task): + def prepareTreeData(self, pkgs_to_build, task, halt=False): """ Prepare a runqueue and taskdata object for iteration over pkgs_to_build """ # We set halt to False here to prevent unbuildable targets raising # an exception when we're just generating data - taskdata, runlist = self.buildTaskData(pkgs_to_build, task, False, allowincomplete=True) + taskdata, runlist = self.buildTaskData(pkgs_to_build, task, halt, allowincomplete=True) return runlist, taskdata @@ -701,7 +755,7 @@ class BBCooker: if not task.startswith("do_"): task = "do_%s" % task - runlist, taskdata = self.prepareTreeData(pkgs_to_build, task) + runlist, taskdata = self.prepareTreeData(pkgs_to_build, task, halt=True) rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist) rq.rqdata.prepare() return self.buildDependTree(rq, taskdata) @@ -896,10 +950,11 @@ class BBCooker: depgraph = self.generateTaskDepTreeData(pkgs_to_build, task) - with open('pn-buildlist', 'w') as f: - for pn in depgraph["pn"]: - f.write(pn + "\n") - logger.info("PN build list saved to 'pn-buildlist'") + pns = depgraph["pn"].keys() + if pns: + with open('pn-buildlist', 'w') as f: + f.write("%s\n" % "\n".join(sorted(pns))) + logger.info("PN build list saved to 'pn-buildlist'") # Remove old format output files to ensure no confusion with stale data try: @@ -933,7 +988,7 @@ class BBCooker: for mc in self.multiconfigs: # First get list of recipes, including skipped recipefns = list(self.recipecaches[mc].pkg_fn.keys()) - recipefns.extend(self.skiplist.keys()) + recipefns.extend(self.skiplist_by_mc[mc].keys()) # Work out list of bbappends that have been applied applied_appends = [] @@ -952,13 +1007,7 @@ class BBCooker: '\n '.join(appends_without_recipes[mc]))) if msgs: - msg = "\n".join(msgs) - warn_only = self.databuilder.mcdata[mc].getVar("BB_DANGLINGAPPENDS_WARNONLY", \ - False) or "no" - if warn_only.lower() in ("1", "yes", "true"): - bb.warn(msg) - else: - bb.fatal(msg) + bb.fatal("\n".join(msgs)) def handlePrefProviders(self): @@ -1338,7 +1387,7 @@ class BBCooker: self.buildSetVars() self.reset_mtime_caches() - bb_caches = bb.cache.MulticonfigCache(self.databuilder, self.data_hash, self.caches_array) + bb_caches = bb.cache.MulticonfigCache(self.databuilder, self.databuilder.data_hash, self.caches_array) layername = self.collections[mc].calc_bbfile_priority(fn)[2] infos = bb_caches[mc].parse(fn, self.collections[mc].get_file_appends(fn), layername) @@ -1399,11 +1448,11 @@ class BBCooker: msg = None interrupted = 0 - if halt or self.state == state.forceshutdown: + if halt or self.state == State.FORCE_SHUTDOWN: rq.finish_runqueue(True) msg = "Forced shutdown" interrupted = 2 - elif self.state == state.shutdown: + elif self.state == State.SHUTDOWN: rq.finish_runqueue(False) msg = "Stopped build" interrupted = 1 @@ -1429,8 +1478,7 @@ class BBCooker: if quietlog: bb.runqueue.logger.setLevel(rqloglevel) return bb.server.process.idleFinish(msg) - if retval is True: - return True + return retval self.idleCallBackRegister(buildFileIdle, rq) @@ -1459,7 +1507,6 @@ class BBCooker: if t in task or getAllTaskSignatures: try: - rq.rqdata.prepare_task_hash(tid) sig.append([pn, t, rq.rqdata.get_task_unihash(tid)]) except KeyError: sig.append(self.getTaskSignatures(target, [t])[0]) @@ -1474,12 +1521,12 @@ class BBCooker: def buildTargetsIdle(server, rq, halt): msg = None interrupted = 0 - if halt or self.state == state.forceshutdown: + if halt or self.state == State.FORCE_SHUTDOWN: bb.event._should_exit.set() rq.finish_runqueue(True) msg = "Forced shutdown" interrupted = 2 - elif self.state == state.shutdown: + elif self.state == State.SHUTDOWN: rq.finish_runqueue(False) msg = "Stopped build" interrupted = 1 @@ -1500,8 +1547,6 @@ class BBCooker: bb.event.disable_heartbeat() return bb.server.process.idleFinish(msg) - if retval is True: - return True return retval self.reset_mtime_caches() @@ -1574,7 +1619,7 @@ class BBCooker: def updateCacheSync(self): - if self.state == state.running: + if self.state == State.RUNNING: return if not self.baseconfig_valid: @@ -1584,19 +1629,19 @@ class BBCooker: # This is called for all async commands when self.state != running def updateCache(self): - if self.state == state.running: + if self.state == State.RUNNING: return - if self.state in (state.shutdown, state.forceshutdown, state.error): + if self.state in (State.SHUTDOWN, State.FORCE_SHUTDOWN, State.ERROR): if hasattr(self.parser, 'shutdown'): self.parser.shutdown(clean=False) self.parser.final_cleanup() raise bb.BBHandledException() - if self.state != state.parsing: + if self.state != State.PARSING: self.updateCacheSync() - if self.state != state.parsing and not self.parsecache_valid: + if self.state != State.PARSING and not self.parsecache_valid: bb.server.process.serverlog("Parsing started") self.parsewatched = {} @@ -1630,9 +1675,10 @@ class BBCooker: self.parser = CookerParser(self, mcfilelist, total_masked) self._parsecache_set(True) - self.state = state.parsing + self.state = State.PARSING if not self.parser.parse_next(): + bb.server.process.serverlog("Parsing completed") collectlog.debug("parsing complete") if self.parser.error: raise bb.BBHandledException() @@ -1640,7 +1686,7 @@ class BBCooker: self.handlePrefProviders() for mc in self.multiconfigs: self.recipecaches[mc].bbfile_priority = self.collections[mc].collection_priorities(self.recipecaches[mc].pkg_fn, self.parser.mcfilelist[mc], self.data) - self.state = state.running + self.state = State.RUNNING # Send an event listing all stamps reachable after parsing # which the metadata may use to clean up stale data @@ -1713,10 +1759,10 @@ class BBCooker: def shutdown(self, force=False): if force: - self.state = state.forceshutdown + self.state = State.FORCE_SHUTDOWN bb.event._should_exit.set() else: - self.state = state.shutdown + self.state = State.SHUTDOWN if self.parser: self.parser.shutdown(clean=False) @@ -1726,7 +1772,7 @@ class BBCooker: if hasattr(self.parser, 'shutdown'): self.parser.shutdown(clean=False) self.parser.final_cleanup() - self.state = state.initial + self.state = State.INITIAL bb.event._should_exit.clear() def reset(self): @@ -1813,8 +1859,8 @@ class CookerCollectFiles(object): bb.event.fire(CookerExit(), eventdata) # We need to track where we look so that we can know when the cache is invalid. There - # is no nice way to do this, this is horrid. We intercept the os.listdir() - # (or os.scandir() for python 3.6+) calls while we run glob(). + # is no nice way to do this, this is horrid. We intercept the os.listdir() and os.scandir() + # calls while we run glob(). origlistdir = os.listdir if hasattr(os, 'scandir'): origscandir = os.scandir @@ -1994,8 +2040,9 @@ class ParsingFailure(Exception): Exception.__init__(self, realexception, recipe) class Parser(multiprocessing.Process): - def __init__(self, jobs, results, quit, profile): + def __init__(self, jobs, next_job_id, results, quit, profile): self.jobs = jobs + self.next_job_id = next_job_id self.results = results self.quit = quit multiprocessing.Process.__init__(self) @@ -2005,6 +2052,7 @@ class Parser(multiprocessing.Process): self.queue_signals = False self.signal_received = [] self.signal_threadlock = threading.Lock() + self.exit = False def catch_sig(self, signum, frame): if self.queue_signals: @@ -2017,24 +2065,10 @@ class Parser(multiprocessing.Process): signal.signal(signal.SIGTERM, signal.SIG_DFL) os.kill(os.getpid(), signal.SIGTERM) elif signum == signal.SIGINT: - signal.default_int_handler(signum, frame) + self.exit = True def run(self): - - if not self.profile: - self.realrun() - return - - try: - import cProfile as profile - except: - import profile - prof = profile.Profile() - try: - profile.Profile.runcall(prof, self.realrun) - finally: - logfile = "profile-parse-%s.log" % multiprocessing.current_process().name - prof.dump_stats(logfile) + bb.utils.profile_function("parsing" in self.profile, self.realrun, "profile-parse-%s.log" % multiprocessing.current_process().name, process=False) def realrun(self): # Signal handling here is hard. We must not terminate any process or thread holding the write @@ -2055,15 +2089,19 @@ class Parser(multiprocessing.Process): pending = [] havejobs = True try: - while havejobs or pending: + while (havejobs or pending) and not self.exit: if self.quit.is_set(): break job = None - try: - job = self.jobs.pop() - except IndexError: - havejobs = False + if havejobs: + with self.next_job_id.get_lock(): + if self.next_job_id.value < len(self.jobs): + job = self.jobs[self.next_job_id.value] + self.next_job_id.value += 1 + else: + havejobs = False + if job: result = self.parse(*job) # Clear the siggen cache after parsing to control memory usage, its huge @@ -2112,7 +2150,7 @@ class CookerParser(object): self.mcfilelist = mcfilelist self.cooker = cooker self.cfgdata = cooker.data - self.cfghash = cooker.data_hash + self.cfghash = cooker.databuilder.data_hash self.cfgbuilder = cooker.databuilder # Accounting statistics @@ -2129,13 +2167,13 @@ class CookerParser(object): self.bb_caches = bb.cache.MulticonfigCache(self.cfgbuilder, self.cfghash, cooker.caches_array) self.fromcache = set() - self.willparse = set() + self.willparse = [] for mc in self.cooker.multiconfigs: for filename in self.mcfilelist[mc]: appends = self.cooker.collections[mc].get_file_appends(filename) layername = self.cooker.collections[mc].calc_bbfile_priority(filename)[2] if not self.bb_caches[mc].cacheValid(filename, appends): - self.willparse.add((mc, self.bb_caches[mc], filename, appends, layername)) + self.willparse.append((mc, self.bb_caches[mc], filename, appends, layername)) else: self.fromcache.add((mc, self.bb_caches[mc], filename, appends, layername)) @@ -2154,18 +2192,18 @@ class CookerParser(object): def start(self): self.results = self.load_cached() self.processes = [] + if self.toparse: bb.event.fire(bb.event.ParseStarted(self.toparse), self.cfgdata) + next_job_id = multiprocessing.Value(ctypes.c_int, 0) self.parser_quit = multiprocessing.Event() self.result_queue = multiprocessing.Queue() - def chunkify(lst,n): - return [lst[i::n] for i in range(n)] - self.jobs = chunkify(list(self.willparse), self.num_processes) - + # Have to pass in willparse at fork time so all parsing processes have the unpickleable data + # then access it by index from the parse queue. for i in range(0, self.num_processes): - parser = Parser(self.jobs[i], self.result_queue, self.parser_quit, self.cooker.configuration.profile) + parser = Parser(self.willparse, next_job_id, self.result_queue, self.parser_quit, self.cooker.configuration.profile) parser.start() self.process_names.append(parser.name) self.processes.append(parser) @@ -2192,11 +2230,17 @@ class CookerParser(object): # Cleanup the queue before call process.join(), otherwise there might be # deadlocks. - while True: - try: - self.result_queue.get(timeout=0.25) - except queue.Empty: - break + def read_results(): + while True: + try: + self.result_queue.get(timeout=0.25) + except queue.Empty: + break + except KeyError: + # The restore state from SiggenRecipeInfo in cache.py can + # fail here if this is an unclean shutdown since the state may have been + # reset. Ignore key errors for that reason, we don't care. + pass def sync_caches(): for c in self.bb_caches.values(): @@ -2208,15 +2252,19 @@ class CookerParser(object): self.parser_quit.set() + read_results() + for process in self.processes: - process.join(0.5) + process.join(2) for process in self.processes: if process.exitcode is None: os.kill(process.pid, signal.SIGINT) + read_results() + for process in self.processes: - process.join(0.5) + process.join(2) for process in self.processes: if process.exitcode is None: @@ -2224,9 +2272,8 @@ class CookerParser(object): for process in self.processes: process.join() - # Added in 3.7, cleans up zombies - if hasattr(process, "close"): - process.close() + # clean up zombies + process.close() bb.codeparser.parser_cache_save() bb.codeparser.parser_cache_savemerge() @@ -2236,12 +2283,13 @@ class CookerParser(object): profiles = [] for i in self.process_names: logfile = "profile-parse-%s.log" % i - if os.path.exists(logfile): + if os.path.exists(logfile) and os.path.getsize(logfile): profiles.append(logfile) - pout = "profile-parse.log.processed" - bb.utils.process_profilelog(profiles, pout = pout) - print("Processed parsing statistics saved to %s" % (pout)) + if profiles: + fn_out = "profile-parse.log.report" + bb.utils.process_profilelog(profiles, fn_out=fn_out) + print("Processed parsing statistics saved to %s" % (fn_out)) def final_cleanup(self): if self.syncthread: @@ -2273,7 +2321,7 @@ class CookerParser(object): yield result if not (self.parsed >= self.toparse): - raise bb.parse.ParseError("Not all recipes parsed, parser thread killed/died? Exiting.", None) + raise bb.parse.ParseError("Not all recipes parsed, parser thread killed/died? (%s %s of %s) Exiting." % (len(self.processes), self.parsed, self.toparse), None) def parse_next(self): @@ -2359,7 +2407,7 @@ class CookerParser(object): for virtualfn, info_array in result: if info_array[0].skipped: self.skipped += 1 - self.cooker.skiplist[virtualfn] = SkippedPackage(info_array[0]) + self.cooker.skiplist_by_mc[mc][virtualfn] = SkippedPackage(info_array[0]) self.bb_caches[mc].add_info(virtualfn, info_array, self.cooker.recipecaches[mc], parsed=parsed, watcher = self.cooker.add_filewatch) return True diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py index 0649e409..59f808c9 100644 --- a/bitbake/lib/bb/cookerdata.py +++ b/bitbake/lib/bb/cookerdata.py @@ -1,3 +1,4 @@ + # # Copyright (C) 2003, 2004 Chris Larson # Copyright (C) 2003, 2004 Phil Blundell @@ -136,6 +137,7 @@ class CookerConfiguration(object): self.build_verbose_stdout = False self.dry_run = False self.tracking = False + self.skip_fragments = False self.writeeventlog = False self.limited_deps = False self.runall = [] @@ -225,12 +227,15 @@ class CookerDataBuilder(object): self.prefiles = cookercfg.prefile self.postfiles = cookercfg.postfile self.tracking = cookercfg.tracking + self.skip_fragments = cookercfg.skip_fragments bb.utils.set_context(bb.utils.clean_context()) bb.event.set_class_handlers(bb.event.clean_class_handlers()) self.basedata = bb.data.init() if self.tracking: self.basedata.enableTracking() + if self.skip_fragments: + self.basedata.setVar("_BB_SKIP_FRAGMENTS", "1") # Keep a datastore of the initial environment variables and their # values from when BitBake was launched to enable child processes @@ -254,14 +259,25 @@ class CookerDataBuilder(object): self.data = self.basedata self.mcdata = {} + def calc_datastore_hashes(self, clean=True): + if not clean: + self.data_hash = None + return + + data_hash = hashlib.sha256() + data_hash.update(self.data.get_hash().encode('utf-8')) + multiconfig = (self.data.getVar("BBMULTICONFIG") or "").split() + for config in multiconfig: + data_hash.update(self.mcdata[config].get_hash().encode('utf-8')) + self.data_hash = data_hash.hexdigest() + def parseBaseConfiguration(self, worker=False): mcdata = {} - data_hash = hashlib.sha256() try: self.data = self.parseConfigurationFiles(self.prefiles, self.postfiles) - if self.data.getVar("BB_WORKERCONTEXT", False) is None and not worker: - bb.fetch.fetcher_init(self.data) + servercontext = self.data.getVar("BB_WORKERCONTEXT", False) is None and not worker + bb.fetch.fetcher_init(self.data, servercontext) bb.parse.init_parser(self.data) bb.event.fire(bb.event.ConfigParsed(), self.data) @@ -279,7 +295,6 @@ class CookerDataBuilder(object): bb.event.fire(bb.event.ConfigParsed(), self.data) bb.parse.init_parser(self.data) - data_hash.update(self.data.get_hash().encode('utf-8')) mcdata[''] = self.data multiconfig = (self.data.getVar("BBMULTICONFIG") or "").split() @@ -289,11 +304,9 @@ class CookerDataBuilder(object): parsed_mcdata = self.parseConfigurationFiles(self.prefiles, self.postfiles, config) bb.event.fire(bb.event.ConfigParsed(), parsed_mcdata) mcdata[config] = parsed_mcdata - data_hash.update(parsed_mcdata.get_hash().encode('utf-8')) if multiconfig: bb.event.fire(bb.event.MultiConfigParsed(mcdata), self.data) - self.data_hash = data_hash.hexdigest() except bb.data_smart.ExpansionError as e: logger.error(str(e)) raise bb.BBHandledException() @@ -328,6 +341,7 @@ class CookerDataBuilder(object): for mc in mcdata: self.mcdata[mc] = bb.data.createCopy(mcdata[mc]) self.data = self.mcdata[''] + self.calc_datastore_hashes() def reset(self): # We may not have run parseBaseConfiguration() yet @@ -340,7 +354,7 @@ class CookerDataBuilder(object): def _findLayerConf(self, data): return findConfigFile("bblayers.conf", data) - def parseConfigurationFiles(self, prefiles, postfiles, mc = "default"): + def parseConfigurationFiles(self, prefiles, postfiles, mc = ""): data = bb.data.createCopy(self.basedata) data.setVar("BB_CURRENT_MC", mc) @@ -498,6 +512,8 @@ class CookerDataBuilder(object): bb_data.setVar("__BBMULTICONFIG", mc) bb_data.setVar("FILE_LAYERNAME", layername) + bb_data.setVar("__BB_RECIPE_FILE", bbfile) + bbfile_loc = os.path.abspath(os.path.dirname(bbfile)) bb.parse.cached_mtime_noerror(bbfile_loc) diff --git a/bitbake/lib/bb/daemonize.py b/bitbake/lib/bb/daemonize.py index 76894044..70d9b8ee 100644 --- a/bitbake/lib/bb/daemonize.py +++ b/bitbake/lib/bb/daemonize.py @@ -15,9 +15,12 @@ import os import sys import io import traceback +import warnings import bb +warnings.filterwarnings("ignore", category=DeprecationWarning, message=".*use.of.fork.*may.lead.to.deadlocks.in.the.child.*") + def createDaemon(function, logfile): """ Detach a process from the controlling terminal and run it in the diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py index 505f4295..f81f44ea 100644 --- a/bitbake/lib/bb/data.py +++ b/bitbake/lib/bb/data.py @@ -94,7 +94,7 @@ def expandKeys(alterdata, readdata = None): val = alterdata.getVar(key, False) if val is not None: bb.warn("Variable key %s (%s) replaces original key %s (%s)." % (key, val, ekey, newval)) - alterdata.renameVar(key, ekey) + alterdata.renameVar(key, ekey, recurse=False) def inheritFromOS(d, savedenv, permitted): """Inherit variables from the initial environment.""" @@ -171,6 +171,7 @@ def emit_var(var, o=sys.__stdout__, d = init(), all=False): # if we're going to output this within doublequotes, # to a shell, we need to escape the quotes in the var alter = re.sub('"', '\\"', val) + alter = re.sub('`', '\\`', alter) alter = re.sub('\n', ' \\\n', alter) alter = re.sub('\\$', '\\\\$', alter) o.write('%s="%s"\n' % (varExpanded, alter)) @@ -192,6 +193,12 @@ def exported_keys(d): not bb.utils.to_boolean(d.getVarFlag(key, 'unexport'))) def exported_vars(d): + """Return the exported variables as a list of (key, value) pairs. + + Every value is expanded before returning, so a caller that changes the + environment afterwards still gets what the expansion saw. + """ + exported = [] k = list(exported_keys(d)) for key in k: try: @@ -201,7 +208,9 @@ def exported_vars(d): continue if value is not None: - yield key, str(value) + exported.append((key, str(value))) + + return exported def emit_func(func, o=sys.__stdout__, d = init()): """Emits all items in the data store in a format such that it can be sourced by a shell.""" @@ -293,7 +302,7 @@ def build_dependencies(key, keys, mod_funcs, shelldeps, varflagsexcl, ignored_va if key in mod_funcs: exclusions = set() moddep = bb.codeparser.modulecode_deps[key] - value = handle_contains("", moddep[3], exclusions, d) + value = handle_contains(moddep[4], moddep[3], exclusions, d) return frozenset((moddep[0] | keys & moddep[1]) - ignored_vars), value if key[-1] == ']': @@ -366,7 +375,7 @@ def build_dependencies(key, keys, mod_funcs, shelldeps, varflagsexcl, ignored_va except bb.parse.SkipRecipe: raise except Exception as e: - bb.warn("Exception during build_dependencies for %s" % key) + bb.warn("Exception during build_dependencies for %s: %s" % (key, repr(e))) raise return frozenset(deps), value #bb.note("Variable %s references %s and calls %s" % (key, str(deps), str(execs))) @@ -377,7 +386,7 @@ def generate_dependencies(d, ignored_vars): mod_funcs = set(bb.codeparser.modulecode_deps.keys()) keys = set(key for key in d if not key.startswith("__")) | mod_funcs shelldeps = set(key for key in d.getVar("__exportlist", False) if bb.utils.to_boolean(d.getVarFlag(key, "export")) and not bb.utils.to_boolean(d.getVarFlag(key, "unexport"))) - varflagsexcl = d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS') + varflagsexcl = (d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS') or "").split() codeparserd = d.createCopy() for forced in (d.getVar('BB_HASH_CODEPARSER_VALS') or "").split(): diff --git a/bitbake/lib/bb/data_smart.py b/bitbake/lib/bb/data_smart.py index 0128a5bb..e26b0659 100644 --- a/bitbake/lib/bb/data_smart.py +++ b/bitbake/lib/bb/data_smart.py @@ -24,6 +24,7 @@ from collections.abc import MutableMapping import logging import hashlib import bb, bb.codeparser +import bb.filter from bb import utils from bb.COW import COWDictBase @@ -31,7 +32,7 @@ logger = logging.getLogger("BitBake.Data") __setvar_keyword__ = [":append", ":prepend", ":remove"] __setvar_regexp__ = re.compile(r'(?P.*?)(?P:append|:prepend|:remove)(:(?P[^A-Z]*))?$') -__expand_var_regexp__ = re.compile(r"\${[a-zA-Z0-9\-_+./~:]+?}") +__expand_var_regexp__ = re.compile(r"\${[a-zA-Z0-9\-_+./~:]+}") __expand_python_regexp__ = re.compile(r"\${@(?:{.*?}|.)+?}") __whitespace_split__ = re.compile(r'(\s)') __override_regexp__ = re.compile(r'[a-z0-9]+') @@ -106,52 +107,52 @@ class VariableParse: self.contains = {} def var_sub(self, match): - key = match.group()[2:-1] - if self.varname and key: - if self.varname == key: - raise Exception("variable %s references itself!" % self.varname) - var = self.d.getVarFlag(key, "_content") - self.references.add(key) - if var is not None: - return var - else: - return match.group() + key = match.group()[2:-1] + if self.varname and key: + if self.varname == key: + raise Exception("variable %s references itself!" % self.varname) + var = self.d.getVarFlag(key, "_content") + self.references.add(key) + if var is not None: + return var + else: + return match.group() def python_sub(self, match): - if isinstance(match, str): - code = match - else: - code = match.group()[3:-1] + if isinstance(match, str): + code = match + else: + code = match.group()[3:-1] - # Do not run code that contains one or more unexpanded variables - # instead return the code with the characters we removed put back - if __expand_var_regexp__.findall(code): - return "${@" + code + "}" + # Do not run code that contains one or more unexpanded variables + # instead return the code with the characters we removed put back + if __expand_var_regexp__.findall(code): + return "${@" + code + "}" - if self.varname: - varname = 'Var <%s>' % self.varname - else: - varname = '' - codeobj = compile(code.strip(), varname, "eval") - - parser = bb.codeparser.PythonParser(self.varname, logger) - parser.parse_python(code) - if self.varname: - vardeps = self.d.getVarFlag(self.varname, "vardeps") - if vardeps is None: - parser.log.flush() - else: + if self.varname: + varname = 'Var <%s>' % self.varname + else: + varname = '' + codeobj = compile(code.strip(), varname, "eval") + + parser = bb.codeparser.PythonParser(self.varname, logger) + parser.parse_python(code) + if self.varname: + vardeps = self.d.getVarFlag(self.varname, "vardeps") + if vardeps is None: parser.log.flush() - self.references |= parser.references - self.execs |= parser.execs + else: + parser.log.flush() + self.references |= parser.references + self.execs |= parser.execs - for k in parser.contains: - if k not in self.contains: - self.contains[k] = parser.contains[k].copy() - else: - self.contains[k].update(parser.contains[k]) - value = utils.better_eval(codeobj, DataContext(self.d), {'d' : self.d}) - return str(value) + for k in parser.contains: + if k not in self.contains: + self.contains[k] = parser.contains[k].copy() + else: + self.contains[k].update(parser.contains[k]) + value = utils.better_eval(codeobj, DataContext(self.d), {'d' : self.d}) + return str(value) class DataContext(dict): excluded = set([i for i in dir(builtins) if not i.startswith('_')] + ['oe']) @@ -272,12 +273,9 @@ class VariableHistory(object): return if 'op' not in loginfo or not loginfo['op']: loginfo['op'] = 'set' - if 'detail' in loginfo: - loginfo['detail'] = str(loginfo['detail']) if 'variable' not in loginfo or 'file' not in loginfo: raise ValueError("record() missing variable or file.") var = loginfo['variable'] - if var not in self.variables: self.variables[var] = [] if not isinstance(self.variables[var], list): @@ -336,7 +334,8 @@ class VariableHistory(object): flag = '[%s] ' % (event['flag']) else: flag = '' - o.write("# %s %s:%s%s\n# %s\"%s\"\n" % (event['op'], event['file'], event['line'], display_func, flag, re.sub('\n', '\n# ', event['detail']))) + o.write("# %s %s:%s%s\n# %s\"%s\"\n" % \ + (event['op'], event['file'], event['line'], display_func, flag, re.sub('\n', '\n# ', str(event['detail'])))) if len(history) > 1: o.write("# pre-expansion value:\n") o.write('# "%s"\n' % (commentVal)) @@ -390,7 +389,7 @@ class VariableHistory(object): if isset and event['op'] == 'set?': continue isset = True - items = d.expand(event['detail']).split() + items = d.expand(str(event['detail'])).split() for item in items: # This is a little crude but is belt-and-braces to avoid us # having to handle every possible operation type specifically @@ -429,6 +428,7 @@ class DataSmart(MutableMapping): self.inchistory = IncludeHistory() self.varhistory = VariableHistory(self) + self.filters = {} self._tracking = False self._var_renames = {} self._var_renames.update(bitbake_renamed_vars) @@ -582,12 +582,10 @@ class DataSmart(MutableMapping): else: loginfo['op'] = keyword self.varhistory.record(**loginfo) - # todo make sure keyword is not __doc__ or __module__ # pay the cookie monster # more cookies for the cookie monster - if ':' in var: - self._setvar_update_overrides(base, **loginfo) + self._setvar_update_overrides(base, **loginfo) if base in self.overridevars: self._setvar_update_overridevars(var, value) @@ -640,6 +638,7 @@ class DataSmart(MutableMapping): nextnew.update(vardata.contains.keys()) new = nextnew self.overrides = None + self.expand_cache = {} def _setvar_update_overrides(self, var, **loginfo): # aka pay the cookie monster @@ -662,7 +661,7 @@ class DataSmart(MutableMapping): def getVar(self, var, expand=True, noweakdefault=False, parsing=False): return self.getVarFlag(var, "_content", expand, noweakdefault, parsing) - def renameVar(self, key, newkey, **loginfo): + def renameVar(self, key, newkey, recurse=True, **loginfo): """ Rename the variable key to newkey """ @@ -681,6 +680,7 @@ class DataSmart(MutableMapping): srcflags = self.getVarFlags(key, False, True) or {} for i in srcflags: + if i not in (__setvar_keyword__): continue src = srcflags[i] @@ -693,7 +693,8 @@ class DataSmart(MutableMapping): self.overridedata[newkey] = [] for (v, o) in self.overridedata[key]: self.overridedata[newkey].append([v.replace(key, newkey), o]) - self.renameVar(v, v.replace(key, newkey)) + if recurse: + self.renameVar(v, v.replace(key, newkey)) if ':' in newkey and val is None: self._setvar_update_overrides(newkey, **loginfo) @@ -781,10 +782,10 @@ class DataSmart(MutableMapping): return None cachename = var + "[" + flag + "]" - if not expand and retparser and cachename in self.expand_cache: + if not expand and retparser and cachename in self.expand_cache and not noweakdefault: return self.expand_cache[cachename].unexpanded_value, self.expand_cache[cachename] - if expand and cachename in self.expand_cache: + if expand and cachename in self.expand_cache and not noweakdefault: return self.expand_cache[cachename].value local_var = self._findVar(var) @@ -829,6 +830,8 @@ class DataSmart(MutableMapping): value = copy.copy(local_var[flag]) elif flag == "_content" and "_defaultval" in local_var and not noweakdefault: value = copy.copy(local_var["_defaultval"]) + elif "_defaultval_flag_"+flag in local_var and not noweakdefault: + value = copy.copy(local_var["_defaultval_flag_"+flag]) if flag == "_content" and local_var is not None and ":append" in local_var and not parsing: @@ -896,7 +899,13 @@ class DataSmart(MutableMapping): if expand: value = parser.value - if parser: + if value and expand and flag == "_content": + basevar = var.split(":")[0] + if basevar in self.filters: + value = bb.filter.apply_filters(value, [self.filters[basevar],]) + parser.value = value + + if parser and not noweakdefault: self.expand_cache[cachename] = parser if retparser: @@ -920,6 +929,8 @@ class DataSmart(MutableMapping): self.varhistory.record(**loginfo) del self.dict[var][flag] + if ("_defaultval_flag_" + flag) in self.dict[var]: + del self.dict[var]["_defaultval_flag_" + flag] def appendVarFlag(self, var, flag, value, **loginfo): loginfo['op'] = 'append' @@ -954,17 +965,22 @@ class DataSmart(MutableMapping): flags = {} if local_var: - for i in local_var: - if i.startswith(("_", ":")) and not internalflags: + for i, val in local_var.items(): + if i.startswith("_defaultval_flag_") and not internalflags: + i = i[len("_defaultval_flag_"):] + if i not in local_var: + flags[i] = val + elif i.startswith(("_", ":")) and not internalflags: continue - flags[i] = local_var[i] + else: + flags[i] = val + if expand and i in expand: flags[i] = self.expand(flags[i], var + "[" + i + "]") if len(flags) == 0: return None return flags - def delVarFlags(self, var, **loginfo): self.expand_cache = {} if not var in self.dict: @@ -994,6 +1010,7 @@ class DataSmart(MutableMapping): data.varhistory = self.varhistory.copy() data.varhistory.dataroot = data data.inchistory = self.inchistory.copy() + data.filters = self.filters.copy() data._tracking = self._tracking data._var_renames = self._var_renames @@ -1022,6 +1039,15 @@ class DataSmart(MutableMapping): if referrervalue and isinstance(referrervalue, str) and ref in referrervalue: self.setVar(key, referrervalue.replace(ref, value)) + def setVarFilter(self, var, filter): + if filter: + self.filters[var] = filter + else: + try: + del self.filters[var] + except KeyError: + pass + def localkeys(self): for key in self.dict: if key not in ['_data']: @@ -1114,5 +1140,10 @@ class DataSmart(MutableMapping): value = d.getVar(i, False) or "" data.update({i:value}) + moddeps = bb.codeparser.modulecode_deps + for dep in sorted(moddeps): + # Ignore visitor code, sort sets + data.update({'moddep[%s]' % dep : [sorted(moddeps[dep][0]), sorted(moddeps[dep][1]), sorted(moddeps[dep][2]), sorted(moddeps[dep][3]), moddeps[dep][4]]}) + data_str = str([(k, data[k]) for k in sorted(data.keys())]) return hashlib.sha256(data_str.encode("utf-8")).hexdigest() diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py index 952c85c0..ddf1006c 100644 --- a/bitbake/lib/bb/event.py +++ b/bitbake/lib/bb/event.py @@ -90,7 +90,7 @@ def disable_heartbeat(): # In long running code, this function should be called periodically # to check if we should exit due to an interuption (.e.g Ctrl+C from the UI) # -def check_for_interrupts(d): +def check_for_interrupts(): global _should_exit if _should_exit.is_set(): bb.warn("Exiting due to interrupt.") @@ -194,7 +194,12 @@ def fire_ui_handlers(event, d): ui_queue.append(event) return - with bb.utils.lock_timeout(_thread_lock): + with bb.utils.lock_timeout_nocheck(_thread_lock) as lock: + if not lock: + # If we can't get the lock, we may be recursively called, queue and return + ui_queue.append(event) + return + errors = [] for h in _ui_handlers: #print "Sending event %s" % event @@ -213,6 +218,9 @@ def fire_ui_handlers(event, d): for h in errors: del _ui_handlers[h] + while ui_queue: + fire_ui_handlers(ui_queue.pop(), d) + def fire(event, d): """Fire off an Event""" @@ -228,9 +236,12 @@ def fire(event, d): # If messages have been queued up, clear the queue global _uiready, ui_queue if _uiready and ui_queue: - for queue_event in ui_queue: + with bb.utils.lock_timeout_nocheck(_thread_lock): + queue = ui_queue + ui_queue = [] + for queue_event in queue: fire_ui_handlers(queue_event, d) - ui_queue = [] + fire_ui_handlers(event, d) def fire_from_worker(event, d): @@ -423,6 +434,16 @@ class RecipeEvent(Event): self.fn = fn Event.__init__(self) +class RecipePreDeferredInherits(RecipeEvent): + """ + Called before deferred inherits are processed so code can snoop on class extensions for example + Limitations: It won't see inherits of inherited classes and the data is unexpanded + """ + def __init__(self, fn, inherits): + self.fn = fn + self.inherits = inherits + Event.__init__(self) + class RecipePreFinalise(RecipeEvent): """ Recipe Parsing Complete but not yet finalised""" diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py index 5bf2c4b8..d5f9c7d8 100644 --- a/bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/__init__.py @@ -23,17 +23,19 @@ import collections import subprocess import pickle import errno -import bb.persist_data, bb.utils +import shlex +import bb.utils import bb.checksum import bb.process import bb.event __version__ = "2" _checksum_cache = bb.checksum.FileChecksumCache() +_revisions_cache = bb.checksum.RevisionsCache() logger = logging.getLogger("BitBake.Fetcher") -CHECKSUM_LIST = [ "md5", "sha256", "sha1", "sha384", "sha512" ] +CHECKSUM_LIST = [ "goh1", "md5", "sha256", "sha1", "sha384", "sha512" ] SHOWN_CHECKSUM_LIST = ["sha256"] class BBFetchException(Exception): @@ -95,6 +97,23 @@ class UnpackError(BBFetchException): BBFetchException.__init__(self, msg) self.args = (message, url) +class LocalModificationsError(UnpackError): + """Exception raised when a checkout cannot be updated due to local modifications""" + def __init__(self, repodir, url, git_output): + message = ("Repository at %s has uncommitted changes, unable to update:\n%s\n" + "Commit, stash or discard your changes and re-run the update." % (repodir, git_output)) + UnpackError.__init__(self, message, url) + self.args = (repodir, url, git_output) + +class RebaseError(UnpackError): + """Exception raised when a checkout has local commits that could not be rebased onto the new upstream revision""" + def __init__(self, repodir, url, git_output): + message = ("Repository at %s has local commits that could not be rebased onto the new upstream revision:\n%s\n" + "Note: the 'dldir' remote points to the local download cache and may be used to resolve the conflict manually.\n" + "Once resolved, re-run the update." % (repodir, git_output)) + UnpackError.__init__(self, message, url) + self.args = (repodir, url, git_output) + class NoMethodError(BBFetchException): """Exception raised when there is no method to obtain a supplied url or set of urls""" def __init__(self, url): @@ -237,7 +256,7 @@ class URI(object): # to RFC compliant URL format. E.g.: # file://foo.diff -> file:foo.diff if urlp.scheme in self._netloc_forbidden: - uri = re.sub("(?<=:)//(?!/)", "", uri, 1) + uri = re.sub(r"(?<=:)//(?!/)", "", uri, count=1) reparse = 1 if reparse: @@ -285,7 +304,7 @@ class URI(object): def _query_str(self): return ( - ''.join(['?', self._param_str_join(self.query, "&")]) + ''.join(['?', self._query_str_join(self.query, "&")]) if self.query else '') def _param_str_split(self, string, elmdelim, kvdelim="="): @@ -294,9 +313,12 @@ class URI(object): ret[k] = v return ret - def _param_str_join(self, dict_, elmdelim, kvdelim="="): + def _query_str_join(self, dict_, elmdelim, kvdelim="="): return elmdelim.join([kvdelim.join([k, v]) if v else k for k, v in dict_.items()]) + def _param_str_join(self, dict_, elmdelim, kvdelim="="): + return elmdelim.join([kvdelim.join([k, v]) for k, v in dict_.items()]) + @property def hostport(self): if not self.port: @@ -352,6 +374,14 @@ def decodeurl(url): user, password, parameters). """ + uri = URI(url) + path = uri.path if uri.path else "/" + return uri.scheme, uri.hostport, path, uri.username, uri.password, uri.params + +def decodemirrorurl(url): + """Decodes a mirror URL into the tokens (scheme, network location, path, + user, password, parameters). + """ m = re.compile('(?P[^:]*)://((?P[^/;]+)@)?(?P[^;]+)(;(?P.*))?').match(url) if not m: raise MalformedUrl(url) @@ -370,6 +400,9 @@ def decodeurl(url): elif type.lower() == 'file': host = "" path = location + if user: + path = user + '@' + path + user = "" else: host = location path = "/" @@ -402,32 +435,34 @@ def encodeurl(decoded): if not type: raise MissingParameterError('type', "encoded from the data %s" % str(decoded)) - url = ['%s://' % type] + uri = URI() + uri.scheme = type if user and type != "file": - url.append("%s" % user) + uri.username = user if pswd: - url.append(":%s" % pswd) - url.append("@") + uri.password = pswd if host and type != "file": - url.append("%s" % host) + uri.hostname = host if path: # Standardise path to ensure comparisons work while '//' in path: path = path.replace("//", "/") - url.append("%s" % urllib.parse.quote(path)) + uri.path = path + if type == "file": + # Use old not IETF compliant style + uri.relative = False if p: - for parm in p: - url.append(";%s=%s" % (parm, p[parm])) + uri.params = p - return "".join(url) + return str(uri) def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None): if not ud.url or not uri_find or not uri_replace: logger.error("uri_replace: passed an undefined value, not replacing") return None - uri_decoded = list(decodeurl(ud.url)) - uri_find_decoded = list(decodeurl(uri_find)) - uri_replace_decoded = list(decodeurl(uri_replace)) + uri_decoded = list(decodemirrorurl(ud.url)) + uri_find_decoded = list(decodemirrorurl(uri_find)) + uri_replace_decoded = list(decodemirrorurl(uri_replace)) logger.debug2("For url %s comparing %s to %s" % (uri_decoded, uri_find_decoded, uri_replace_decoded)) result_decoded = ['', '', '', '', '', {}] # 0 - type, 1 - host, 2 - path, 3 - user, 4- pswd, 5 - params @@ -460,7 +495,7 @@ def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None): for k in replacements: uri_replace_decoded[loc] = uri_replace_decoded[loc].replace(k, replacements[k]) #bb.note("%s %s %s" % (regexp, uri_replace_decoded[loc], uri_decoded[loc])) - result_decoded[loc] = re.sub(regexp, uri_replace_decoded[loc], uri_decoded[loc], 1) + result_decoded[loc] = re.sub(regexp, uri_replace_decoded[loc], uri_decoded[loc], count=1) if loc == 2: # Handle path manipulations basename = None @@ -493,18 +528,23 @@ methods = [] urldata_cache = {} saved_headrevs = {} -def fetcher_init(d): +def fetcher_init(d, servercontext=True): """ Called to initialize the fetchers once the configuration data is known. Calls before this must not hit the cache. """ - revs = bb.persist_data.persist('BB_URI_HEADREVS', d) + _checksum_cache.init_cache(d.getVar("BB_CACHEDIR")) + _revisions_cache.init_cache(d.getVar("BB_CACHEDIR")) + + if not servercontext: + return + try: # fetcher_init is called multiple times, so make sure we only save the # revs the first time it is called. if not bb.fetch2.saved_headrevs: - bb.fetch2.saved_headrevs = dict(revs) + bb.fetch2.saved_headrevs = _revisions_cache.get_revs() except: pass @@ -514,11 +554,10 @@ def fetcher_init(d): logger.debug("Keeping SRCREV cache due to cache policy of: %s", srcrev_policy) elif srcrev_policy == "clear": logger.debug("Clearing SRCREV cache due to cache policy of: %s", srcrev_policy) - revs.clear() + _revisions_cache.clear_cache() else: raise FetchError("Invalid SRCREV cache policy of: %s" % srcrev_policy) - _checksum_cache.init_cache(d.getVar("BB_CACHEDIR")) for m in methods: if hasattr(m, "init"): @@ -526,9 +565,11 @@ def fetcher_init(d): def fetcher_parse_save(): _checksum_cache.save_extras() + _revisions_cache.save_extras() def fetcher_parse_done(): _checksum_cache.save_merge() + _revisions_cache.save_merge() def fetcher_compare_revisions(d): """ @@ -536,7 +577,7 @@ def fetcher_compare_revisions(d): when bitbake was started and return true if they have changed. """ - headrevs = dict(bb.persist_data.persist('BB_URI_HEADREVS', d)) + headrevs = _revisions_cache.get_revs() return headrevs != bb.fetch2.saved_headrevs def mirror_from_string(data): @@ -786,8 +827,8 @@ def _get_srcrev(d, method_name='sortable_revision'): return "", revs - if len(scms) == 1 and len(urldata[scms[0]].names) == 1: - autoinc, rev = getattr(urldata[scms[0]].method, method_name)(urldata[scms[0]], d, urldata[scms[0]].names[0]) + if len(scms) == 1: + autoinc, rev = getattr(urldata[scms[0]].method, method_name)(urldata[scms[0]], d, urldata[scms[0]].name) revs.append(rev) if len(rev) > 10: rev = rev[:10] @@ -808,13 +849,12 @@ def _get_srcrev(d, method_name='sortable_revision'): seenautoinc = False for scm in scms: ud = urldata[scm] - for name in ud.names: - autoinc, rev = getattr(ud.method, method_name)(ud, d, name) - revs.append(rev) - seenautoinc = seenautoinc or autoinc - if len(rev) > 10: - rev = rev[:10] - name_to_rev[name] = rev + autoinc, rev = getattr(ud.method, method_name)(ud, d, ud.name) + revs.append(rev) + seenautoinc = seenautoinc or autoinc + if len(rev) > 10: + rev = rev[:10] + name_to_rev[ud.name] = rev # Replace names by revisions in the SRCREV_FORMAT string. The approach used # here can handle names being prefixes of other names and names appearing # as substrings in revisions (in which case the name should not be @@ -858,6 +898,7 @@ FETCH_EXPORT_VARS = ['HOME', 'PATH', 'FTPS_PROXY', 'ftps_proxy', 'NO_PROXY', 'no_proxy', 'ALL_PROXY', 'all_proxy', + 'GIT_CONFIG_GLOBAL', 'GIT_PROXY_COMMAND', 'GIT_SSH', 'GIT_SSH_COMMAND', @@ -878,6 +919,7 @@ FETCH_EXPORT_VARS = ['HOME', 'PATH', 'AWS_SESSION_TOKEN', 'GIT_CACHE_PATH', 'REMOTE_CONTAINERS_IPC', + 'GITHUB_TOKEN', 'SSL_CERT_DIR'] def get_fetcher_environment(d): @@ -1010,6 +1052,7 @@ def build_mirroruris(origud, mirrors, ld): newud = FetchData(newuri, ld) newud.ignore_checksums = True newud.setup_localpath(ld) + newud.origud = ud except bb.fetch2.BBFetchException as e: logger.debug("Mirror fetch failure for url %s (original url: %s)" % (newuri, origud.url)) logger.debug(str(e)) @@ -1047,10 +1090,10 @@ def try_mirror_url(fetch, origud, ud, ld, check = False): # Return of None or a value means we're finished # False means try another url - if ud.lockfile and ud.lockfile != origud.lockfile: - lf = bb.utils.lockfile(ud.lockfile) - try: + if ud.lockfile and ud.lockfile != origud.lockfile: + lf = bb.utils.lockfile(ud.lockfile) + if check: found = ud.method.checkstatus(fetch, ud, ld) if found: @@ -1072,6 +1115,10 @@ def try_mirror_url(fetch, origud, ud, ld, check = False): # If that tarball is a local file:// we need to provide a symlink to it dldir = ld.getVar("DL_DIR") + if bb.utils.to_boolean(ld.getVar("BB_FETCH_PREMIRRORONLY")): + ld = ld.createCopy() + ld.setVar("BB_NO_NETWORK", "1") + if origud.mirrortarballs and os.path.basename(ud.localpath) in origud.mirrortarballs and os.path.basename(ud.localpath) != os.path.basename(origud.localpath): # Create donestamp in old format to avoid triggering a re-download if ud.donestamp: @@ -1093,7 +1140,10 @@ def try_mirror_url(fetch, origud, ud, ld, check = False): origud.method.build_mirror_data(origud, ld) return origud.localpath # Otherwise the result is a local file:// and we symlink to it - ensure_symlink(ud.localpath, origud.localpath) + # This may also be a link to a shallow archive + # When using shallow mode, add a symlink to the original fullshallow + # path to ensure a valid symlink even in the `PREMIRRORS` case + origud.method.update_mirror_links(ud, origud) update_stamp(origud, ld) return ud.localpath @@ -1127,25 +1177,6 @@ def try_mirror_url(fetch, origud, ud, ld, check = False): if ud.lockfile and ud.lockfile != origud.lockfile: bb.utils.unlockfile(lf) - -def ensure_symlink(target, link_name): - if not os.path.exists(link_name): - dirname = os.path.dirname(link_name) - bb.utils.mkdirhier(dirname) - if os.path.islink(link_name): - # Broken symbolic link - os.unlink(link_name) - - # In case this is executing without any file locks held (as is - # the case for file:// URLs), two tasks may end up here at the - # same time, in which case we do not want the second task to - # fail when the link has already been created by the first task. - try: - os.symlink(target, link_name) - except FileExistsError: - pass - - def try_mirrors(fetch, d, origud, mirrors, check = False): """ Try to use a mirrored version of the sources. @@ -1174,7 +1205,7 @@ def trusted_network(d, url): if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")): return True - pkgname = d.expand(d.getVar('PN', False)) + pkgname = d.getVar('PN') trusted_hosts = None if pkgname: trusted_hosts = d.getVarFlag('BB_ALLOWED_NETWORKS', pkgname, False) @@ -1227,20 +1258,17 @@ def srcrev_internal_helper(ud, d, name): if srcrev and srcrev != "INVALID": break - if 'rev' in ud.parm and 'tag' in ud.parm: - raise FetchError("Please specify a ;rev= parameter or a ;tag= parameter in the url %s but not both." % (ud.url)) - - if 'rev' in ud.parm or 'tag' in ud.parm: - if 'rev' in ud.parm: - parmrev = ud.parm['rev'] - else: - parmrev = ud.parm['tag'] + if 'rev' in ud.parm: + parmrev = ud.parm['rev'] if srcrev == "INVALID" or not srcrev: return parmrev if srcrev != parmrev: raise FetchError("Conflicting revisions (%s from SRCREV and %s from the url) found, please specify one valid value" % (srcrev, parmrev)) return parmrev + if 'tag' in ud.parm and (srcrev == "INVALID" or not srcrev): + return ud.parm['tag'] + if srcrev == "INVALID" or not srcrev: raise FetchError("Please set a valid SRCREV for url %s (possible key names are %s, or use a ;rev=X URL parameter)" % (str(attempts), ud.url), ud.url) if srcrev == "AUTOINC": @@ -1263,7 +1291,7 @@ def get_checksum_file_list(d): found = False paths = ud.method.localfile_searchpaths(ud, d) for f in paths: - pth = ud.decodedurl + pth = ud.path if os.path.exists(f): found = True filelist.append(f + ":" + str(os.path.exists(f))) @@ -1308,23 +1336,28 @@ class FetchData(object): self.setup = False def configure_checksum(checksum_id): + checksum_plain_name = "%ssum" % checksum_id if "name" in self.parm: checksum_name = "%s.%ssum" % (self.parm["name"], checksum_id) else: - checksum_name = "%ssum" % checksum_id - - setattr(self, "%s_name" % checksum_id, checksum_name) + checksum_name = checksum_plain_name if checksum_name in self.parm: checksum_expected = self.parm[checksum_name] - elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3", "az", "crate", "gs"]: + elif checksum_plain_name in self.parm: + checksum_expected = self.parm[checksum_plain_name] + checksum_name = checksum_plain_name + elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3", "az", "crate", "gs", "gomod", "npm"]: checksum_expected = None else: checksum_expected = d.getVarFlag("SRC_URI", checksum_name) + setattr(self, "%s_name" % checksum_id, checksum_name) setattr(self, "%s_expected" % checksum_id, checksum_expected) - self.names = self.parm.get("name",'default').split(',') + self.name = self.parm.get("name",'default') + if "," in self.name: + raise ParameterError("The fetcher no longer supports multiple name parameters in a single url", self.url) self.method = None for m in methods: @@ -1376,13 +1409,7 @@ class FetchData(object): self.lockfile = basepath + '.lock' def setup_revisions(self, d): - self.revisions = {} - for name in self.names: - self.revisions[name] = srcrev_internal_helper(self, d, name) - - # add compatibility code for non name specified case - if len(self.names) == 1: - self.revision = self.revisions[self.names[0]] + self.revision = srcrev_internal_helper(self, d, self.name) def setup_localpath(self, d): if not self.localpath: @@ -1510,7 +1537,7 @@ class FetchMethod(object): (file, urldata.parm.get('unpack'))) base, ext = os.path.splitext(file) - if ext in ['.gz', '.bz2', '.Z', '.xz', '.lz']: + if ext in ['.gz', '.bz2', '.Z', '.xz', '.lz', '.zst']: efile = os.path.join(rootdir, os.path.basename(base)) else: efile = file @@ -1519,7 +1546,10 @@ class FetchMethod(object): if unpack: tar_cmd = 'tar --extract --no-same-owner' if 'striplevel' in urldata.parm: - tar_cmd += ' --strip-components=%s' % urldata.parm['striplevel'] + striplevel = urldata.parm['striplevel'] + if not striplevel.isdigit(): + raise UnpackError("Invalid striplevel parameter: %s" % striplevel, urldata.url) + tar_cmd += ' --strip-components=%s' % striplevel if file.endswith('.tar'): cmd = '%s -f %s' % (tar_cmd, file) elif file.endswith('.tgz') or file.endswith('.tar.gz') or file.endswith('.tar.Z'): @@ -1559,24 +1589,27 @@ class FetchMethod(object): elif file.endswith('.rpm') or file.endswith('.srpm'): if 'extract' in urldata.parm: unpack_file = urldata.parm.get('extract') - cmd = 'rpm2cpio.sh %s | cpio -id %s' % (file, unpack_file) + cmd = 'rpm2cpio.sh %s | cpio --no-absolute-filenames -id %s' % (file, unpack_file) iterate = True iterate_file = unpack_file else: - cmd = 'rpm2cpio.sh %s | cpio -id' % (file) + cmd = 'rpm2cpio.sh %s | cpio --no-absolute-filenames -id' % (file) elif file.endswith('.deb') or file.endswith('.ipk'): output = subprocess.check_output(['ar', '-t', file], preexec_fn=subprocess_setup) datafile = None + valid_datafiles = ('data.tar', 'data.tar.gz', 'data.tar.xz', + 'data.tar.zst', 'data.tar.bz2', 'data.tar.lzma') if output: for line in output.decode().splitlines(): - if line.startswith('data.tar.'): + if line in valid_datafiles: datafile = line break else: - raise UnpackError("Unable to unpack deb/ipk package - does not contain data.tar.* file", urldata.url) + raise UnpackError("Unable to unpack deb/ipk package - does not contain supported data.tar* file", urldata.url) else: raise UnpackError("Unable to unpack deb/ipk package - could not list contents", urldata.url) - cmd = 'ar x %s %s && %s -p -f %s && rm %s' % (file, datafile, tar_cmd, datafile, datafile) + quoted_datafile = shlex.quote(datafile) + cmd = 'ar x %s %s && %s -p -f %s && rm %s' % (shlex.quote(file), quoted_datafile, tar_cmd, quoted_datafile, quoted_datafile) # If 'subdir' param exists, create a dir and use it as destination for unpack cmd if 'subdir' in urldata.parm: @@ -1606,7 +1639,7 @@ class FetchMethod(object): if urlpath.find("/") != -1: destdir = urlpath.rsplit("/", 1)[0] + '/' bb.utils.mkdirhier("%s/%s" % (unpackdir, destdir)) - cmd = 'cp -fpPRH "%s" "%s"' % (file, destdir) + cmd = 'cp --force --preserve=timestamps --no-dereference --recursive -H "%s" "%s"' % (file, destdir) else: urldata.unpack_tracer.unpack("archive-extract", unpackdir) @@ -1629,12 +1662,37 @@ class FetchMethod(object): return + def unpack_update(self, urldata, rootdir, data): + raise RuntimeError("No method available for this url type: %s" % urldata.type) + def clean(self, urldata, d): """ Clean any existing full or partial download """ bb.utils.remove(urldata.localpath) + def ensure_symlink(self, target, link_name): + if not os.path.exists(link_name): + dirname = os.path.dirname(link_name) + bb.utils.mkdirhier(dirname) + if os.path.islink(link_name): + # Broken symbolic link + os.unlink(link_name) + + # In case this is executing without any file locks held (as is + # the case for file:// URLs), two tasks may end up here at the + # same time, in which case we do not want the second task to + # fail when the link has already been created by the first task. + try: + os.symlink(target, link_name) + except FileExistsError: + pass + + def update_mirror_links(self, ud, origud): + # For local file:// results, create a symlink to them + # This may also be a link to a shallow archive + self.ensure_symlink(ud.localpath, origud.localpath) + def try_premirror(self, urldata, d): """ Should premirrors be used? @@ -1662,13 +1720,13 @@ class FetchMethod(object): if not hasattr(self, "_latest_revision"): raise ParameterError("The fetcher for this URL does not support _latest_revision", ud.url) - revs = bb.persist_data.persist('BB_URI_HEADREVS', d) key = self.generate_revision_key(ud, d, name) - try: - return revs[key] - except KeyError: - revs[key] = rev = self._latest_revision(ud, d, name) - return rev + + rev = _revisions_cache.get_rev(key) + if rev is None: + rev = self._latest_revision(ud, d, name) + _revisions_cache.set_rev(key, rev) + return rev def sortable_revision(self, ud, d, name): latest_rev = self._build_revision(ud, d, name) @@ -1677,7 +1735,7 @@ class FetchMethod(object): def generate_revision_key(self, ud, d, name): return self._revision_key(ud, d, name) - def latest_versionstring(self, ud, d): + def latest_versionstring(self, ud, d, filter_regex=None): """ Compute the latest release name like "x.y.x" in "x.y.x+gitHASH" by searching through the tags output of ls-remote, comparing @@ -1806,7 +1864,7 @@ class Fetch(object): self.ud[url] = FetchData(url, self.d) self.ud[url].setup_localpath(self.d) - return self.d.expand(self.ud[url].localpath) + return self.ud[url].localpath def localpaths(self): """ @@ -1838,10 +1896,10 @@ class Fetch(object): m = ud.method done = False - if ud.lockfile: - lf = bb.utils.lockfile(ud.lockfile) - try: + if ud.lockfile: + lf = bb.utils.lockfile(ud.lockfile) + self.d.setVar("BB_NO_NETWORK", network) if m.verify_donestamp(ud, self.d) and not m.need_update(ud, self.d): done = True @@ -1859,25 +1917,28 @@ class Fetch(object): logger.debug(str(e)) done = False + d = self.d if premirroronly: - self.d.setVar("BB_NO_NETWORK", "1") + # Only disable the network in a copy + d = bb.data.createCopy(self.d) + d.setVar("BB_NO_NETWORK", "1") firsterr = None verified_stamp = False if done: - verified_stamp = m.verify_donestamp(ud, self.d) - if not done and (not verified_stamp or m.need_update(ud, self.d)): + verified_stamp = m.verify_donestamp(ud, d) + if not done and (not verified_stamp or m.need_update(ud, d)): try: - if not trusted_network(self.d, ud.url): + if not trusted_network(d, ud.url): raise UntrustedUrl(ud.url) logger.debug("Trying Upstream") - m.download(ud, self.d) + m.download(ud, d) if hasattr(m, "build_mirror_data"): - m.build_mirror_data(ud, self.d) + m.build_mirror_data(ud, d) done = True # early checksum verify, so that if checksum mismatched, # fetcher still have chance to fetch from mirror - m.update_donestamp(ud, self.d) + m.update_donestamp(ud, d) except bb.fetch2.NetworkAccess: raise @@ -1896,22 +1957,23 @@ class Fetch(object): firsterr = e # Remove any incomplete fetch if not verified_stamp and m.cleanup_upon_failure(): - m.clean(ud, self.d) + m.clean(ud, d) logger.debug("Trying MIRRORS") - mirrors = mirror_from_string(self.d.getVar('MIRRORS')) - done = m.try_mirrors(self, ud, self.d, mirrors) + mirrors = mirror_from_string(d.getVar('MIRRORS')) + done = m.try_mirrors(self, ud, d, mirrors) - if not done or not m.done(ud, self.d): + if not done or not m.done(ud, d): if firsterr: logger.error(str(firsterr)) raise FetchError("Unable to fetch URL from any source.", u) - m.update_donestamp(ud, self.d) + m.update_donestamp(ud, d) except IOError as e: if e.errno in [errno.ESTALE]: logger.error("Stale Error Observed %s." % u) raise ChecksumError("Stale Error Detected") + raise except BBFetchException as e: if isinstance(e, NoChecksumError): @@ -1960,7 +2022,7 @@ class Fetch(object): if not ret: raise FetchError("URL doesn't work", u) - def unpack(self, root, urls=None): + def unpack(self, root, urls=None, update=False): """ Unpack urls to root """ @@ -1971,21 +2033,29 @@ class Fetch(object): unpack_tracer.start(root, self.ud, self.d) for u in urls: - ud = self.ud[u] - ud.setup_localpath(self.d) + try: + ud = self.ud[u] + ud.setup_localpath(self.d) - if ud.lockfile: - lf = bb.utils.lockfile(ud.lockfile) + if ud.lockfile: + lf = bb.utils.lockfile(ud.lockfile) - unpack_tracer.start_url(u) - ud.method.unpack(ud, root, self.d) - unpack_tracer.finish_url(u) + unpack_tracer.start_url(u) + if update: + ud.method.unpack_update(ud, root, self.d) + else: + ud.method.unpack(ud, root, self.d) + unpack_tracer.finish_url(u) - if ud.lockfile: - bb.utils.unlockfile(lf) + finally: + if ud.lockfile: + bb.utils.unlockfile(lf) unpack_tracer.complete() + def unpack_update(self, root, urls=None): + self.unpack(root, urls, update=True) + def clean(self, urls=None): """ Clean files that the fetcher gets or places @@ -1995,23 +2065,25 @@ class Fetch(object): urls = self.urls for url in urls: - if url not in self.ud: - self.ud[url] = FetchData(url, self.d) - ud = self.ud[url] - ud.setup_localpath(self.d) + try: + if url not in self.ud: + self.ud[url] = FetchData(url, self.d) + ud = self.ud[url] + ud.setup_localpath(self.d) - if not ud.localfile and ud.localpath is None: - continue + if not ud.localfile and ud.localpath is None: + continue - if ud.lockfile: - lf = bb.utils.lockfile(ud.lockfile) + if ud.lockfile: + lf = bb.utils.lockfile(ud.lockfile) - ud.method.clean(ud, self.d) - if ud.donestamp: - bb.utils.remove(ud.donestamp) + ud.method.clean(ud, self.d) + if ud.donestamp: + bb.utils.remove(ud.donestamp) - if ud.lockfile: - bb.utils.unlockfile(lf) + finally: + if ud.lockfile: + bb.utils.unlockfile(lf) def expanded_urldata(self, urls=None): """ @@ -2067,7 +2139,6 @@ class FetchConnectionCache(object): self.cache[cn].close() del self.cache[cn] -from . import cvs from . import git from . import gitsm from . import gitannex @@ -2078,9 +2149,7 @@ from . import ssh from . import sftp from . import s3 from . import perforce -from . import bzr from . import hg -from . import osc from . import repo from . import clearcase from . import npm @@ -2088,6 +2157,7 @@ from . import npmsw from . import az from . import crate from . import gcp +from . import gomod methods.append(local.Local()) methods.append(wget.Wget()) @@ -2095,14 +2165,11 @@ methods.append(svn.Svn()) methods.append(git.Git()) methods.append(gitsm.GitSM()) methods.append(gitannex.GitANNEX()) -methods.append(cvs.Cvs()) methods.append(ssh.SSH()) methods.append(sftp.SFTP()) methods.append(s3.S3()) methods.append(perforce.Perforce()) -methods.append(bzr.Bzr()) methods.append(hg.Hg()) -methods.append(osc.Osc()) methods.append(repo.Repo()) methods.append(clearcase.ClearCase()) methods.append(npm.Npm()) @@ -2110,3 +2177,5 @@ methods.append(npmsw.NpmShrinkWrap()) methods.append(az.Az()) methods.append(crate.Crate()) methods.append(gcp.GCP()) +methods.append(gomod.GoMod()) +methods.append(gomod.GoModGit()) diff --git a/bitbake/lib/bb/fetch2/az.py b/bitbake/lib/bb/fetch2/az.py index 3ccc594c..1d3664f2 100644 --- a/bitbake/lib/bb/fetch2/az.py +++ b/bitbake/lib/bb/fetch2/az.py @@ -36,6 +36,8 @@ class Az(Wget): az_sas = d.getVar('AZ_SAS') if az_sas and az_sas not in ud.url: + if not az_sas.startswith('?'): + raise FetchError("When using AZ_SAS, it must start with a '?' character to mark the start of the query-parameters.") ud.url += az_sas return Wget.checkstatus(self, fetch, ud, d, try_again) @@ -62,15 +64,18 @@ class Az(Wget): az_sas = d.getVar('AZ_SAS') if az_sas: + if not az_sas.startswith('?'): + raise FetchError("When using AZ_SAS, it must start with a '?' character to mark the start of the query-parameters.") azuri = '%s%s%s%s' % ('https://', ud.host, ud.path, az_sas) else: azuri = '%s%s%s' % ('https://', ud.host, ud.path) + dldir = d.getVar("DL_DIR") if os.path.exists(ud.localpath): # file exists, but we didnt complete it.. trying again. - fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % azuri) + fetchcmd += " -c -P %s '%s'" % (dldir, azuri) else: - fetchcmd += d.expand(" -P ${DL_DIR} '%s'" % azuri) + fetchcmd += " -P %s '%s'" % (dldir, azuri) try: self._runwget(ud, d, fetchcmd, False) diff --git a/bitbake/lib/bb/fetch2/bzr.py b/bitbake/lib/bb/fetch2/bzr.py deleted file mode 100644 index fc558f50..00000000 --- a/bitbake/lib/bb/fetch2/bzr.py +++ /dev/null @@ -1,128 +0,0 @@ -""" -BitBake 'Fetch' implementation for bzr. - -""" - -# Copyright (C) 2007 Ross Burton -# Copyright (C) 2007 Richard Purdie -# -# Classes for obtaining upstream sources for the -# BitBake build tools. -# Copyright (C) 2003, 2004 Chris Larson -# -# SPDX-License-Identifier: GPL-2.0-only -# - -import os -import bb -from bb.fetch2 import FetchMethod -from bb.fetch2 import FetchError -from bb.fetch2 import runfetchcmd -from bb.fetch2 import logger - -class Bzr(FetchMethod): - def supports(self, ud, d): - return ud.type in ['bzr'] - - def urldata_init(self, ud, d): - """ - init bzr specific variable within url data - """ - # Create paths to bzr checkouts - bzrdir = d.getVar("BZRDIR") or (d.getVar("DL_DIR") + "/bzr") - relpath = self._strip_leading_slashes(ud.path) - ud.pkgdir = os.path.join(bzrdir, ud.host, relpath) - - ud.setup_revisions(d) - - if not ud.revision: - ud.revision = self.latest_revision(ud, d) - - ud.localfile = d.expand('bzr_%s_%s_%s.tar.gz' % (ud.host, ud.path.replace('/', '.'), ud.revision)) - - def _buildbzrcommand(self, ud, d, command): - """ - Build up an bzr commandline based on ud - command is "fetch", "update", "revno" - """ - - basecmd = d.getVar("FETCHCMD_bzr") or "/usr/bin/env bzr" - - proto = ud.parm.get('protocol', 'http') - - bzrroot = ud.host + ud.path - - options = [] - - if command == "revno": - bzrcmd = "%s revno %s %s://%s" % (basecmd, " ".join(options), proto, bzrroot) - else: - if ud.revision: - options.append("-r %s" % ud.revision) - - if command == "fetch": - bzrcmd = "%s branch %s %s://%s" % (basecmd, " ".join(options), proto, bzrroot) - elif command == "update": - bzrcmd = "%s pull %s --overwrite" % (basecmd, " ".join(options)) - else: - raise FetchError("Invalid bzr command %s" % command, ud.url) - - return bzrcmd - - def download(self, ud, d): - """Fetch url""" - - if os.access(os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir), '.bzr'), os.R_OK): - bzrcmd = self._buildbzrcommand(ud, d, "update") - logger.debug("BZR Update %s", ud.url) - bb.fetch2.check_network_access(d, bzrcmd, ud.url) - runfetchcmd(bzrcmd, d, workdir=os.path.join(ud.pkgdir, os.path.basename(ud.path))) - else: - bb.utils.remove(os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir)), True) - bzrcmd = self._buildbzrcommand(ud, d, "fetch") - bb.fetch2.check_network_access(d, bzrcmd, ud.url) - logger.debug("BZR Checkout %s", ud.url) - bb.utils.mkdirhier(ud.pkgdir) - logger.debug("Running %s", bzrcmd) - runfetchcmd(bzrcmd, d, workdir=ud.pkgdir) - - scmdata = ud.parm.get("scmdata", "") - if scmdata == "keep": - tar_flags = "" - else: - tar_flags = "--exclude='.bzr' --exclude='.bzrtags'" - - # tar them up to a defined filename - runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.basename(ud.pkgdir)), - d, cleanup=[ud.localpath], workdir=ud.pkgdir) - - def supports_srcrev(self): - return True - - def _revision_key(self, ud, d, name): - """ - Return a unique key for the url - """ - return "bzr:" + ud.pkgdir - - def _latest_revision(self, ud, d, name): - """ - Return the latest upstream revision number - """ - logger.debug2("BZR fetcher hitting network for %s", ud.url) - - bb.fetch2.check_network_access(d, self._buildbzrcommand(ud, d, "revno"), ud.url) - - output = runfetchcmd(self._buildbzrcommand(ud, d, "revno"), d, True) - - return output.strip() - - def sortable_revision(self, ud, d, name): - """ - Return a sortable revision number which in our case is the revision number - """ - - return False, self._build_revision(ud, d) - - def _build_revision(self, ud, d): - return ud.revision diff --git a/bitbake/lib/bb/fetch2/clearcase.py b/bitbake/lib/bb/fetch2/clearcase.py index 1a9c8637..17500daf 100644 --- a/bitbake/lib/bb/fetch2/clearcase.py +++ b/bitbake/lib/bb/fetch2/clearcase.py @@ -108,7 +108,7 @@ class ClearCase(FetchMethod): ud.module.replace("/", "."), ud.label.replace("/", ".")) - ud.viewname = "%s-view%s" % (ud.identifier, d.getVar("DATETIME", d, True)) + ud.viewname = "%s-view%s" % (ud.identifier, d.getVar("DATETIME")) ud.csname = "%s-config-spec" % (ud.identifier) ud.ccasedir = os.path.join(d.getVar("DL_DIR"), ud.type) ud.viewdir = os.path.join(ud.ccasedir, ud.viewname) @@ -130,8 +130,6 @@ class ClearCase(FetchMethod): self.debug("configspecfile = %s" % ud.configspecfile) self.debug("localfile = %s" % ud.localfile) - ud.localfile = os.path.join(d.getVar("DL_DIR"), ud.localfile) - def _build_ccase_command(self, ud, command): """ Build up a commandline based on ud @@ -196,7 +194,7 @@ class ClearCase(FetchMethod): def need_update(self, ud, d): if ("LATEST" in ud.label) or (ud.customspec and "LATEST" in ud.customspec): - ud.identifier += "-%s" % d.getVar("DATETIME",d, True) + ud.identifier += "-%s" % d.getVar("DATETIME") return True if os.path.exists(ud.localpath): return False diff --git a/bitbake/lib/bb/fetch2/crate.py b/bitbake/lib/bb/fetch2/crate.py index 01d49435..6133d375 100644 --- a/bitbake/lib/bb/fetch2/crate.py +++ b/bitbake/lib/bb/fetch2/crate.py @@ -13,6 +13,7 @@ BitBake 'Fetch' implementation for crates.io import hashlib import json import os +import re import subprocess import bb from bb.fetch2 import logger, subprocess_setup, UnpackError @@ -45,6 +46,17 @@ class Crate(Wget): super(Crate, self).urldata_init(ud, d) + def _generate_index_path(self, name): + # https://doc.rust-lang.org/cargo/reference/registry-index.html#index-files + if len(name) == 1: + return f"1/{name}" + elif len(name) == 2: + return f"2/{name}" + elif len(name) == 3: + return f"3/{name[0]}/{name}" + else: + return f"{name[0:2]}/{name[2:4]}/{name}" + def _crate_urldata_init(self, ud, d): """ Sets up the download for a crate @@ -65,11 +77,15 @@ class Crate(Wget): # host (this is to allow custom crate registries to be specified host = '/'.join(parts[2:-2]) - # if using upstream just fix it up nicely + # If using crates.io use the CDN directly as per https://crates.io/data-access if host == 'crates.io': - host = 'crates.io/api/v1/crates' + ud.url = "https://static.crates.io/crates/%s/%s/download" % (name, version) + ud.versionsurl = 'https://index.crates.io/' + self._generate_index_path(name) + self.latest_versionstring = self.latest_versionstring_from_index + else: + ud.url = "https://%s/%s/%s/download" % (host, name, version) + ud.versionsurl = "https://%s/%s/versions" % (host, name) - ud.url = "https://%s/%s/%s/download" % (host, name, version) ud.parm['downloadfilename'] = "%s-%s.crate" % (name, version) if 'name' not in ud.parm: ud.parm['name'] = '%s-%s' % (name, version) @@ -139,3 +155,37 @@ class Crate(Wget): mdpath = os.path.join(bbpath, cratepath, mdfile) with open(mdpath, "w") as f: json.dump(metadata, f) + + def latest_versionstring(self, ud, d, filter_regex=None): + """ + Return the latest version available when versionsurl is the [name]/versions URL. + """ + from functools import cmp_to_key + json_data = json.loads(self._fetch_index(ud.versionsurl, ud, d)) + versions = [(0, i["num"], "") for i in json_data["versions"]] + if filter_regex: + versions = [v for v in versions if re.match(filter_regex, v[1])] + versions = sorted(versions, key=cmp_to_key(bb.utils.vercmp)) + + return (versions[-1][1], "") if versions else ("", "") + + def latest_versionstring_from_index(self, ud, d, filter_regex=None): + """ + Return the latest version available when versionsurl is a Cargo index + file. + https://doc.rust-lang.org/cargo/reference/registry-index.html#index-files + """ + from functools import cmp_to_key + + versions = [] + response = self._fetch_index(ud.versionsurl, ud, d) + for line in response.splitlines(): + data = json.loads(line) + if not data.get("yanked", False): + versions.append((0, data["vers"], "")) + + if filter_regex: + versions = [v for v in versions if re.match(filter_regex, v[1])] + + versions = sorted(versions, key=cmp_to_key(bb.utils.vercmp)) + return (versions[-1][1], "") if versions else ("", "") diff --git a/bitbake/lib/bb/fetch2/cvs.py b/bitbake/lib/bb/fetch2/cvs.py deleted file mode 100644 index 01de5ff4..00000000 --- a/bitbake/lib/bb/fetch2/cvs.py +++ /dev/null @@ -1,157 +0,0 @@ -""" -BitBake 'Fetch' implementations - -Classes for obtaining upstream sources for the -BitBake build tools. - -""" - -# Copyright (C) 2003, 2004 Chris Larson -# -# SPDX-License-Identifier: GPL-2.0-only -# -# Based on functions from the base bb module, Copyright 2003 Holger Schurig -# - -import os -import bb -from bb.fetch2 import FetchMethod, FetchError, MissingParameterError, logger -from bb.fetch2 import runfetchcmd - -class Cvs(FetchMethod): - """ - Class to fetch a module or modules from cvs repositories - """ - def supports(self, ud, d): - """ - Check to see if a given url can be fetched with cvs. - """ - return ud.type in ['cvs'] - - def urldata_init(self, ud, d): - if not "module" in ud.parm: - raise MissingParameterError("module", ud.url) - ud.module = ud.parm["module"] - - ud.tag = ud.parm.get('tag', "") - - # Override the default date in certain cases - if 'date' in ud.parm: - ud.date = ud.parm['date'] - elif ud.tag: - ud.date = "" - - norecurse = '' - if 'norecurse' in ud.parm: - norecurse = '_norecurse' - - fullpath = '' - if 'fullpath' in ud.parm: - fullpath = '_fullpath' - - ud.localfile = d.expand('%s_%s_%s_%s%s%s.tar.gz' % (ud.module.replace('/', '.'), ud.host, ud.tag, ud.date, norecurse, fullpath)) - - pkg = d.getVar('PN') - cvsdir = d.getVar("CVSDIR") or (d.getVar("DL_DIR") + "/cvs") - ud.pkgdir = os.path.join(cvsdir, pkg) - - def need_update(self, ud, d): - if (ud.date == "now"): - return True - if not os.path.exists(ud.localpath): - return True - return False - - def download(self, ud, d): - - method = ud.parm.get('method', 'pserver') - localdir = ud.parm.get('localdir', ud.module) - cvs_port = ud.parm.get('port', '') - - cvs_rsh = None - if method == "ext": - if "rsh" in ud.parm: - cvs_rsh = ud.parm["rsh"] - - if method == "dir": - cvsroot = ud.path - else: - cvsroot = ":" + method - cvsproxyhost = d.getVar('CVS_PROXY_HOST') - if cvsproxyhost: - cvsroot += ";proxy=" + cvsproxyhost - cvsproxyport = d.getVar('CVS_PROXY_PORT') - if cvsproxyport: - cvsroot += ";proxyport=" + cvsproxyport - cvsroot += ":" + ud.user - if ud.pswd: - cvsroot += ":" + ud.pswd - cvsroot += "@" + ud.host + ":" + cvs_port + ud.path - - options = [] - if 'norecurse' in ud.parm: - options.append("-l") - if ud.date: - # treat YYYYMMDDHHMM specially for CVS - if len(ud.date) == 12: - options.append("-D \"%s %s:%s UTC\"" % (ud.date[0:8], ud.date[8:10], ud.date[10:12])) - else: - options.append("-D \"%s UTC\"" % ud.date) - if ud.tag: - options.append("-r %s" % ud.tag) - - cvsbasecmd = d.getVar("FETCHCMD_cvs") or "/usr/bin/env cvs" - cvscmd = cvsbasecmd + " '-d" + cvsroot + "' co " + " ".join(options) + " " + ud.module - cvsupdatecmd = cvsbasecmd + " '-d" + cvsroot + "' update -d -P " + " ".join(options) - - if cvs_rsh: - cvscmd = "CVS_RSH=\"%s\" %s" % (cvs_rsh, cvscmd) - cvsupdatecmd = "CVS_RSH=\"%s\" %s" % (cvs_rsh, cvsupdatecmd) - - # create module directory - logger.debug2("Fetch: checking for module directory") - moddir = os.path.join(ud.pkgdir, localdir) - workdir = None - if os.access(os.path.join(moddir, 'CVS'), os.R_OK): - logger.info("Update " + ud.url) - bb.fetch2.check_network_access(d, cvsupdatecmd, ud.url) - # update sources there - workdir = moddir - cmd = cvsupdatecmd - else: - logger.info("Fetch " + ud.url) - # check out sources there - bb.utils.mkdirhier(ud.pkgdir) - workdir = ud.pkgdir - logger.debug("Running %s", cvscmd) - bb.fetch2.check_network_access(d, cvscmd, ud.url) - cmd = cvscmd - - runfetchcmd(cmd, d, cleanup=[moddir], workdir=workdir) - - if not os.access(moddir, os.R_OK): - raise FetchError("Directory %s was not readable despite sucessful fetch?!" % moddir, ud.url) - - scmdata = ud.parm.get("scmdata", "") - if scmdata == "keep": - tar_flags = "" - else: - tar_flags = "--exclude='CVS'" - - # tar them up to a defined filename - workdir = None - if 'fullpath' in ud.parm: - workdir = ud.pkgdir - cmd = "tar %s -czf %s %s" % (tar_flags, ud.localpath, localdir) - else: - workdir = os.path.dirname(os.path.realpath(moddir)) - cmd = "tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.basename(moddir)) - - runfetchcmd(cmd, d, cleanup=[ud.localpath], workdir=workdir) - - def clean(self, ud, d): - """ Clean CVS Files and tarballs """ - - bb.utils.remove(ud.pkgdir, True) - bb.utils.remove(ud.localpath) - diff --git a/bitbake/lib/bb/fetch2/gcp.py b/bitbake/lib/bb/fetch2/gcp.py index f40ce2ea..86546d40 100644 --- a/bitbake/lib/bb/fetch2/gcp.py +++ b/bitbake/lib/bb/fetch2/gcp.py @@ -46,8 +46,7 @@ class GCP(FetchMethod): else: ud.basename = os.path.basename(ud.path) - ud.localfile = d.expand(urllib.parse.unquote(ud.basename)) - ud.basecmd = "gsutil stat" + ud.localfile = ud.basename def get_gcp_client(self): from google.cloud import storage @@ -58,17 +57,20 @@ class GCP(FetchMethod): Fetch urls using the GCP API. Assumes localpath was called first. """ + from google.api_core.exceptions import NotFound logger.debug2(f"Trying to download gs://{ud.host}{ud.path} to {ud.localpath}") if self.gcp_client is None: self.get_gcp_client() - bb.fetch2.check_network_access(d, ud.basecmd, f"gs://{ud.host}{ud.path}") - runfetchcmd("%s %s" % (ud.basecmd, f"gs://{ud.host}{ud.path}"), d) + bb.fetch2.check_network_access(d, "blob.download_to_filename", f"gs://{ud.host}{ud.path}") # Path sometimes has leading slash, so strip it path = ud.path.lstrip("/") blob = self.gcp_client.bucket(ud.host).blob(path) - blob.download_to_filename(ud.localpath) + try: + blob.download_to_filename(ud.localpath) + except NotFound: + raise FetchError("The GCP API threw a NotFound exception") # Additional sanity checks copied from the wget class (although there # are no known issues which mean these are required, treat the GCP API @@ -90,8 +92,7 @@ class GCP(FetchMethod): if self.gcp_client is None: self.get_gcp_client() - bb.fetch2.check_network_access(d, ud.basecmd, f"gs://{ud.host}{ud.path}") - runfetchcmd("%s %s" % (ud.basecmd, f"gs://{ud.host}{ud.path}"), d) + bb.fetch2.check_network_access(d, "gcp_client.bucket(ud.host).blob(path).exists()", f"gs://{ud.host}{ud.path}") # Path sometimes has leading slash, so strip it path = ud.path.lstrip("/") diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index c7ff769f..0649429e 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -9,15 +9,6 @@ Supported SRC_URI options are: - branch The git branch to retrieve from. The default is "master" - This option also supports multiple branch fetching, with branches - separated by commas. In multiple branches case, the name option - must have the same number of names to match the branches, which is - used to specify the SRC_REV for the branch - e.g: - SRC_URI="git://some.host/somepath;branch=branchX,branchY;name=nameX,nameY" - SRCREV_nameX = "xxxxxxxxxxxxxxxxxxxx" - SRCREV_nameY = "YYYYYYYYYYYYYYYYYYYY" - - tag The git tag to retrieve. The default is "master" @@ -81,6 +72,7 @@ import shlex import shutil import subprocess import tempfile +import urllib import bb import bb.progress from contextlib import contextmanager @@ -171,7 +163,7 @@ class Git(FetchMethod): bb.warn("URL: %s uses git protocol which is no longer supported by github. Please change to ;protocol=https in the url." % ud.url) if not ud.proto in ('git', 'file', 'ssh', 'http', 'https', 'rsync'): - raise bb.fetch2.ParameterError("Invalid protocol type", ud.url) + raise bb.fetch2.ParameterError(f"Invalid protocol type: '{ud.proto}'", ud.url) ud.nocheckout = ud.parm.get("nocheckout","0") == "1" @@ -190,14 +182,11 @@ class Git(FetchMethod): ud.bareclone = ud.parm.get("bareclone","0") == "1" if ud.bareclone: ud.nocheckout = 1 - - ud.unresolvedrev = {} - branches = ud.parm.get("branch", "").split(',') - if branches == [""] and not ud.nobranch: - bb.warn("URL: %s does not set any branch parameter. The future default branch used by tools and repositories is uncertain and we will therefore soon require this is set in all git urls." % ud.url) - branches = ["master"] - if len(branches) != len(ud.names): - raise bb.fetch2.ParameterError("The number of name and branch parameters is not balanced", ud.url) + + ud.unresolvedrev = "" + ud.branch = ud.parm.get("branch", "") + if not ud.branch and not ud.nobranch: + raise bb.fetch2.ParameterError("The url does not set any branch parameter or set nobranch=1.", ud.url) ud.noshared = d.getVar("BB_GIT_NOSHARED") == "1" @@ -207,8 +196,11 @@ class Git(FetchMethod): if ud.bareclone: ud.cloneflags += " --mirror" + ud.shallow_skip_fast = False ud.shallow = d.getVar("BB_GIT_SHALLOW") == "1" ud.shallow_extra_refs = (d.getVar("BB_GIT_SHALLOW_EXTRA_REFS") or "").split() + if 'tag' in ud.parm: + ud.shallow_extra_refs.append("refs/tags/" + ud.parm['tag']) depth_default = d.getVar("BB_GIT_SHALLOW_DEPTH") if depth_default is not None: @@ -225,32 +217,27 @@ class Git(FetchMethod): revs_default = d.getVar("BB_GIT_SHALLOW_REVS") ud.shallow_revs = [] - ud.branches = {} - for pos, name in enumerate(ud.names): - branch = branches[pos] - ud.branches[name] = branch - ud.unresolvedrev[name] = branch - - shallow_depth = d.getVar("BB_GIT_SHALLOW_DEPTH_%s" % name) - if shallow_depth is not None: - try: - shallow_depth = int(shallow_depth or 0) - except ValueError: - raise bb.fetch2.FetchError("Invalid depth for BB_GIT_SHALLOW_DEPTH_%s: %s" % (name, shallow_depth)) - else: - if shallow_depth < 0: - raise bb.fetch2.FetchError("Invalid depth for BB_GIT_SHALLOW_DEPTH_%s: %s" % (name, shallow_depth)) - ud.shallow_depths[name] = shallow_depth - - revs = d.getVar("BB_GIT_SHALLOW_REVS_%s" % name) - if revs is not None: - ud.shallow_revs.extend(revs.split()) - elif revs_default is not None: - ud.shallow_revs.extend(revs_default.split()) - - if (ud.shallow and - not ud.shallow_revs and - all(ud.shallow_depths[n] == 0 for n in ud.names)): + + ud.unresolvedrev = ud.branch + + shallow_depth = d.getVar("BB_GIT_SHALLOW_DEPTH_%s" % ud.name) + if shallow_depth is not None: + try: + shallow_depth = int(shallow_depth or 0) + except ValueError: + raise bb.fetch2.FetchError("Invalid depth for BB_GIT_SHALLOW_DEPTH_%s: %s" % (ud.name, shallow_depth)) + else: + if shallow_depth < 0: + raise bb.fetch2.FetchError("Invalid depth for BB_GIT_SHALLOW_DEPTH_%s: %s" % (ud.name, shallow_depth)) + ud.shallow_depths[ud.name] = shallow_depth + + revs = d.getVar("BB_GIT_SHALLOW_REVS_%s" % ud.name) + if revs is not None: + ud.shallow_revs.extend(revs.split()) + elif revs_default is not None: + ud.shallow_revs.extend(revs_default.split()) + + if ud.shallow and not ud.shallow_revs and ud.shallow_depths[ud.name] == 0: # Shallow disabled for this URL ud.shallow = False @@ -259,10 +246,9 @@ class Git(FetchMethod): # rev of this repository. This will get resolved into a revision # later. If an actual revision happens to have also been provided # then this setting will be overridden. - for name in ud.names: - ud.unresolvedrev[name] = 'HEAD' + ud.unresolvedrev = 'HEAD' - ud.basecmd = d.getVar("FETCHCMD_git") or "git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all" + ud.basecmd = d.getVar("FETCHCMD_git") or "git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all -c clone.defaultRemoteName=origin" write_tarballs = d.getVar("BB_GENERATE_MIRROR_TARBALLS") or "0" ud.write_tarballs = write_tarballs != "0" or ud.rebaseable @@ -270,12 +256,11 @@ class Git(FetchMethod): ud.setup_revisions(d) - for name in ud.names: - # Ensure any revision that doesn't look like a SHA-1 is translated into one - if not sha1_re.match(ud.revisions[name] or ''): - if ud.revisions[name]: - ud.unresolvedrev[name] = ud.revisions[name] - ud.revisions[name] = self.latest_revision(ud, d, name) + # Ensure any revision that doesn't look like a SHA-1 is translated into one + if not sha1_re.match(ud.revision or ''): + if ud.revision: + ud.unresolvedrev = ud.revision + ud.revision = self.latest_revision(ud, d, ud.name) gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) if gitsrcname.startswith('.'): @@ -286,8 +271,7 @@ class Git(FetchMethod): # upstream repo in the future, the mirror will remain intact and still # contain the revision if ud.rebaseable: - for name in ud.names: - gitsrcname = gitsrcname + '_' + ud.revisions[name] + gitsrcname = gitsrcname + '_' + ud.revision dl_dir = d.getVar("DL_DIR") gitdir = d.getVar("GITDIR") or (dl_dir + "/git2") @@ -305,15 +289,14 @@ class Git(FetchMethod): if ud.shallow_revs: tarballname = "%s_%s" % (tarballname, "_".join(sorted(ud.shallow_revs))) - for name, revision in sorted(ud.revisions.items()): - tarballname = "%s_%s" % (tarballname, ud.revisions[name][:7]) - depth = ud.shallow_depths[name] - if depth: - tarballname = "%s-%s" % (tarballname, depth) + tarballname = "%s_%s" % (tarballname, ud.revision[:7]) + depth = ud.shallow_depths[ud.name] + if depth: + tarballname = "%s-%s" % (tarballname, depth) shallow_refs = [] if not ud.nobranch: - shallow_refs.extend(ud.branches.values()) + shallow_refs.append(ud.branch) if ud.shallow_extra_refs: shallow_refs.extend(r.replace('refs/heads/', '').replace('*', 'ALL') for r in ud.shallow_extra_refs) if shallow_refs: @@ -338,18 +321,21 @@ class Git(FetchMethod): return True if ud.shallow and ud.write_shallow_tarballs and self.clonedir_need_shallow_revs(ud, d): return True - for name in ud.names: - if not self._contains_ref(ud, d, name, ud.clonedir): - return True + if not self._contains_ref(ud, d, ud.name, ud.clonedir): + return True + if 'tag' in ud.parm and not self._contains_ref(ud, d, ud.name, ud.clonedir, tag=True): + return True return False def lfs_need_update(self, ud, d): + if not self._need_lfs(ud): + return False + if self.clonedir_need_update(ud, d): return True - for name in ud.names: - if not self._lfs_objects_downloaded(ud, d, name, ud.clonedir): - return True + if not self._lfs_objects_downloaded(ud, d, ud.clonedir): + return True return False def clonedir_need_shallow_revs(self, ud, d): @@ -366,6 +352,13 @@ class Git(FetchMethod): def tarball_need_update(self, ud): return ud.write_tarballs and not os.path.exists(ud.fullmirror) + def update_mirror_links(self, ud, origud): + super().update_mirror_links(ud, origud) + # When using shallow mode, add a symlink to the original fullshallow + # path to ensure a valid symlink even in the `PREMIRRORS` case + if origud.shallow and not os.path.exists(origud.fullshallow): + self.ensure_symlink(ud.localpath, origud.fullshallow) + def try_premirror(self, ud, d): # If we don't do this, updating an existing checkout with only premirrors # is not possible @@ -399,14 +392,14 @@ class Git(FetchMethod): bb.utils.mkdirhier(ud.clonedir) runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir) else: - tmpdir = tempfile.mkdtemp(dir=d.getVar('DL_DIR')) - runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=tmpdir) - output = runfetchcmd("%s remote" % ud.basecmd, d, quiet=True, workdir=ud.clonedir) - if 'mirror' in output: - runfetchcmd("%s remote rm mirror" % ud.basecmd, d, workdir=ud.clonedir) - runfetchcmd("%s remote add --mirror=fetch mirror %s" % (ud.basecmd, tmpdir), d, workdir=ud.clonedir) - fetch_cmd = "LANG=C %s fetch -f --update-head-ok --progress mirror " % (ud.basecmd) - runfetchcmd(fetch_cmd, d, workdir=ud.clonedir) + with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir: + runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=tmpdir) + output = runfetchcmd("%s remote" % ud.basecmd, d, quiet=True, workdir=ud.clonedir) + if 'mirror' in output: + runfetchcmd("%s remote rm mirror" % ud.basecmd, d, workdir=ud.clonedir) + runfetchcmd("%s remote add --mirror=fetch mirror %s" % (ud.basecmd, tmpdir), d, workdir=ud.clonedir) + fetch_cmd = "LANG=C %s fetch -f --update-head-ok --progress mirror " % (ud.basecmd) + runfetchcmd(fetch_cmd, d, workdir=ud.clonedir) repourl = self._get_repo_url(ud) needs_clone = False @@ -446,6 +439,24 @@ class Git(FetchMethod): if ud.proto.lower() != 'file': bb.fetch2.check_network_access(d, clone_cmd, ud.url) progresshandler = GitProgressHandler(d) + + # Try creating a fast initial shallow clone + # Enabling ud.shallow_skip_fast will skip this + # If the Git error "Server does not allow request for unadvertised object" + # occurs, shallow_skip_fast is enabled automatically. + # This may happen if the Git server does not allow the request + # or if the Git client has issues with this functionality. + if ud.shallow and not ud.shallow_skip_fast: + try: + self.clone_shallow_with_tarball(ud, d) + # When the shallow clone has succeeded, use the shallow tarball + ud.localpath = ud.fullshallow + return + except: + logger.warning("Creating fast initial shallow clone failed, try initial regular clone now.") + + # When skipping fast initial shallow or the fast inital shallow clone failed: + # Try again with an initial regular clone runfetchcmd(clone_cmd, d, log=progresshandler) # Update the checkout if needed @@ -464,18 +475,17 @@ class Git(FetchMethod): bb.fetch2.check_network_access(d, fetch_cmd, ud.url) progresshandler = GitProgressHandler(d) runfetchcmd(fetch_cmd, d, log=progresshandler, workdir=ud.clonedir) - runfetchcmd("%s prune-packed" % ud.basecmd, d, workdir=ud.clonedir) + runfetchcmd("%s repack -adk" % ud.basecmd, d, workdir=ud.clonedir) runfetchcmd("%s pack-refs --all" % ud.basecmd, d, workdir=ud.clonedir) - runfetchcmd("%s pack-redundant --all | xargs -r rm" % ud.basecmd, d, workdir=ud.clonedir) + runfetchcmd("%s prune-packed" % ud.basecmd, d, workdir=ud.clonedir) try: os.unlink(ud.fullmirror) except OSError as exc: if exc.errno != errno.ENOENT: raise - for name in ud.names: - if not self._contains_ref(ud, d, name, ud.clonedir): - raise bb.fetch2.FetchError("Unable to find revision %s in branch %s even from upstream" % (ud.revisions[name], ud.branches[name])) + if not self._contains_ref(ud, d, ud.name, ud.clonedir): + raise bb.fetch2.FetchError("Unable to find revision %s in branch %s even from upstream" % (ud.revision, ud.branch)) if ud.shallow and ud.write_shallow_tarballs: missing_rev = self.clonedir_need_shallow_revs(ud, d) @@ -483,136 +493,177 @@ class Git(FetchMethod): raise bb.fetch2.FetchError("Unable to find revision %s even from upstream" % missing_rev) if self.lfs_need_update(ud, d): - # Unpack temporary working copy, use it to run 'git checkout' to force pre-fetching - # of all LFS blobs needed at the srcrev. - # - # It would be nice to just do this inline here by running 'git-lfs fetch' - # on the bare clonedir, but that operation requires a working copy on some - # releases of Git LFS. - with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir: - # Do the checkout. This implicitly involves a Git LFS fetch. - Git.unpack(self, ud, tmpdir, d) - - # Scoop up a copy of any stuff that Git LFS downloaded. Merge them into - # the bare clonedir. - # - # As this procedure is invoked repeatedly on incremental fetches as - # a recipe's SRCREV is bumped throughout its lifetime, this will - # result in a gradual accumulation of LFS blobs in /lfs - # corresponding to all the blobs reachable from the different revs - # fetched across time. - # - # Only do this if the unpack resulted in a .git/lfs directory being - # created; this only happens if at least one blob needed to be - # downloaded. - if os.path.exists(os.path.join(ud.destdir, ".git", "lfs")): - runfetchcmd("tar -cf - lfs | tar -xf - -C %s" % ud.clonedir, d, workdir="%s/.git" % ud.destdir) - - def build_mirror_data(self, ud, d): + self.lfs_fetch(ud, d, ud.clonedir, ud.revision) - # Create as a temp file and move atomically into position to avoid races - @contextmanager - def create_atomic(filename): - fd, tfile = tempfile.mkstemp(dir=os.path.dirname(filename)) - try: - yield tfile - umask = os.umask(0o666) - os.umask(umask) - os.chmod(tfile, (0o666 & ~umask)) - os.rename(tfile, filename) - finally: - os.close(fd) + def lfs_fetch(self, ud, d, clonedir, revision, fetchall=False, progresshandler=None): + """Helper method for fetching Git LFS data""" + try: + if self._need_lfs(ud) and self._contains_lfs(ud, d, clonedir) and len(revision): + self._ensure_git_lfs(d, ud) + + # Using worktree with the revision because .lfsconfig may exists + worktree_add_cmd = "%s worktree add wt %s" % (ud.basecmd, revision) + runfetchcmd(worktree_add_cmd, d, log=progresshandler, workdir=clonedir) + lfs_fetch_cmd = "%s lfs fetch %s" % (ud.basecmd, "--all" if fetchall else "") + runfetchcmd(lfs_fetch_cmd, d, log=progresshandler, workdir=(clonedir + "/wt")) + worktree_rem_cmd = "%s worktree remove -f wt" % ud.basecmd + runfetchcmd(worktree_rem_cmd, d, log=progresshandler, workdir=clonedir) + except: + logger.warning("Fetching LFS did not succeed.") + + @contextmanager + def create_atomic(self, filename): + """Create as a temp file and move atomically into position to avoid races""" + fd, tfile = tempfile.mkstemp(dir=os.path.dirname(filename)) + try: + yield tfile + umask = os.umask(0o666) + os.umask(umask) + os.chmod(tfile, (0o666 & ~umask)) + os.rename(tfile, filename) + finally: + os.close(fd) + def build_mirror_data(self, ud, d): if ud.shallow and ud.write_shallow_tarballs: if not os.path.exists(ud.fullshallow): if os.path.islink(ud.fullshallow): os.unlink(ud.fullshallow) - tempdir = tempfile.mkdtemp(dir=d.getVar('DL_DIR')) - shallowclone = os.path.join(tempdir, 'git') - try: - self.clone_shallow_local(ud, shallowclone, d) - - logger.info("Creating tarball of git repository") - with create_atomic(ud.fullshallow) as tfile: - runfetchcmd("tar -czf %s ." % tfile, d, workdir=shallowclone) - runfetchcmd("touch %s.done" % ud.fullshallow, d) - finally: - bb.utils.remove(tempdir, recurse=True) + self.clone_shallow_with_tarball(ud, d) elif ud.write_tarballs and not os.path.exists(ud.fullmirror): if os.path.islink(ud.fullmirror): os.unlink(ud.fullmirror) logger.info("Creating tarball of git repository") - with create_atomic(ud.fullmirror) as tfile: + with self.create_atomic(ud.fullmirror) as tfile: mtime = runfetchcmd("{} log --all -1 --format=%cD".format(ud.basecmd), d, quiet=True, workdir=ud.clonedir) runfetchcmd("tar -czf %s --owner oe:0 --group oe:0 --mtime \"%s\" ." % (tfile, mtime), d, workdir=ud.clonedir) runfetchcmd("touch %s.done" % ud.fullmirror, d) + def clone_shallow_with_tarball(self, ud, d): + for fast in [True, False]: + ud.shallow_skip_fast = not fast + with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tempdir: + shallowclone = os.path.join(tempdir, 'git') + try: + self.clone_shallow_local(ud, shallowclone, d) + except: + if not fast: + raise + logger.warning("Fast shallow clone failed, try to skip fast mode now.") + continue + logger.info("Creating tarball of git repository") + with self.create_atomic(ud.fullshallow) as tfile: + runfetchcmd("tar -czf %s ." % tfile, d, workdir=shallowclone) + runfetchcmd("touch %s.done" % ud.fullshallow, d) + return True + return False + def clone_shallow_local(self, ud, dest, d): - """Clone the repo and make it shallow. + """ + Shallow fetch from ud.clonedir (${DL_DIR}/git2/ by default): + - For BB_GIT_SHALLOW_DEPTH: git fetch --depth rev + - For BB_GIT_SHALLOW_REVS: git fetch --shallow-exclude= rev + """ - The upstream url of the new clone isn't set at this time, as it'll be - set correctly when unpacked.""" - runfetchcmd("%s clone %s %s %s" % (ud.basecmd, ud.cloneflags, ud.clonedir, dest), d) + progresshandler = GitProgressHandler(d) + repourl = self._get_repo_url(ud) + bb.utils.mkdirhier(dest) + init_cmd = "%s init -q" % ud.basecmd + if ud.bareclone: + init_cmd += " --bare" + runfetchcmd(init_cmd, d, workdir=dest) + # Use repourl when creating a fast initial shallow clone + # Prefer already existing full bare clones if available + if not ud.shallow_skip_fast and not os.path.exists(ud.clonedir): + remote = shlex.quote(repourl) + else: + remote = ud.clonedir + runfetchcmd("%s remote add origin %s" % (ud.basecmd, remote), d, workdir=dest) - to_parse, shallow_branches = [], [] - for name in ud.names: - revision = ud.revisions[name] - depth = ud.shallow_depths[name] - if depth: - to_parse.append('%s~%d^{}' % (revision, depth - 1)) + # Check the histories which should be excluded + shallow_exclude = '' + for revision in ud.shallow_revs: + shallow_exclude += " --shallow-exclude=%s" % revision - # For nobranch, we need a ref, otherwise the commits will be - # removed, and for non-nobranch, we truncate the branch to our - # srcrev, to avoid keeping unnecessary history beyond that. - branch = ud.branches[name] - if ud.nobranch: - ref = "refs/shallow/%s" % name - elif ud.bareclone: - ref = "refs/heads/%s" % branch - else: - ref = "refs/remotes/origin/%s" % branch + revision = ud.revision + depth = ud.shallow_depths[ud.name] + + # The --depth and --shallow-exclude can't be used together + if depth and shallow_exclude: + raise bb.fetch2.FetchError("BB_GIT_SHALLOW_REVS is set, but BB_GIT_SHALLOW_DEPTH is not 0.") - shallow_branches.append(ref) - runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest) + # For nobranch, we need a ref, otherwise the commits will be + # removed, and for non-nobranch, we truncate the branch to our + # srcrev, to avoid keeping unnecessary history beyond that. + branch = ud.branch + if ud.nobranch: + ref = "refs/shallow/%s" % ud.name + elif ud.bareclone: + ref = "refs/heads/%s" % branch + else: + ref = "refs/remotes/origin/%s" % branch - # Map srcrev+depths to revisions - parsed_depths = runfetchcmd("%s rev-parse %s" % (ud.basecmd, " ".join(to_parse)), d, workdir=dest) + fetch_cmd = "%s fetch origin %s" % (ud.basecmd, revision) + if depth: + fetch_cmd += " --depth %s" % depth - # Resolve specified revisions - parsed_revs = runfetchcmd("%s rev-parse %s" % (ud.basecmd, " ".join('"%s^{}"' % r for r in ud.shallow_revs)), d, workdir=dest) - shallow_revisions = parsed_depths.splitlines() + parsed_revs.splitlines() + if shallow_exclude: + fetch_cmd += shallow_exclude + + # Advertise the revision for lower version git such as 2.25.1: + # error: Server does not allow request for unadvertised object. + # The ud.clonedir is a local temporary dir, will be removed when + # fetch is done, so we can do anything on it. + adv_cmd = 'git branch -f advertise-%s %s' % (revision, revision) + if ud.shallow_skip_fast: + runfetchcmd(adv_cmd, d, workdir=ud.clonedir) + + runfetchcmd(fetch_cmd, d, workdir=dest) + runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest) + # Fetch Git LFS data + self.lfs_fetch(ud, d, dest, ud.revision) # Apply extra ref wildcards - all_refs = runfetchcmd('%s for-each-ref "--format=%%(refname)"' % ud.basecmd, - d, workdir=dest).splitlines() + all_refs_remote = runfetchcmd("%s ls-remote origin 'refs/*'" % ud.basecmd, \ + d, workdir=dest).splitlines() + all_refs = [] + for line in all_refs_remote: + all_refs.append(line.split()[-1]) + extra_refs = [] for r in ud.shallow_extra_refs: if not ud.bareclone: r = r.replace('refs/heads/', 'refs/remotes/origin/') if '*' in r: matches = filter(lambda a: fnmatch.fnmatchcase(a, r), all_refs) - shallow_branches.extend(matches) + extra_refs.extend(matches) else: - shallow_branches.append(r) + extra_refs.append(r) + + for ref in extra_refs: + ref_fetch = ref.replace('refs/heads/', '').replace('refs/remotes/origin/', '').replace('refs/tags/', '') + runfetchcmd("%s fetch origin --depth 1 -- %s" % + (ud.basecmd, shlex.quote(ref_fetch)), d, workdir=dest) + revision = runfetchcmd("%s rev-parse FETCH_HEAD" % ud.basecmd, d, workdir=dest) + runfetchcmd("%s update-ref %s %s" % + (ud.basecmd, shlex.quote(ref), revision), d, workdir=dest) - # Make the repository shallow - shallow_cmd = [self.make_shallow_path, '-s'] - for b in shallow_branches: - shallow_cmd.append('-r') - shallow_cmd.append(b) - shallow_cmd.extend(shallow_revisions) - runfetchcmd(subprocess.list2cmdline(shallow_cmd), d, workdir=dest) + # The url is local ud.clonedir, set it to upstream one + runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=dest) - def unpack(self, ud, destdir, d): + def unpack_update(self, ud, destdir, d): + return self.unpack(ud, destdir, d, update=True) + + def unpack(self, ud, destdir, d, update=False): """ unpack the downloaded src to destdir""" subdir = ud.parm.get("subdir") subpath = ud.parm.get("subpath") readpathspec = "" - def_destsuffix = "git/" + def_destsuffix = (d.getVar("BB_GIT_DEFAULT_DESTSUFFIX") or "git") + "/" if subpath: readpathspec = ":%s" % subpath @@ -630,7 +681,7 @@ class Git(FetchMethod): destsuffix = ud.parm.get("destsuffix", def_destsuffix) destdir = ud.destdir = os.path.join(destdir, destsuffix) - if os.path.exists(destdir): + if os.path.exists(destdir) and not update: bb.utils.prunedir(destdir) if not ud.bareclone: ud.unpack_tracer.unpack("git", destdir) @@ -641,11 +692,15 @@ class Git(FetchMethod): ud.basecmd = "GIT_LFS_SKIP_SMUDGE=1 " + ud.basecmd source_found = False + update_mode = False source_error = [] clonedir_is_up_to_date = not self.clonedir_need_update(ud, d) if clonedir_is_up_to_date: - runfetchcmd("%s clone %s %s/ %s" % (ud.basecmd, ud.cloneflags, ud.clonedir, destdir), d) + if update and os.path.exists(destdir): + update_mode = True + else: + runfetchcmd("%s clone %s %s/ %s" % (ud.basecmd, ud.cloneflags, ud.clonedir, destdir), d) source_found = True else: source_error.append("clone directory not available or not up to date: " + ud.clonedir) @@ -653,8 +708,11 @@ class Git(FetchMethod): if not source_found: if ud.shallow: if os.path.exists(ud.fullshallow): - bb.utils.mkdirhier(destdir) - runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=destdir) + if update and os.path.exists(destdir): + update_mode = True + else: + bb.utils.mkdirhier(destdir) + runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=destdir) source_found = True else: source_error.append("shallow clone not available: " + ud.fullshallow) @@ -664,30 +722,83 @@ class Git(FetchMethod): if not source_found: raise bb.fetch2.UnpackError("No up to date source found: " + "; ".join(source_error), ud.url) + if update_mode: + if ud.shallow: + raise bb.fetch2.UnpackError("Can't update shallow clones checkouts without network access, not supported.", ud.url) + + output = runfetchcmd("%s status --untracked-files=no --porcelain" % (ud.basecmd), d, workdir=destdir) + if output: + raise bb.fetch2.LocalModificationsError(destdir, ud.url, output) + + # Set up remote for the download location if it doesn't exist + try: + runfetchcmd("%s remote get-url dldir" % (ud.basecmd), d, workdir=destdir) + except bb.fetch2.FetchError: + if ud.clonedir: + runfetchcmd("%s remote add dldir file://%s" % (ud.basecmd, ud.clonedir), d, workdir=destdir) + try: + runfetchcmd("%s fetch dldir" % (ud.basecmd), d, workdir=destdir) + except bb.fetch2.FetchError as e: + raise bb.fetch2.UnpackError("Failed to fetch from dldir remote: %s" % str(e), ud.url) + try: + runfetchcmd("%s rebase --no-autosquash --no-autostash %s" % (ud.basecmd, ud.revision), d, workdir=destdir) + except bb.fetch2.FetchError as e: + # If rebase failed, abort it + try: + runfetchcmd("%s rebase --abort" % (ud.basecmd), d, workdir=destdir) + except Exception: + pass + raise bb.fetch2.RebaseError(destdir, ud.url, str(e)) + return True + + # If there is a tag parameter in the url and we also have a fixed srcrev, check the tag + # matches the revision + if 'tag' in ud.parm and sha1_re.match(ud.revision): + output = runfetchcmd("%s rev-list -n 1 %s" % (ud.basecmd, ud.parm['tag']), d, workdir=destdir) + output = output.strip() + if output != ud.revision: + # It is possible ud.revision is the revision on an annotated tag which won't match the output of rev-list + # If it resolves to the same thing there isn't a problem. + output2 = runfetchcmd("%s rev-list -n 1 %s" % (ud.basecmd, ud.revision), d, workdir=destdir) + output2 = output2.strip() + if output != output2: + raise bb.fetch2.FetchError("The revision the git tag '%s' resolved to didn't match the SRCREV in use (%s vs %s)" % (ud.parm['tag'], output, ud.revision), ud.url) + repourl = self._get_repo_url(ud) runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=destdir) + if ud.clonedir: + try: + runfetchcmd("%s remote get-url dldir" % (ud.basecmd), d, workdir=destdir) + except bb.fetch2.FetchError: + runfetchcmd("%s remote add dldir file://%s" % (ud.basecmd, ud.clonedir), d, workdir=destdir) if self._contains_lfs(ud, d, destdir): - if need_lfs and not self._find_git_lfs(d): - raise bb.fetch2.FetchError("Repository %s has LFS content, install git-lfs on host to download (or set lfs=0 to ignore it)" % (repourl)) - elif not need_lfs: + if not need_lfs: bb.note("Repository %s has LFS content but it is not being fetched" % (repourl)) else: + self._ensure_git_lfs(d, ud) + runfetchcmd("%s lfs install --local" % ud.basecmd, d, workdir=destdir) if not ud.nocheckout: if subpath: - runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.revisions[ud.names[0]], readpathspec), d, + runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.revision, readpathspec), d, workdir=destdir) runfetchcmd("%s checkout-index -q -f -a" % ud.basecmd, d, workdir=destdir) + runfetchcmd("%s update-ref --no-deref HEAD %s" % (ud.basecmd, ud.revision), + d, workdir=destdir) + if not ud.nobranch: + branchname = ud.branch + runfetchcmd("%s update-ref refs/heads/%s %s" % (ud.basecmd, branchname, + ud.revision), d, workdir=destdir) elif not ud.nobranch: - branchname = ud.branches[ud.names[0]] + branchname = ud.branch runfetchcmd("%s checkout -B %s %s" % (ud.basecmd, branchname, \ - ud.revisions[ud.names[0]]), d, workdir=destdir) + ud.revision), d, workdir=destdir) runfetchcmd("%s branch %s --set-upstream-to origin/%s" % (ud.basecmd, branchname, \ branchname), d, workdir=destdir) else: - runfetchcmd("%s checkout %s" % (ud.basecmd, ud.revisions[ud.names[0]]), d, workdir=destdir) + runfetchcmd("%s checkout %s" % (ud.basecmd, ud.revision), d, workdir=destdir) return True @@ -701,22 +812,29 @@ class Git(FetchMethod): clonedir = os.path.realpath(ud.localpath) to_remove.append(clonedir) + # Remove shallow mirror tarball + if ud.shallow: + to_remove.append(ud.fullshallow) + to_remove.append(ud.fullshallow + ".done") + for r in to_remove: - if os.path.exists(r): + if os.path.exists(r) or os.path.islink(r): bb.note('Removing %s' % r) bb.utils.remove(r, True) def supports_srcrev(self): return True - def _contains_ref(self, ud, d, name, wd): + def _contains_ref(self, ud, d, name, wd, tag=False): cmd = "" + git_ref_name = 'refs/tags/%s' % ud.parm['tag'] if tag else ud.revision + if ud.nobranch: cmd = "%s log --pretty=oneline -n 1 %s -- 2> /dev/null | wc -l" % ( - ud.basecmd, ud.revisions[name]) + ud.basecmd, git_ref_name) else: cmd = "%s branch --contains %s --list %s 2> /dev/null | wc -l" % ( - ud.basecmd, ud.revisions[name], ud.branches[name]) + ud.basecmd, git_ref_name, ud.branch) try: output = runfetchcmd(cmd, d, quiet=True, workdir=wd) except bb.fetch2.FetchError: @@ -725,19 +843,21 @@ class Git(FetchMethod): raise bb.fetch2.FetchError("The command '%s' gave output with more then 1 line unexpectedly, output: '%s'" % (cmd, output)) return output.split()[0] != "0" - def _lfs_objects_downloaded(self, ud, d, name, wd): + def _lfs_objects_downloaded(self, ud, d, wd): """ Verifies whether the LFS objects for requested revisions have already been downloaded """ # Bail out early if this repository doesn't use LFS - if not self._need_lfs(ud) or not self._contains_lfs(ud, d, wd): + if not self._contains_lfs(ud, d, wd): return True + self._ensure_git_lfs(d, ud) + # The Git LFS specification specifies ([1]) the LFS folder layout so it should be safe to check for file # existence. # [1] https://github.com/git-lfs/git-lfs/blob/main/docs/spec.md#intercepting-git cmd = "%s lfs ls-files -l %s" \ - % (ud.basecmd, ud.revisions[name]) + % (ud.basecmd, ud.revision) output = runfetchcmd(cmd, d, quiet=True, workdir=wd).rstrip() # Do not do any further matching if no objects are managed by LFS if not output: @@ -761,18 +881,8 @@ class Git(FetchMethod): """ Check if the repository has 'lfs' (large file) content """ - - if ud.nobranch: - # If no branch is specified, use the current git commit - refname = self._build_revision(ud, d, ud.names[0]) - elif wd == ud.clonedir: - # The bare clonedir doesn't use the remote names; it has the branch immediately. - refname = ud.branches[ud.names[0]] - else: - refname = "origin/%s" % ud.branches[ud.names[0]] - - cmd = "%s grep lfs %s:.gitattributes | wc -l" % ( - ud.basecmd, refname) + cmd = "%s grep '^[^#].*lfs' %s:.gitattributes | wc -l" % ( + ud.basecmd, ud.revision) try: output = runfetchcmd(cmd, d, quiet=True, workdir=wd) @@ -782,12 +892,14 @@ class Git(FetchMethod): pass return False - def _find_git_lfs(self, d): + def _ensure_git_lfs(self, d, ud): """ - Return True if git-lfs can be found, False otherwise. + Ensures that git-lfs is available, raising a FetchError if it isn't. """ - import shutil - return shutil.which("git-lfs", path=d.getVar('PATH')) is not None + if shutil.which("git-lfs", path=d.getVar('PATH')) is None: + raise bb.fetch2.FetchError( + "Repository %s has LFS content, install git-lfs on host to download (or set lfs=0 " + "to ignore it)" % self._get_repo_url(ud)) def _get_repo_url(self, ud): """ @@ -795,21 +907,21 @@ class Git(FetchMethod): """ # Note that we do not support passwords directly in the git urls. There are several # reasons. SRC_URI can be written out to things like buildhistory and people don't - # want to leak passwords like that. Its also all too easy to share metadata without + # want to leak passwords like that. Its also all too easy to share metadata without # removing the password. ssh keys, ~/.netrc and ~/.ssh/config files can be used as # alternatives so we will not take patches adding password support here. if ud.user: username = ud.user + '@' else: username = "" - return "%s://%s%s%s" % (ud.proto, username, ud.host, ud.path) + return "%s://%s%s%s" % (ud.proto, username, ud.host, urllib.parse.quote(ud.path)) def _revision_key(self, ud, d, name): """ Return a unique key for the url """ # Collapse adjacent slashes - return "git:" + ud.host + slash_re.sub(".", ud.path) + ud.unresolvedrev[name] + return "git:" + ud.host + slash_re.sub(".", ud.path) + ud.unresolvedrev def _lsremote(self, ud, d, search): """ @@ -842,28 +954,28 @@ class Git(FetchMethod): Compute the HEAD revision for the url """ if not d.getVar("__BBSRCREV_SEEN"): - raise bb.fetch2.FetchError("Recipe uses a floating tag/branch '%s' for repo '%s' without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE)." % (ud.unresolvedrev[name], ud.host+ud.path)) + raise bb.fetch2.FetchError("Recipe uses a floating tag/branch '%s' for repo '%s' without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE)." % (ud.unresolvedrev, ud.host+ud.path)) # Ensure we mark as not cached bb.fetch2.mark_recipe_nocache(d) output = self._lsremote(ud, d, "") # Tags of the form ^{} may not work, need to fallback to other form - if ud.unresolvedrev[name][:5] == "refs/" or ud.usehead: - head = ud.unresolvedrev[name] - tag = ud.unresolvedrev[name] + if ud.unresolvedrev[:5] == "refs/" or ud.usehead: + head = ud.unresolvedrev + tag = ud.unresolvedrev else: - head = "refs/heads/%s" % ud.unresolvedrev[name] - tag = "refs/tags/%s" % ud.unresolvedrev[name] + head = "refs/heads/%s" % ud.unresolvedrev + tag = "refs/tags/%s" % ud.unresolvedrev for s in [head, tag + "^{}", tag]: for l in output.strip().split('\n'): sha1, ref = l.split() if s == ref: return sha1 raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \ - (ud.unresolvedrev[name], ud.host+ud.path)) + (ud.unresolvedrev, ud.host+ud.path)) - def latest_versionstring(self, ud, d): + def latest_versionstring(self, ud, d, filter_regex=None): """ Compute the latest release name like "x.y.x" in "x.y.x+gitHASH" by searching through the tags output of ls-remote, comparing @@ -903,6 +1015,10 @@ class Git(FetchMethod): pver = m.group('pver').replace("_", ".") + if filter_regex: + if not re.match(filter_regex, pver): + continue + if verstring and bb.utils.vercmp(("0", pver, ""), ("0", verstring, "")) < 0: continue @@ -912,23 +1028,22 @@ class Git(FetchMethod): return pupver def _build_revision(self, ud, d, name): - return ud.revisions[name] + return ud.revision def gitpkgv_revision(self, ud, d, name): """ Return a sortable revision number by counting commits in the history Based on gitpkgv.bblass in meta-openembedded """ - rev = self._build_revision(ud, d, name) + rev = ud.revision localpath = ud.localpath rev_file = os.path.join(localpath, "oe-gitpkgv_" + rev) if not os.path.exists(localpath): commits = None else: if not os.path.exists(rev_file) or not os.path.getsize(rev_file): - from pipes import quote commits = bb.fetch2.runfetchcmd( - "git rev-list %s -- | wc -l" % quote(rev), + "git rev-list %s -- | wc -l" % shlex.quote(rev), d, quiet=True).strip().lstrip('0') if commits: open(rev_file, "w").write("%d\n" % int(commits)) diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py index f7f3af72..8b4ec620 100644 --- a/bitbake/lib/bb/fetch2/gitsm.py +++ b/bitbake/lib/bb/fetch2/gitsm.py @@ -62,36 +62,38 @@ class GitSM(Git): return modules # Collect the defined submodules, and their attributes - for name in ud.names: + try: + gitmodules = runfetchcmd("%s show %s:.gitmodules" % (ud.basecmd, ud.revision), d, quiet=True, workdir=workdir) + except: + # No submodules to update + gitmodules = "" + + for m, md in parse_gitmodules(gitmodules).items(): try: - gitmodules = runfetchcmd("%s show %s:.gitmodules" % (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir) + module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, ud.revision, md['path']), d, quiet=True, workdir=workdir) except: - # No submodules to update + # If the command fails, we don't have a valid file to check. If it doesn't + # fail -- it still might be a failure, see next check... + module_hash = "" + + if not module_hash: + logger.debug("submodule %s is defined, but is not initialized in the repository. Skipping", m) continue - for m, md in parse_gitmodules(gitmodules).items(): - try: - module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, ud.revisions[name], md['path']), d, quiet=True, workdir=workdir) - except: - # If the command fails, we don't have a valid file to check. If it doesn't - # fail -- it still might be a failure, see next check... - module_hash = "" - - if not module_hash: - logger.debug("submodule %s is defined, but is not initialized in the repository. Skipping", m) - continue - - submodules.append(m) - paths[m] = md['path'] - revision[m] = ud.revisions[name] - uris[m] = md['url'] - subrevision[m] = module_hash.split()[2] - - # Convert relative to absolute uri based on parent uri - if uris[m].startswith('..') or uris[m].startswith('./'): + submodules.append(m) + paths[m] = md['path'] + revision[m] = ud.revision + uris[m] = md['url'] + subrevision[m] = module_hash.split()[2] + + # Convert relative to absolute uri based on parent uri + if uris[m].startswith('..') or uris[m].startswith('./'): + if hasattr(ud, "origud"): + newud = copy.copy(ud.origud) + else: newud = copy.copy(ud) - newud.path = os.path.normpath(os.path.join(newud.path, uris[m])) - uris[m] = Git._get_repo_url(self, newud) + newud.path = os.path.normpath(os.path.join(newud.path, uris[m])) + uris[m] = Git._get_repo_url(self, newud) for module in submodules: # Translate the module url into a SRC_URI @@ -123,7 +125,7 @@ class GitSM(Git): url += ";name=%s" % module url += ";subpath=%s" % module url += ";nobranch=1" - url += ";lfs=%s" % self._need_lfs(ud) + url += ";lfs=%s" % ("1" if self._need_lfs(ud) else "0") # Note that adding "user=" here to give credentials to the # submodule is not supported. Since using SRC_URI to give git:// # URL a password is not supported, one have to use one of the @@ -147,6 +149,22 @@ class GitSM(Git): return submodules != [] + def call_process_submodules(self, ud, d, extra_check, subfunc): + # If we're using a shallow mirror tarball it needs to be + # unpacked temporarily so that we can examine the .gitmodules file + # Unpack even when ud.clonedir is not available, + # which may occur during a fast shallow clone + unpack = extra_check or not os.path.exists(ud.clonedir) + if ud.shallow and os.path.exists(ud.fullshallow) and unpack: + tmpdir = tempfile.mkdtemp(dir=d.getVar("DL_DIR")) + try: + runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=tmpdir) + self.process_submodules(ud, tmpdir, subfunc, d) + finally: + shutil.rmtree(tmpdir) + else: + self.process_submodules(ud, ud.clonedir, subfunc, d) + def need_update(self, ud, d): if Git.need_update(self, ud, d): return True @@ -164,15 +182,7 @@ class GitSM(Git): logger.error('gitsm: submodule update check failed: %s %s' % (type(e).__name__, str(e))) need_update_result = True - # If we're using a shallow mirror tarball it needs to be unpacked - # temporarily so that we can examine the .gitmodules file - if ud.shallow and os.path.exists(ud.fullshallow) and not os.path.exists(ud.clonedir): - tmpdir = tempfile.mkdtemp(dir=d.getVar("DL_DIR")) - runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=tmpdir) - self.process_submodules(ud, tmpdir, need_update_submodule, d) - shutil.rmtree(tmpdir) - else: - self.process_submodules(ud, ud.clonedir, need_update_submodule, d) + self.call_process_submodules(ud, d, not os.path.exists(ud.clonedir), need_update_submodule) if need_update_list: logger.debug('gitsm: Submodules requiring update: %s' % (' '.join(need_update_list))) @@ -195,16 +205,7 @@ class GitSM(Git): raise Git.download(self, ud, d) - - # If we're using a shallow mirror tarball it needs to be unpacked - # temporarily so that we can examine the .gitmodules file - if ud.shallow and os.path.exists(ud.fullshallow) and self.need_update(ud, d): - tmpdir = tempfile.mkdtemp(dir=d.getVar("DL_DIR")) - runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=tmpdir) - self.process_submodules(ud, tmpdir, download_submodule, d) - shutil.rmtree(tmpdir) - else: - self.process_submodules(ud, ud.clonedir, download_submodule, d) + self.call_process_submodules(ud, d, self.need_update(ud, d), download_submodule) def unpack(self, ud, destdir, d): def unpack_submodules(ud, url, module, modpath, workdir, d): @@ -247,15 +248,27 @@ class GitSM(Git): ret = self.process_submodules(ud, ud.destdir, unpack_submodules, d) if not ud.bareclone and ret: - # All submodules should already be downloaded and configured in the tree. This simply - # sets up the configuration and checks out the files. The main project config should - # remain unmodified, and no download from the internet should occur. As such, lfs smudge - # should also be skipped as these files were already smudged in the fetch stage if lfs - # was enabled. - runfetchcmd("GIT_LFS_SKIP_SMUDGE=1 %s submodule update --recursive --no-fetch" % (ud.basecmd), d, quiet=True, workdir=ud.destdir) + cmdprefix = "" + # Avoid LFS smudging (replacing the LFS pointers with the actual content) when LFS shouldn't be used but git-lfs is installed. + if not self._need_lfs(ud): + cmdprefix = "GIT_LFS_SKIP_SMUDGE=1 " + runfetchcmd("%s%s submodule update --recursive --no-fetch" % (cmdprefix, ud.basecmd), d, quiet=True, workdir=ud.destdir) + def clean(self, ud, d): + def clean_submodule(ud, url, module, modpath, workdir, d): + url += ";bareclone=1;nobranch=1" + try: + newfetch = Fetch([url], d, cache=False) + newfetch.clean() + except Exception as e: + logger.warning('gitsm: submodule clean failed: %s %s' % (type(e).__name__, str(e))) + + self.call_process_submodules(ud, d, True, clean_submodule) + + # Clean top git dir + Git.clean(self, ud, d) def implicit_urldata(self, ud, d): - import shutil, subprocess, tempfile + import subprocess urldata = [] def add_submodule(ud, url, module, modpath, workdir, d): @@ -263,14 +276,6 @@ class GitSM(Git): newfetch = Fetch([url], d, cache=False) urldata.extend(newfetch.expanded_urldata()) - # If we're using a shallow mirror tarball it needs to be unpacked - # temporarily so that we can examine the .gitmodules file - if ud.shallow and os.path.exists(ud.fullshallow) and ud.method.need_update(ud, d): - tmpdir = tempfile.mkdtemp(dir=d.getVar("DL_DIR")) - subprocess.check_call("tar -xzf %s" % ud.fullshallow, cwd=tmpdir, shell=True) - self.process_submodules(ud, tmpdir, add_submodule, d) - shutil.rmtree(tmpdir) - else: - self.process_submodules(ud, ud.clonedir, add_submodule, d) + self.call_process_submodules(ud, d, ud.method.need_update(ud, d), add_submodule) return urldata diff --git a/bitbake/lib/bb/fetch2/gomod.py b/bitbake/lib/bb/fetch2/gomod.py new file mode 100644 index 00000000..53c1d8d1 --- /dev/null +++ b/bitbake/lib/bb/fetch2/gomod.py @@ -0,0 +1,273 @@ +""" +BitBake 'Fetch' implementation for Go modules + +The gomod/gomodgit fetchers are used to download Go modules to the module cache +from a module proxy or directly from a version control repository. + +Example SRC_URI: + +SRC_URI += "gomod://golang.org/x/net;version=v0.9.0;sha256sum=..." +SRC_URI += "gomodgit://golang.org/x/net;version=v0.9.0;repo=go.googlesource.com/net;srcrev=..." + +Required SRC_URI parameters: + +- version + The version of the module. + +Optional SRC_URI parameters: + +- mod + Fetch and unpack the go.mod file only instead of the complete module. + The go command may need to download go.mod files for many different modules + when computing the build list, and go.mod files are much smaller than + module zip files. + The default is "0", set mod=1 for the go.mod file only. + +- sha256sum + The checksum of the module zip file, or the go.mod file in case of fetching + only the go.mod file. Alternatively, set the SRC_URI varible flag for + "module@version.sha256sum". + +- protocol + The method used when fetching directly from a version control repository. + The default is "https" for git. + +- repo + The URL when fetching directly from a version control repository. Required + when the URL is different from the module path. + +- srcrev + The revision identifier used when fetching directly from a version control + repository. Alternatively, set the SRCREV varible for "module@version". + +- subdir + The module subdirectory when fetching directly from a version control + repository. Required when the module is not located in the root of the + repository. + +Related variables: + +- GO_MOD_PROXY + The module proxy used by the fetcher. + +- GO_MOD_CACHE_DIR + The directory where the module cache is located. + This must match the exported GOMODCACHE variable for the go command to find + the downloaded modules. + +See the Go modules reference, https://go.dev/ref/mod, for more information +about the module cache, module proxies and version control systems. +""" + +import hashlib +import os +import re +import shutil +import subprocess +import zipfile + +import bb +from bb.fetch2 import FetchError +from bb.fetch2 import MissingParameterError +from bb.fetch2 import runfetchcmd +from bb.fetch2 import subprocess_setup +from bb.fetch2.git import Git +from bb.fetch2.wget import Wget + + +def escape(path): + """Escape capital letters using exclamation points.""" + return re.sub(r'([A-Z])', lambda m: '!' + m.group(1).lower(), path) + + +class GoMod(Wget): + """Class to fetch Go modules from a Go module proxy via wget""" + + def supports(self, ud, d): + """Check to see if a given URL is for this fetcher.""" + return ud.type == 'gomod' + + def urldata_init(self, ud, d): + """Set up to download the module from the module proxy. + + Set up to download the module zip file to the module cache directory + and unpack the go.mod file (unless downloading only the go.mod file): + + cache/download//@v/.zip: The module zip file. + cache/download//@v/.mod: The go.mod file. + """ + + proxy = d.getVar('GO_MOD_PROXY') or 'proxy.golang.org' + moddir = d.getVar('GO_MOD_CACHE_DIR') or 'pkg/mod' + + if 'version' not in ud.parm: + raise MissingParameterError('version', ud.url) + + module = ud.host + if ud.path != '/': + module += ud.path + ud.parm['module'] = module + version = ud.parm['version'] + + # Set URL and filename for wget download + if ud.parm.get('mod', '0') == '1': + ext = '.mod' + else: + ext = '.zip' + path = escape(f"{module}/@v/{version}{ext}") + ud.url = bb.fetch2.encodeurl( + ('https', proxy, '/' + path, None, None, None)) + ud.parm['downloadfilename'] = f"{module.replace('/', '.')}@{version}{ext}" + + # Set name for checksum verification + ud.parm['name'] = f"{module}@{version}" + + # Set path for unpack + ud.parm['unpackpath'] = os.path.join(moddir, 'cache/download', path) + + super().urldata_init(ud, d) + + def unpack(self, ud, rootdir, d): + """Unpack the module in the module cache.""" + + # Unpack the module zip file or go.mod file + unpackpath = os.path.join(rootdir, ud.parm['unpackpath']) + unpackdir = os.path.dirname(unpackpath) + bb.utils.mkdirhier(unpackdir) + ud.unpack_tracer.unpack("file-copy", unpackdir) + cmd = f"cp {ud.localpath} {unpackpath}" + path = d.getVar('PATH') + if path: + cmd = f"PATH={path} {cmd}" + name = os.path.basename(unpackpath) + bb.note(f"Unpacking {name} to {unpackdir}/") + subprocess.check_call(cmd, shell=True, preexec_fn=subprocess_setup) + + if name.endswith('.zip'): + # Unpack the go.mod file from the zip file + module = ud.parm['module'] + name = name.rsplit('.', 1)[0] + '.mod' + bb.note(f"Unpacking {name} to {unpackdir}/") + with zipfile.ZipFile(ud.localpath) as zf: + with open(os.path.join(unpackdir, name), mode='wb') as mf: + try: + f = module + '@' + ud.parm['version'] + '/go.mod' + shutil.copyfileobj(zf.open(f), mf) + except KeyError: + # If the module does not have a go.mod file, synthesize + # one containing only a module statement. + mf.write(f'module {module}\n'.encode()) + + +class GoModGit(Git): + """Class to fetch Go modules directly from a git repository""" + + def supports(self, ud, d): + """Check to see if a given URL is for this fetcher.""" + return ud.type == 'gomodgit' + + def urldata_init(self, ud, d): + """Set up to download the module from the git repository. + + Set up to download the git repository to the module cache directory and + unpack the module zip file and the go.mod file: + + cache/vcs/: The bare git repository. + cache/download//@v/.zip: The module zip file. + cache/download//@v/.mod: The go.mod file. + """ + + moddir = d.getVar('GO_MOD_CACHE_DIR') or 'pkg/mod' + + if 'version' not in ud.parm: + raise MissingParameterError('version', ud.url) + + module = ud.host + if ud.path != '/': + module += ud.path + ud.parm['module'] = module + + # Set host, path and srcrev for git download + if 'repo' in ud.parm: + repo = ud.parm['repo'] + idx = repo.find('/') + if idx != -1: + ud.host = repo[:idx] + ud.path = repo[idx:] + else: + ud.host = repo + ud.path = '' + if 'protocol' not in ud.parm: + ud.parm['protocol'] = 'https' + ud.name = f"{module}@{ud.parm['version']}" + srcrev = d.getVar('SRCREV_' + ud.name) + if srcrev: + if 'srcrev' not in ud.parm: + ud.parm['srcrev'] = srcrev + else: + if 'srcrev' in ud.parm: + d.setVar('SRCREV_' + ud.name, ud.parm['srcrev']) + if 'branch' not in ud.parm: + ud.parm['nobranch'] = '1' + + # Set subpath, subdir and bareclone for git unpack + if 'subdir' in ud.parm: + ud.parm['subpath'] = ud.parm['subdir'] + key = f"git3:{ud.parm['protocol']}://{ud.host}{ud.path}".encode() + ud.parm['key'] = key + ud.parm['subdir'] = os.path.join(moddir, 'cache/vcs', + hashlib.sha256(key).hexdigest()) + ud.parm['bareclone'] = '1' + + super().urldata_init(ud, d) + + def unpack(self, ud, rootdir, d): + """Unpack the module in the module cache.""" + + # Unpack the bare git repository + super().unpack(ud, rootdir, d) + + moddir = d.getVar('GO_MOD_CACHE_DIR') or 'pkg/mod' + + # Create the info file + module = ud.parm['module'] + repodir = os.path.join(rootdir, ud.parm['subdir']) + with open(repodir + '.info', 'wb') as f: + f.write(ud.parm['key']) + + # Unpack the go.mod file from the repository + unpackdir = os.path.join(rootdir, moddir, 'cache/download', + escape(module), '@v') + bb.utils.mkdirhier(unpackdir) + srcrev = ud.parm['srcrev'] + version = ud.parm['version'] + escaped_version = escape(version) + cmd = f"git ls-tree -r --name-only '{srcrev}'" + if 'subpath' in ud.parm: + cmd += f" '{ud.parm['subpath']}'" + files = runfetchcmd(cmd, d, workdir=repodir).split() + name = escaped_version + '.mod' + bb.note(f"Unpacking {name} to {unpackdir}/") + with open(os.path.join(unpackdir, name), mode='wb') as mf: + f = 'go.mod' + if 'subpath' in ud.parm: + f = os.path.join(ud.parm['subpath'], f) + if f in files: + cmd = ['git', 'cat-file', 'blob', srcrev + ':' + f] + subprocess.check_call(cmd, stdout=mf, cwd=repodir, + preexec_fn=subprocess_setup) + else: + # If the module does not have a go.mod file, synthesize one + # containing only a module statement. + mf.write(f'module {module}\n'.encode()) + + # Synthesize the module zip file from the repository + name = escaped_version + '.zip' + bb.note(f"Unpacking {name} to {unpackdir}/") + with zipfile.ZipFile(os.path.join(unpackdir, name), mode='w') as zf: + prefix = module + '@' + version + '/' + for f in files: + cmd = ['git', 'cat-file', 'blob', srcrev + ':' + f] + data = subprocess.check_output(cmd, cwd=repodir, + preexec_fn=subprocess_setup) + zf.writestr(prefix + f, data) diff --git a/bitbake/lib/bb/fetch2/local.py b/bitbake/lib/bb/fetch2/local.py index 7d766811..fda56a56 100644 --- a/bitbake/lib/bb/fetch2/local.py +++ b/bitbake/lib/bb/fetch2/local.py @@ -29,11 +29,10 @@ class Local(FetchMethod): def urldata_init(self, ud, d): # We don't set localfile as for this fetcher the file is already local! - ud.decodedurl = urllib.parse.unquote(ud.url.split("://")[1].split(";")[0]) - ud.basename = os.path.basename(ud.decodedurl) - ud.basepath = ud.decodedurl + ud.basename = os.path.basename(ud.path) + ud.basepath = ud.path ud.needdonestamp = False - if "*" in ud.decodedurl: + if "*" in ud.path: raise bb.fetch2.ParameterError("file:// urls using globbing are no longer supported. Please place the files in a directory and reference that instead.", ud.url) return @@ -48,7 +47,7 @@ class Local(FetchMethod): Return the local filename of a given url assuming a successful fetch. """ searched = [] - path = urldata.decodedurl + path = urldata.path newpath = path if path[0] == "/": logger.debug2("Using absolute %s" % (path)) diff --git a/bitbake/lib/bb/fetch2/npm.py b/bitbake/lib/bb/fetch2/npm.py index 15f3f19b..ed9ed167 100644 --- a/bitbake/lib/bb/fetch2/npm.py +++ b/bitbake/lib/bb/fetch2/npm.py @@ -42,11 +42,12 @@ from bb.utils import is_semver def npm_package(package): """Convert the npm package name to remove unsupported character""" - # Scoped package names (with the @) use the same naming convention - # as the 'npm pack' command. + # For scoped package names ('@user/package') the '/' is replaced by a '-'. + # This is similar to what 'npm pack' does, but 'npm pack' also strips the + # leading '@', which can lead to ambiguous package names. name = re.sub("/", "-", package) name = name.lower() - name = re.sub(r"[^\-a-z0-9]", "", name) + name = re.sub(r"[^\-a-z0-9@]", "", name) name = name.strip("-") return name @@ -90,6 +91,12 @@ class NpmEnvironment(object): self.d = d self.user_config = tempfile.NamedTemporaryFile(mode="w", buffering=1) + + hn = self._home_npmrc(d) + if hn is not None: + with open(hn, 'r') as hnf: + self.user_config.write(hnf.read()) + for key, value in configs: self.user_config.write("%s=%s\n" % (key, value)) @@ -102,6 +109,15 @@ class NpmEnvironment(object): if self.user_config: self.user_config.close() + def _home_npmrc(self, d): + """Function to return user's HOME .npmrc file (or None if it doesn't exist)""" + home_npmrc_file = os.path.join(os.environ.get("HOME"), ".npmrc") + if d.getVar("BB_USE_HOME_NPMRC") == "1" and os.path.exists(home_npmrc_file): + bb.warn(f"BB_USE_HOME_NPMRC flag set and valid .npmrc detected - "\ + f"npm fetcher will use {home_npmrc_file}") + return home_npmrc_file + return None + def run(self, cmd, args=None, configs=None, workdir=None): """Run npm command in a controlled environment""" with tempfile.TemporaryDirectory() as tmpdir: @@ -134,7 +150,11 @@ class Npm(FetchMethod): def supports(self, ud, d): """Check if a given url can be fetched with npm""" - return ud.type in ["npm"] + #return ud.type in ["npm"] + if ud.type in ["npm"]: + from bb.parse import SkipRecipe + raise SkipRecipe("The npm fetcher has been disabled due to security issues and there is no maintainer to address them") + return False def urldata_init(self, ud, d): """Init npm specific variables within url data""" @@ -165,7 +185,7 @@ class Npm(FetchMethod): # Using the 'downloadfilename' parameter as local filename # or the npm package name. if "downloadfilename" in ud.parm: - ud.localfile = npm_localfile(d.expand(ud.parm["downloadfilename"])) + ud.localfile = npm_localfile(ud.parm["downloadfilename"]) else: ud.localfile = npm_localfile(ud.package, ud.version) diff --git a/bitbake/lib/bb/fetch2/npmsw.py b/bitbake/lib/bb/fetch2/npmsw.py index ff5f8dc7..85f4482a 100644 --- a/bitbake/lib/bb/fetch2/npmsw.py +++ b/bitbake/lib/bb/fetch2/npmsw.py @@ -37,45 +37,37 @@ def foreach_dependencies(shrinkwrap, callback=None, dev=False): """ Run a callback for each dependencies of a shrinkwrap file. The callback is using the format: - callback(name, params, deptree) + callback(name, data, location) with: name = the package name (string) - params = the package parameters (dictionary) - destdir = the destination of the package (string) + data = the package data (dictionary) + location = the location of the package (string) """ - # For handling old style dependencies entries in shinkwrap files - def _walk_deps(deps, deptree): - for name in deps: - subtree = [*deptree, name] - _walk_deps(deps[name].get("dependencies", {}), subtree) - if callback is not None: - if deps[name].get("dev", False) and not dev: - continue - elif deps[name].get("bundled", False): - continue - destsubdirs = [os.path.join("node_modules", dep) for dep in subtree] - destsuffix = os.path.join(*destsubdirs) - callback(name, deps[name], destsuffix) - - # packages entry means new style shrinkwrap file, else use dependencies - packages = shrinkwrap.get("packages", None) - if packages is not None: - for package in packages: - if package != "": - name = package.split('node_modules/')[-1] - package_infos = packages.get(package, {}) - if dev == False and package_infos.get("dev", False): - continue - callback(name, package_infos, package) - else: - _walk_deps(shrinkwrap.get("dependencies", {}), []) + packages = shrinkwrap.get("packages") + if not packages: + raise FetchError("Invalid shrinkwrap file format") + + for location, data in packages.items(): + # Skip empty main and local link target packages + if not location.startswith('node_modules/'): + continue + elif not dev and data.get("dev", False): + continue + elif data.get("inBundle", False): + continue + name = location.split('node_modules/')[-1] + callback(name, data, location) class NpmShrinkWrap(FetchMethod): """Class to fetch all package from a shrinkwrap file""" def supports(self, ud, d): """Check if a given url can be fetched with npmsw""" - return ud.type in ["npmsw"] + #return ud.type in ["npmsw"] + if ud.type in ["npmsw"]: + from bb.parse import SkipRecipe + raise SkipRecipe("The npmsw fetcher has been disabled due to security issues and there is no maintainer to address them") + return False def urldata_init(self, ud, d): """Init npmsw specific variables within url data""" @@ -95,12 +87,18 @@ class NpmShrinkWrap(FetchMethod): extrapaths = [] unpack = True - integrity = params.get("integrity", None) - resolved = params.get("resolved", None) - version = params.get("version", None) + integrity = params.get("integrity") + resolved = params.get("resolved") + version = params.get("version") + link = params.get("link", False) + + # Handle link sources + if link: + localpath = resolved + unpack = False # Handle registry sources - if is_semver(version) and integrity: + elif version and is_semver(version) and integrity: # Handle duplicate dependencies without url if not resolved: return @@ -128,10 +126,10 @@ class NpmShrinkWrap(FetchMethod): extrapaths.append(resolvefile) # Handle http tarball sources - elif version.startswith("http") and integrity: - localfile = npm_localfile(os.path.basename(version)) + elif resolved.startswith("http") and integrity: + localfile = npm_localfile(os.path.basename(resolved)) - uri = URI(version) + uri = URI(resolved) uri.params["downloadfilename"] = localfile checksum_name, checksum_expected = npm_integrity(integrity) @@ -141,28 +139,12 @@ class NpmShrinkWrap(FetchMethod): localpath = os.path.join(d.getVar("DL_DIR"), localfile) - # Handle local tarball and link sources - elif version.startswith("file"): - localpath = version[5:] - if not version.endswith(".tgz"): - unpack = False + # Handle local tarball sources + elif resolved.startswith("file"): + localpath = resolved[5:] # Handle git sources - elif version.startswith(("git", "bitbucket","gist")) or ( - not version.endswith((".tgz", ".tar", ".tar.gz")) - and not version.startswith((".", "@", "/")) - and "/" in version - ): - if version.startswith("github:"): - version = "git+https://github.com/" + version[len("github:"):] - elif version.startswith("gist:"): - version = "git+https://gist.github.com/" + version[len("gist:"):] - elif version.startswith("bitbucket:"): - version = "git+https://bitbucket.org/" + version[len("bitbucket:"):] - elif version.startswith("gitlab:"): - version = "git+https://gitlab.com/" + version[len("gitlab:"):] - elif not version.startswith(("git+","git:")): - version = "git+https://github.com/" + version + elif resolved.startswith("git"): regex = re.compile(r""" ^ git\+ @@ -174,16 +156,16 @@ class NpmShrinkWrap(FetchMethod): $ """, re.VERBOSE) - match = regex.match(version) - + match = regex.match(resolved) if not match: - raise ParameterError("Invalid git url: %s" % version, ud.url) + raise ParameterError("Invalid git url: %s" % resolved, ud.url) groups = match.groupdict() uri = URI("git://" + str(groups["url"])) uri.params["protocol"] = str(groups["protocol"]) uri.params["rev"] = str(groups["rev"]) + uri.params["nobranch"] = "1" uri.params["destsuffix"] = destsuffix url = str(uri) @@ -268,7 +250,7 @@ class NpmShrinkWrap(FetchMethod): def unpack(self, ud, rootdir, d): """Unpack the downloaded dependencies""" - destdir = d.getVar("S") + destdir = rootdir destsuffix = ud.parm.get("destsuffix") if destsuffix: destdir = os.path.join(rootdir, destsuffix) diff --git a/bitbake/lib/bb/fetch2/osc.py b/bitbake/lib/bb/fetch2/osc.py deleted file mode 100644 index 495ac8a3..00000000 --- a/bitbake/lib/bb/fetch2/osc.py +++ /dev/null @@ -1,165 +0,0 @@ -# -# Copyright BitBake Contributors -# -# SPDX-License-Identifier: GPL-2.0-only -# -""" -Bitbake "Fetch" implementation for osc (Opensuse build service client). -Based on the svn "Fetch" implementation. - -""" - -import logging -import os -import re -import bb -from bb.fetch2 import FetchMethod -from bb.fetch2 import FetchError -from bb.fetch2 import MissingParameterError -from bb.fetch2 import runfetchcmd - -logger = logging.getLogger(__name__) - -class Osc(FetchMethod): - """Class to fetch a module or modules from Opensuse build server - repositories.""" - - def supports(self, ud, d): - """ - Check to see if a given url can be fetched with osc. - """ - return ud.type in ['osc'] - - def urldata_init(self, ud, d): - if not "module" in ud.parm: - raise MissingParameterError('module', ud.url) - - ud.module = ud.parm["module"] - - # Create paths to osc checkouts - oscdir = d.getVar("OSCDIR") or (d.getVar("DL_DIR") + "/osc") - relpath = self._strip_leading_slashes(ud.path) - ud.oscdir = oscdir - ud.pkgdir = os.path.join(oscdir, ud.host) - ud.moddir = os.path.join(ud.pkgdir, relpath, ud.module) - - if 'rev' in ud.parm: - ud.revision = ud.parm['rev'] - else: - pv = d.getVar("PV", False) - rev = bb.fetch2.srcrev_internal_helper(ud, d, '') - if rev: - ud.revision = rev - else: - ud.revision = "" - - ud.localfile = d.expand('%s_%s_%s.tar.gz' % (ud.module.replace('/', '.'), relpath.replace('/', '.'), ud.revision)) - - def _buildosccommand(self, ud, d, command): - """ - Build up an ocs commandline based on ud - command is "fetch", "update", "info" - """ - - basecmd = d.getVar("FETCHCMD_osc") or "/usr/bin/env osc" - - proto = ud.parm.get('protocol', 'https') - - options = [] - - config = "-c %s" % self.generate_config(ud, d) - - if getattr(ud, 'revision', ''): - options.append("-r %s" % ud.revision) - - coroot = self._strip_leading_slashes(ud.path) - - if command == "fetch": - osccmd = "%s %s -A %s://%s co %s/%s %s" % (basecmd, config, proto, ud.host, coroot, ud.module, " ".join(options)) - elif command == "update": - osccmd = "%s %s -A %s://%s up %s" % (basecmd, config, proto, ud.host, " ".join(options)) - elif command == "api_source": - osccmd = "%s %s -A %s://%s api source/%s/%s" % (basecmd, config, proto, ud.host, coroot, ud.module) - else: - raise FetchError("Invalid osc command %s" % command, ud.url) - - return osccmd - - def _latest_revision(self, ud, d, name): - """ - Fetch latest revision for the given package - """ - api_source_cmd = self._buildosccommand(ud, d, "api_source") - - output = runfetchcmd(api_source_cmd, d) - match = re.match(r'', output) - if match is None: - raise FetchError("Unable to parse osc response", ud.url) - return match.groups()[0] - - def _revision_key(self, ud, d, name): - """ - Return a unique key for the url - """ - # Collapse adjacent slashes - slash_re = re.compile(r"/+") - rev = getattr(ud, 'revision', "latest") - return "osc:%s%s.%s.%s" % (ud.host, slash_re.sub(".", ud.path), name, rev) - - def download(self, ud, d): - """ - Fetch url - """ - - logger.debug2("Fetch: checking for module directory '" + ud.moddir + "'") - - if os.access(ud.moddir, os.R_OK): - oscupdatecmd = self._buildosccommand(ud, d, "update") - logger.info("Update "+ ud.url) - # update sources there - logger.debug("Running %s", oscupdatecmd) - bb.fetch2.check_network_access(d, oscupdatecmd, ud.url) - runfetchcmd(oscupdatecmd, d, workdir=ud.moddir) - else: - oscfetchcmd = self._buildosccommand(ud, d, "fetch") - logger.info("Fetch " + ud.url) - # check out sources there - bb.utils.mkdirhier(ud.pkgdir) - logger.debug("Running %s", oscfetchcmd) - bb.fetch2.check_network_access(d, oscfetchcmd, ud.url) - runfetchcmd(oscfetchcmd, d, workdir=ud.pkgdir) - - # tar them up to a defined filename - runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d, - cleanup=[ud.localpath], workdir=os.path.join(ud.pkgdir + ud.path)) - - def supports_srcrev(self): - return False - - def generate_config(self, ud, d): - """ - Generate a .oscrc to be used for this run. - """ - - config_path = os.path.join(ud.oscdir, "oscrc") - if not os.path.exists(ud.oscdir): - bb.utils.mkdirhier(ud.oscdir) - - if (os.path.exists(config_path)): - os.remove(config_path) - - f = open(config_path, 'w') - proto = ud.parm.get('protocol', 'https') - f.write("[general]\n") - f.write("apiurl = %s://%s\n" % (proto, ud.host)) - f.write("su-wrapper = su -c\n") - f.write("build-root = %s\n" % d.getVar('WORKDIR')) - f.write("urllist = %s\n" % d.getVar("OSCURLLIST")) - f.write("extra-pkgs = gzip\n") - f.write("\n") - f.write("[%s://%s]\n" % (proto, ud.host)) - f.write("user = %s\n" % ud.parm["user"]) - f.write("pass = %s\n" % ud.parm["pswd"]) - f.close() - - return config_path diff --git a/bitbake/lib/bb/fetch2/s3.py b/bitbake/lib/bb/fetch2/s3.py index 6b8ffd53..22c05381 100644 --- a/bitbake/lib/bb/fetch2/s3.py +++ b/bitbake/lib/bb/fetch2/s3.py @@ -77,7 +77,7 @@ class S3(FetchMethod): else: ud.basename = os.path.basename(ud.path) - ud.localfile = d.expand(urllib.parse.unquote(ud.basename)) + ud.localfile = ud.basename ud.basecmd = d.getVar("FETCHCMD_s3") or "/usr/bin/env aws s3" diff --git a/bitbake/lib/bb/fetch2/sftp.py b/bitbake/lib/bb/fetch2/sftp.py index 7884cce9..bee71a0d 100644 --- a/bitbake/lib/bb/fetch2/sftp.py +++ b/bitbake/lib/bb/fetch2/sftp.py @@ -77,7 +77,7 @@ class SFTP(FetchMethod): else: ud.basename = os.path.basename(ud.path) - ud.localfile = d.expand(urllib.parse.unquote(ud.basename)) + ud.localfile = ud.basename def download(self, ud, d): """Fetch urls""" diff --git a/bitbake/lib/bb/fetch2/ssh.py b/bitbake/lib/bb/fetch2/ssh.py index 0cbb2a6f..2a0f2cb4 100644 --- a/bitbake/lib/bb/fetch2/ssh.py +++ b/bitbake/lib/bb/fetch2/ssh.py @@ -73,8 +73,7 @@ class SSH(FetchMethod): path = m.group('path') path = urllib.parse.unquote(path) host = m.group('host') - urldata.localpath = os.path.join(d.getVar('DL_DIR'), - os.path.basename(os.path.normpath(path))) + urldata.localfile = os.path.basename(os.path.normpath(path)) def download(self, urldata, d): dldir = d.getVar('DL_DIR') diff --git a/bitbake/lib/bb/fetch2/svn.py b/bitbake/lib/bb/fetch2/svn.py index d40e4d29..a097ffb7 100644 --- a/bitbake/lib/bb/fetch2/svn.py +++ b/bitbake/lib/bb/fetch2/svn.py @@ -34,7 +34,7 @@ class Svn(FetchMethod): if not "module" in ud.parm: raise MissingParameterError('module', ud.url) - ud.basecmd = d.getVar("FETCHCMD_svn") or "/usr/bin/env svn --non-interactive --trust-server-cert" + ud.basecmd = d.getVar("FETCHCMD_svn") or "/usr/bin/env svn --non-interactive" ud.module = ud.parm["module"] @@ -210,3 +210,6 @@ class Svn(FetchMethod): def _build_revision(self, ud, d): return ud.revision + + def supports_checksum(self, urldata): + return False diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py index fbfa6938..58f1a86e 100644 --- a/bitbake/lib/bb/fetch2/wget.py +++ b/bitbake/lib/bb/fetch2/wget.py @@ -22,6 +22,7 @@ import bb.progress import socket import http.client import urllib.request, urllib.parse, urllib.error +import subprocess from bb.fetch2 import FetchMethod from bb.fetch2 import FetchError from bb.fetch2 import logger @@ -53,11 +54,6 @@ class WgetProgressHandler(bb.progress.LineFilterProgressHandler): class Wget(FetchMethod): """Class to fetch urls via 'wget'""" - # CDNs like CloudFlare may do a 'browser integrity test' which can fail - # with the standard wget/urllib User-Agent, so pretend to be a modern - # browser. - user_agent = "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" - def check_certs(self, d): """ Should certificates be checked? @@ -83,11 +79,11 @@ class Wget(FetchMethod): else: ud.basename = os.path.basename(ud.path) - ud.localfile = d.expand(urllib.parse.unquote(ud.basename)) + ud.localfile = ud.basename if not ud.localfile: - ud.localfile = d.expand(urllib.parse.unquote(ud.host + ud.path).replace("/", ".")) + ud.localfile = ud.host + ud.path.replace("/", ".") - self.basecmd = d.getVar("FETCHCMD_wget") or "/usr/bin/env wget -t 2 -T 30" + self.basecmd = d.getVar("FETCHCMD_wget") or "/usr/bin/env wget --tries=2 --timeout=100" if ud.type == 'ftp' or ud.type == 'ftps': self.basecmd += " --passive-ftp" @@ -101,16 +97,20 @@ class Wget(FetchMethod): logger.debug2("Fetching %s using command '%s'" % (ud.url, command)) bb.fetch2.check_network_access(d, command, ud.url) - runfetchcmd(command + ' --progress=dot -v', d, quiet, log=progresshandler, workdir=workdir) + runfetchcmd(command + ' --progress=dot --verbose', d, quiet, log=progresshandler, workdir=workdir) def download(self, ud, d): """Fetch urls""" fetchcmd = self.basecmd - localpath = os.path.join(d.getVar("DL_DIR"), ud.localfile) + ".tmp" + dldir = os.path.realpath(d.getVar("DL_DIR")) + # Where we ultimately want the file + finalpath = os.path.join(dldir, ud.localfile) + # A temp location while processing, keeping in mind max path lengths + localpath = finalpath[:250] + ".tmp" bb.utils.mkdirhier(os.path.dirname(localpath)) - fetchcmd += " -O %s" % shlex.quote(localpath) + fetchcmd += " --output-document=%s" % shlex.quote(localpath) if ud.user and ud.pswd: fetchcmd += " --auth-no-challenge" @@ -126,14 +126,18 @@ class Wget(FetchMethod): fetchcmd += " --user=%s --password=%s" % (ud.user, ud.pswd) uri = ud.url.split(";")[0] - if os.path.exists(ud.localpath): - # file exists, but we didnt complete it.. trying again.. - fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri) - else: - fetchcmd += d.expand(" -P ${DL_DIR} '%s'" % uri) - + fetchcmd += " --continue --directory-prefix=%s '%s'" % (dldir, uri) self._runwget(ud, d, fetchcmd, False) + # Sanity check since wget can pretend it succeed when it didn't + # Also, this used to happen if sourceforge sent us to the mirror page + if not os.path.exists(localpath): + raise FetchError("The fetch command returned success for url %s but %s doesn't exist?!" % (uri, localpath), uri) + + if os.path.getsize(localpath) == 0: + os.remove(localpath) + raise FetchError("The fetch of %s resulted in a zero size file?! Deleting and failing since this isn't right." % (uri), uri) + # Try and verify any checksum now, meaning if it isn't correct, we don't remove the # original file, which might be a race (imagine two recipes referencing the same # source, one with an incorrect checksum) @@ -141,16 +145,7 @@ class Wget(FetchMethod): # Remove the ".tmp" and move the file into position atomically # Our lock prevents multiple writers but mirroring code may grab incomplete files - os.rename(localpath, localpath[:-4]) - - # Sanity check since wget can pretend it succeed when it didn't - # Also, this used to happen if sourceforge sent us to the mirror page - if not os.path.exists(ud.localpath): - raise FetchError("The fetch command returned success for url %s but %s doesn't exist?!" % (uri, ud.localpath), uri) - - if os.path.getsize(ud.localpath) == 0: - os.remove(ud.localpath) - raise FetchError("The fetch of %s resulted in a zero size file?! Deleting and failing since this isn't right." % (uri), uri) + os.rename(localpath, finalpath) return True @@ -224,26 +219,20 @@ class Wget(FetchMethod): try: h.request(req.get_method(), req.selector, req.data, headers) - except socket.error as err: # XXX what error? - # Don't close connection when cache is enabled. - # Instead, try to detect connections that are no longer - # usable (for example, closed unexpectedly) and remove - # them from the cache. - if fetch.connection_cache is None: - h.close() - elif isinstance(err, OSError) and err.errno == errno.EBADF: - # This happens when the server closes the connection despite the Keep-Alive. - # Apparently urllib then uses the file descriptor, expecting it to be - # connected, when in reality the connection is already gone. - # We let the request fail and expect it to be - # tried once more ("try_again" in check_status()), - # with the dead connection removed from the cache. - # If it still fails, we give up, which can happen for bad - # HTTP proxy settings. - fetch.connection_cache.remove_connection(h.host, h.port) - raise urllib.error.URLError(err) - else: r = h.getresponse() + except: + # This can happen when the server closes the connection despite the Keep-Alive. + # Apparently urllib then uses the file descriptor, expecting it to be + # connected, when in reality the connection is already gone. + # We let the request fail and expect it to be + # tried once more ("try_again" in check_status()), + # with the dead connection removed from the cache. + # If it still fails, we give up, which can happen for bad + # HTTP proxy settings. + if fetch.connection_cache: + fetch.connection_cache.remove_connection(h.host, h.port) + h.close() + raise # Pick apart the HTTPResponse object to get the addinfourl # object initialized properly. @@ -302,15 +291,70 @@ class Wget(FetchMethod): http_error_403 = http_error_405 + def _url_origin(url): + parsed = urllib.parse.urlsplit(url) + scheme = parsed.scheme.lower() + host = parsed.hostname.lower() if parsed.hostname else "" + port = parsed.port + if port is None: + port = {"http": 80, "https": 443}.get(scheme) + return (scheme, host, port) + + def _same_origin(url_a, url_b): + return _url_origin(url_a) == _url_origin(url_b) + class FixedHTTPRedirectHandler(urllib.request.HTTPRedirectHandler): """ - urllib2.HTTPRedirectHandler resets the method to GET on redirect, - when we want to follow redirects using the original method. + urllib2.HTTPRedirectHandler before 3.13 has two flaws: + + It resets the method to GET on redirect when we want to follow + redirects using the original method (typically HEAD). This was fixed + in 759e8e7. + + It also doesn't handle 308 (Permanent Redirect). This was fixed in + c379bc5. + + Until we depend on Python 3.13 onwards, copy the redirect_request + method to fix these issues. + + Additionally, strip sensitive headers (Authorization, Cookie) when + redirecting to a different origin to avoid credential leaks. """ def redirect_request(self, req, fp, code, msg, headers, newurl): - newreq = urllib.request.HTTPRedirectHandler.redirect_request(self, req, fp, code, msg, headers, newurl) - newreq.get_method = req.get_method - return newreq + m = req.get_method() + if (not (code in (301, 302, 303, 307, 308) and m in ("GET", "HEAD") + or code in (301, 302, 303) and m == "POST")): + raise urllib.HTTPError(req.full_url, code, msg, headers, fp) + + # Strictly (according to RFC 2616), 301 or 302 in response to + # a POST MUST NOT cause a redirection without confirmation + # from the user (of urllib.request, in this case). In practice, + # essentially all clients do redirect in this case, so we do + # the same. + + # Be conciliant with URIs containing a space. This is mainly + # redundant with the more complete encoding done in http_error_302(), + # but it is kept for compatibility with other callers. + newurl = newurl.replace(' ', '%20') + + CONTENT_HEADERS = ("content-length", "content-type") + SENSITIVE_REDIRECT_HEADERS = ("authorization", "cookie") + same_origin = _same_origin(req.get_full_url(), newurl) + newheaders = {} + for k, v in req.headers.items(): + header = k.lower() + if header in CONTENT_HEADERS: + continue + if not same_origin and header in SENSITIVE_REDIRECT_HEADERS: + continue + newheaders[k] = v + return urllib.request.Request(newurl, + method="HEAD" if m == "HEAD" else "GET", + headers=newheaders, + origin_req_host=req.origin_req_host, + unverifiable=True) + + http_error_308 = urllib.request.HTTPRedirectHandler.http_error_302 # We need to update the environment here as both the proxy and HTTPS # handlers need variables set. The proxy needs http_proxy and friends to @@ -343,14 +387,17 @@ class Wget(FetchMethod): opener = urllib.request.build_opener(*handlers) try: - uri_base = ud.url.split(";")[0] - uri = "{}://{}{}".format(urllib.parse.urlparse(uri_base).scheme, ud.host, ud.path) + parts = urllib.parse.urlparse(ud.url.split(";")[0]) + if parts.query: + uri = "{}://{}{}?{}".format(parts.scheme, parts.netloc, parts.path, parts.query) + else: + uri = "{}://{}{}".format(parts.scheme, parts.netloc, parts.path) r = urllib.request.Request(uri) r.get_method = lambda: "HEAD" # Some servers (FusionForge, as used on Alioth) require that the # optional Accept header is set. r.add_header("Accept", "*/*") - r.add_header("User-Agent", self.user_agent) + r.add_header("User-Agent", "bitbake/{}".format(bb.__version__)) def add_basic_auth(login_str, request): '''Adds Basic auth to http request, pass in login:password as string''' import base64 @@ -370,15 +417,20 @@ class Wget(FetchMethod): except (FileNotFoundError, netrc.NetrcParseError): pass - with opener.open(r, timeout=30) as response: + with opener.open(r, timeout=100) as response: pass - except (urllib.error.URLError, ConnectionResetError, TimeoutError) as e: + except (urllib.error.URLError, OSError, http.client.RemoteDisconnected) as e: if try_again: - logger.debug2("checkstatus: trying again") + logger.debug2("checkstatus: trying again after exception %s" % str(e)) return self.checkstatus(fetch, ud, d, False) else: # debug for now to avoid spamming the logs in e.g. remote sstate searches - logger.debug2("checkstatus() urlopen failed for %s: %s" % (uri,e)) + logger.debug2("checkstatus() urlopen failed for %s: %s" % (uri, e)) + if "Bad file descriptor" in str(e): + files = os.listdir("/proc/self/fd") + logger.warn("listing of /proc/self/fd (%d):" % len(files) + str(files)) + logger.warn("lsof output: " + subprocess.check_output(["lsof", "-p", str(os.getpid())])) + logger.warn("checkstatus() urlopen failed for %s: %s" % (uri, e)) return False return True @@ -457,7 +509,7 @@ class Wget(FetchMethod): f = tempfile.NamedTemporaryFile() with tempfile.TemporaryDirectory(prefix="wget-index-") as workdir, tempfile.NamedTemporaryFile(dir=workdir, prefix="wget-listing-") as f: fetchcmd = self.basecmd - fetchcmd += " -O " + f.name + " --user-agent='" + self.user_agent + "' '" + uri + "'" + fetchcmd += " --output-document=%s '%s'" % (f.name, uri) try: self._runwget(ud, d, fetchcmd, True, workdir=workdir) fetchresult = f.read() @@ -466,7 +518,7 @@ class Wget(FetchMethod): return fetchresult - def _check_latest_version(self, url, package, package_regex, current_version, ud, d): + def _check_latest_version(self, url, package, package_regex, current_version, ud, d, filter_regex=None): """ Return the latest version of a package inside a given directory path If error or no version, return "" @@ -489,6 +541,11 @@ class Wget(FetchMethod): newver = self._parse_path(package_regex, str(line)) if newver: + if filter_regex: + # filter_regex is derived from PV, and uses . as separator + # while upstream may be using _ as separator + if not re.match(filter_regex, re.sub('_', '.', newver[1])): + continue bb.debug(3, "Upstream version found: %s" % newver[1]) if valid == 0: version = newver @@ -506,14 +563,14 @@ class Wget(FetchMethod): return "" - def _check_latest_version_by_dir(self, dirver, package, package_regex, current_version, ud, d): + def _check_latest_version_by_dir(self, dirver, package, package_regex, current_version, ud, d, filter_regex=None): """ Scan every directory in order to get upstream version. """ version_dir = ['', '', ''] version = ['', '', ''] - dirver_regex = re.compile(r"(?P\D*)(?P(\d+[\.\-_])*(\d+))") + dirver_regex = re.compile(r"^(?P\D*)(?P(\d+[\.\-_])*(\d+))$") s = dirver_regex.search(dirver) if s: version_dir[1] = s.group('ver') @@ -550,7 +607,7 @@ class Wget(FetchMethod): ud.user, ud.pswd, {}]) pupver = self._check_latest_version(uri, - package, package_regex, current_version, ud, d) + package, package_regex, current_version, ud, d, filter_regex) if pupver: version[1] = pupver @@ -611,19 +668,23 @@ class Wget(FetchMethod): return package_custom_regex_comp - def latest_versionstring(self, ud, d): + def latest_versionstring(self, ud, d, filter_regex=None): """ Manipulate the URL and try to obtain the latest package version sanity check to ensure same name and type. """ - package = ud.path.split("/")[-1] + if 'downloadfilename' in ud.parm: + package = ud.parm['downloadfilename'] + else: + package = ud.path.split("/")[-1] current_version = ['', d.getVar('PV'), ''] """possible to have no version in pkg name, such as spectrum-fw""" if not re.search(r"\d+", package): current_version[1] = re.sub('_', '.', current_version[1]) current_version[1] = re.sub('-', '.', current_version[1]) + bb.debug(3, "latest_versionstring: no version found in %s" % package) return (current_version[1], '') package_regex = self._init_regexes(package, ud, d) @@ -648,11 +709,11 @@ class Wget(FetchMethod): dirver_pn_regex = re.compile(r"%s\d?" % (re.escape(pn))) if not dirver_pn_regex.search(dirver): return (self._check_latest_version_by_dir(dirver, - package, package_regex, current_version, ud, d), '') + package, package_regex, current_version, ud, d, filter_regex), '') uri = bb.fetch.encodeurl([ud.type, ud.host, path, ud.user, ud.pswd, {}]) else: uri = regex_uri return (self._check_latest_version(uri, package, package_regex, - current_version, ud, d), '') + current_version, ud, d, filter_regex), '') diff --git a/bitbake/lib/bb/filter.py b/bitbake/lib/bb/filter.py new file mode 100644 index 00000000..0b5b5d92 --- /dev/null +++ b/bitbake/lib/bb/filter.py @@ -0,0 +1,142 @@ +# +# Copyright (C) 2025 Garmin Ltd. or its subsidiaries +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import builtins + +# Purposely blank out __builtins__ which prevents users from +# calling any normal builtin python functions +FILTERS = { + "__builtins__": {}, +} + +CACHE = {} + + +def apply_filters(val, expressions): + g = FILTERS.copy() + + for e in expressions: + e = e.strip() + if not e: + continue + + k = (val, e) + if k not in CACHE: + # Set val as a local so it can be cleared out while keeping the + # globals + l = {"val": val} + + CACHE[k] = eval(e, g, l) + + val = CACHE[k] + + return val + + +class Namespace(object): + """ + Helper class to simulate a python namespace. The object properties can be + set as if it were a dictionary. Properties cannot be changed or deleted + through the object interface + """ + + def __getitem__(self, name): + return self.__dict__[name] + + def __setitem__(self, name, value): + self.__dict__[name] = value + + def __contains__(self, name): + return name in self.__dict__ + + def __setattr__(self, name, value): + raise AttributeError(f"Attribute {name!r} cannot be changed") + + def __delattr__(self, name): + raise AttributeError(f"Attribute {name!r} cannot be deleted") + + +def filter_proc(*, name=None): + """ + Decorator to mark a function that can be called in `apply_filters`, either + directly in a filter expression, or indirectly. The `name` argument can be + used to specify an alternate name for the function if the actual name is + not desired. The `name` can be a fully qualified namespace if desired. + + All functions must be "pure" in that they do not depend on global state and + have no global side effects (e.g. the output only depends on the input + arguments); the results of filter expressions are cached to optimize + repeated calls. + """ + + def inner(func): + global FILTERS + nonlocal name + + if name is None: + name = func.__name__ + + ns = name.split(".") + o = FILTERS + for n in ns[:-1]: + if not n in o: + o[n] = Namespace() + o = o[n] + + o[ns[-1]] = func + + return func + + return inner + + +# A select set of builtins that are supported in filter expressions +filter_proc()(all) +filter_proc()(all) +filter_proc()(any) +filter_proc()(bin) +filter_proc()(bool) +filter_proc()(chr) +filter_proc()(enumerate) +filter_proc()(float) +filter_proc()(format) +filter_proc()(hex) +filter_proc()(int) +filter_proc()(len) +filter_proc()(map) +filter_proc()(max) +filter_proc()(min) +filter_proc()(oct) +filter_proc()(ord) +filter_proc()(pow) +filter_proc()(str) +filter_proc()(sum) + + +@filter_proc() +def suffix(val, suffix): + return " ".join(v + suffix for v in val.split()) + + +@filter_proc() +def prefix(val, prefix): + return " ".join(prefix + v for v in val.split()) + + +@filter_proc() +def sort(val): + return " ".join(sorted(val.split())) + + +@filter_proc() +def remove(val, remove, sep=None): + if isinstance(remove, str): + remove = remove.split(sep) + new = [i for i in val.split(sep) if not i in remove] + + if not sep: + return " ".join(new) + return sep.join(new) diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py index bca8ebfa..597cb278 100755 --- a/bitbake/lib/bb/main.py +++ b/bitbake/lib/bb/main.py @@ -208,8 +208,10 @@ def create_bitbake_parser(): "failed and anything depending on it cannot be built, as much as " "possible will be built before stopping.") - exec_group.add_argument("-P", "--profile", action="store_true", - help="Profile the command and save reports.") + exec_group.add_argument("-P", "--profile", action="append", + default=[], + help="Profile the command and save reports. Specify 'main', 'idle' or 'parsing' " + "to indicate which bitbake code to profile.") exec_group.add_argument("-S", "--dump-signatures", action="append", default=[], metavar="SIGNATURE_HANDLER", diff --git a/bitbake/lib/bb/parse/__init__.py b/bitbake/lib/bb/parse/__init__.py index a4358f13..d428d8a4 100644 --- a/bitbake/lib/bb/parse/__init__.py +++ b/bitbake/lib/bb/parse/__init__.py @@ -49,20 +49,23 @@ class SkipPackage(SkipRecipe): __mtime_cache = {} def cached_mtime(f): if f not in __mtime_cache: - __mtime_cache[f] = os.stat(f)[stat.ST_MTIME] + res = os.stat(f) + __mtime_cache[f] = (res.st_mtime_ns, res.st_size, res.st_ino) return __mtime_cache[f] def cached_mtime_noerror(f): if f not in __mtime_cache: try: - __mtime_cache[f] = os.stat(f)[stat.ST_MTIME] + res = os.stat(f) + __mtime_cache[f] = (res.st_mtime_ns, res.st_size, res.st_ino) except OSError: return 0 return __mtime_cache[f] def check_mtime(f, mtime): try: - current_mtime = os.stat(f)[stat.ST_MTIME] + res = os.stat(f) + current_mtime = (res.st_mtime_ns, res.st_size, res.st_ino) __mtime_cache[f] = current_mtime except OSError: current_mtime = 0 @@ -70,7 +73,8 @@ def check_mtime(f, mtime): def update_mtime(f): try: - __mtime_cache[f] = os.stat(f)[stat.ST_MTIME] + res = os.stat(f) + __mtime_cache[f] = (res.st_mtime_ns, res.st_size, res.st_ino) except OSError: if f in __mtime_cache: del __mtime_cache[f] @@ -172,4 +176,41 @@ def get_file_depends(d): dep_files.append(os.path.abspath(fn)) return " ".join(dep_files) +def vardeps(*varnames): + """ + Function decorator that can be used to instruct the bitbake dependency + parsing to add a dependency on the specified variables names + + Example: + + @bb.parse.vardeps("FOO", "BAR") + def my_function(): + ... + + """ + def inner(f): + if not hasattr(f, "bb_vardeps"): + f.bb_vardeps = set() + f.bb_vardeps |= set(varnames) + return f + return inner + +def vardepsexclude(*varnames): + """ + Function decorator that can be used to instruct the bitbake dependency + parsing to ignore dependencies on the specified variable names in the code + + Example: + + @bb.parse.vardepsexclude("FOO", "BAR") + def my_function(): + ... + """ + def inner(f): + if not hasattr(f, "bb_vardepsexclude"): + f.bb_vardepsexclude = set() + f.bb_vardepsexclude |= set(varnames) + return f + return inner + from bb.parse.parse_py import __version__, ConfHandler, BBHandler diff --git a/bitbake/lib/bb/parse/ast.py b/bitbake/lib/bb/parse/ast.py index 7581d003..e23ef6db 100644 --- a/bitbake/lib/bb/parse/ast.py +++ b/bitbake/lib/bb/parse/ast.py @@ -43,6 +43,21 @@ class IncludeNode(AstNode): else: bb.parse.ConfHandler.include(self.filename, s, self.lineno, data, False) +class IncludeAllNode(AstNode): + def __init__(self, filename, lineno, what_file): + AstNode.__init__(self, filename, lineno) + self.what_file = what_file + + def eval(self, data): + """ + Include the file and evaluate the statements + """ + s = data.expand(self.what_file) + logger.debug2("CONF %s:%s: including %s", self.filename, self.lineno, s) + + for path in data.getVar("BBPATH").split(":"): + bb.parse.ConfHandler.include(self.filename, os.path.join(path, s), self.lineno, data, False) + class ExportNode(AstNode): def __init__(self, filename, lineno, var): AstNode.__init__(self, filename, lineno) @@ -137,7 +152,10 @@ class DataNode(AstNode): flag = None if 'flag' in groupd and groupd['flag'] is not None: - flag = groupd['flag'] + if groupd["lazyques"]: + flag = "_defaultval_flag_"+groupd['flag'] + else: + flag = groupd['flag'] elif groupd["lazyques"]: flag = "_defaultval" @@ -240,14 +258,16 @@ class ExportFuncsNode(AstNode): data.setVar(func, sentinel + " " + calledfunc + "\n", parsing=True) class AddTaskNode(AstNode): - def __init__(self, filename, lineno, func, before, after): + def __init__(self, filename, lineno, tasks, before, after): AstNode.__init__(self, filename, lineno) - self.func = func + self.tasks = tasks self.before = before self.after = after def eval(self, data): - bb.build.addtask(self.func, self.before, self.after, data) + tasks = self.tasks.split() + for task in tasks: + bb.build.addtask(task, self.before, self.after, data) class DelTaskNode(AstNode): def __init__(self, filename, lineno, tasks): @@ -320,13 +340,101 @@ class InheritDeferredNode(AstNode): self.inherit = (classes, filename, lineno) def eval(self, data): - inherits = data.getVar('__BBDEFINHERITS', False) or [] - inherits.append(self.inherit) - data.setVar('__BBDEFINHERITS', inherits) + bb.parse.BBHandler.inherit_defer(*self.inherit, data) + +class AddFragmentsNode(AstNode): + def __init__(self, filename, lineno, fragments_path_prefix, fragments_variable, flagged_variables_list_variable, builtin_fragments_variable): + AstNode.__init__(self, filename, lineno) + self.fragments_path_prefix = fragments_path_prefix + self.fragments_variable = fragments_variable + self.flagged_variables_list_variable = flagged_variables_list_variable + self.builtin_fragments_variable = builtin_fragments_variable + + def eval(self, data): + if data.getVar('_BB_SKIP_FRAGMENTS') == '1': + return + + # No need to use mark_dependency since we would only match a fragment + # from a specific layer and there can only be a single layer with a + # given namespace. + def find_fragment(layers, layerid, full_fragment_name): + for layerpath in layers.split(): + candidate_fragment_path = os.path.join(layerpath, full_fragment_name) + if os.path.exists(candidate_fragment_path) and bb.utils.get_file_layer(candidate_fragment_path, data) == layerid: + return candidate_fragment_path + return None + + def check_and_set_builtin_fragment(fragment, data, builtin_fragments): + prefix, value = fragment.split('/', 1) + if prefix in builtin_fragments.keys(): + if data.getVar(builtin_fragments[prefix], noweakdefault=True) != None: + bb.fatal( + ("A builtin fragment '%s' is used while %s has already got an assignment.\n" + "Please either disable the fragment or remove the value assignment.\n" + "To disable the fragment, use 'bitbake-config-build disable-fragment %s'." + ) % (fragment, builtin_fragments[prefix], fragment)) + fragment_history = data.varhistory.variable(self.fragments_variable) + loginfo={} + for fh in fragment_history[::-1]: + if fh['op'] in ("set", "append") and fragment in fh["detail"]: + loginfo["file"] = fh["file"] + loginfo["line"] = fh["line"] + loginfo["detail"] = f"{value} ({self.fragments_variable} contains \"{fragment}\")" + break + # parsing=True since we want to emulate X=Y and allow X:override=Z to continue to exist + data.setVar(builtin_fragments[prefix], value, parsing=True, **loginfo) + return True + return False + + fragments = data.getVar(self.fragments_variable) + layers = data.getVar('BBLAYERS') + flagged_variables = data.getVar(self.flagged_variables_list_variable).split() + builtin_fragments = {f[0]:f[1] for f in [f.split(':') for f in data.getVar(self.builtin_fragments_variable).split()] } + + if not fragments: + return + + # Check for multiple builtin fragments setting the same variable + for builtin_fragment_key in builtin_fragments.keys(): + builtin_fragments_list = list( + filter( + lambda f: f.startswith(builtin_fragment_key + "/"), + fragments.split(), + ) + ) + if len(builtin_fragments_list) > 1: + bb.warn( + ("Multiple builtin fragments are enabled for %s via variable %s: %s. " + "This likely points to a mis-configuration in the metadata, as only " + "one of them should be set. The build will use the last value.") + % ( + builtin_fragment_key, + self.fragments_variable, + " ".join(builtin_fragments_list), + ) + ) + + for f in fragments.split(): + if check_and_set_builtin_fragment(f, data, builtin_fragments): + continue + layerid, fragment_name = f.split('/', 1) + full_fragment_name = data.expand("{}/{}.conf".format(self.fragments_path_prefix, fragment_name)) + fragment_path = find_fragment(layers, layerid, full_fragment_name) + if fragment_path: + bb.parse.ConfHandler.include(self.filename, fragment_path, self.lineno, data, "include fragment") + for flagged_var in flagged_variables: + val = data.getVar(flagged_var) + data.setVarFlag(flagged_var, f, val) + data.setVar(flagged_var, None) + else: + bb.fatal("Could not find fragment {} in enabled layers:\n\t{}".format(f, "\n\t".join(layers.split()))) def handleInclude(statements, filename, lineno, m, force): statements.append(IncludeNode(filename, lineno, m.group(1), force)) +def handleIncludeAll(statements, filename, lineno, m): + statements.append(IncludeAllNode(filename, lineno, m.group(1))) + def handleExport(statements, filename, lineno, m): statements.append(ExportNode(filename, lineno, m.group(1))) @@ -348,21 +456,11 @@ def handlePythonMethod(statements, filename, lineno, funcname, modulename, body) def handleExportFuncs(statements, filename, lineno, m, classname): statements.append(ExportFuncsNode(filename, lineno, m.group(1), classname)) -def handleAddTask(statements, filename, lineno, m): - func = m.group("func") - before = m.group("before") - after = m.group("after") - if func is None: - return +def handleAddTask(statements, filename, lineno, tasks, before, after): + statements.append(AddTaskNode(filename, lineno, tasks, before, after)) - statements.append(AddTaskNode(filename, lineno, func, before, after)) - -def handleDelTask(statements, filename, lineno, m): - func = m.group(1) - if func is None: - return - - statements.append(DelTaskNode(filename, lineno, func)) +def handleDelTask(statements, filename, lineno, tasks): + statements.append(DelTaskNode(filename, lineno, tasks)) def handleBBHandlers(statements, filename, lineno, m): statements.append(BBHandlerNode(filename, lineno, m.group(1))) @@ -378,12 +476,43 @@ def handleInheritDeferred(statements, filename, lineno, m): classes = m.group(1) statements.append(InheritDeferredNode(filename, lineno, classes)) +def handleAddFragments(statements, filename, lineno, m): + fragments_path_prefix = m.group(1) + fragments_variable = m.group(2) + flagged_variables_list_variable = m.group(3) + builtin_fragments_variable = m.group(4) + statements.append(AddFragmentsNode(filename, lineno, fragments_path_prefix, fragments_variable, flagged_variables_list_variable, builtin_fragments_variable)) + def runAnonFuncs(d): code = [] for funcname in d.getVar("__BBANONFUNCS", False) or []: code.append("%s(d)" % funcname) bb.utils.better_exec("\n".join(code), {"d": d}) +# Handle recipe level PREFERRED_PROVIDERs +def handleVirtRecipeProviders(tasklist, d): + depends = (d.getVar("DEPENDS") or "").split() + virtprovs = (d.getVar("BB_RECIPE_VIRTUAL_PROVIDERS") or "").split() + newdeps = [] + for dep in depends: + if dep in virtprovs: + newdep = d.getVar("PREFERRED_PROVIDER_" + dep) + if not newdep: + bb.fatal("Error, recipe virtual provider PREFERRED_PROVIDER_%s not set" % dep) + newdeps.append(newdep) + else: + newdeps.append(dep) + d.setVar("DEPENDS", " ".join(newdeps)) + for task in tasklist: + taskdeps = (d.getVarFlag(task, "depends") or "").split() + remapped = [] + for entry in taskdeps: + r, t = entry.split(":") + if r in virtprovs: + r = d.getVar("PREFERRED_PROVIDER_" + r) + remapped.append("%s:%s" % (r, t)) + d.setVarFlag(task, "depends", " ".join(remapped)) + def finalize(fn, d, variant = None): saved_handlers = bb.event.get_handlers().copy() try: @@ -391,6 +520,17 @@ def finalize(fn, d, variant = None): if d.getVar("_FAILPARSINGERRORHANDLED", False) == True: raise bb.BBHandledException() + inherits = [x[0] for x in (d.getVar('__BBDEFINHERITS', False) or [('',)])] + bb.event.fire(bb.event.RecipePreDeferredInherits(fn, inherits), d) + + while True: + inherits = d.getVar('__BBDEFINHERITS', False) or [] + if not inherits: + break + inherit, filename, lineno = inherits.pop(0) + d.setVar('__BBDEFINHERITS', inherits) + bb.parse.BBHandler.inherit(inherit, filename, lineno, d, deferred=True) + for var in d.getVar('__BBHANDLERS', False) or []: # try to add the handler handlerfn = d.getVarFlag(var, "filename", False) @@ -409,6 +549,7 @@ def finalize(fn, d, variant = None): tasklist = d.getVar('__BBTASKS', False) or [] bb.event.fire(bb.event.RecipeTaskPreProcess(fn, list(tasklist)), d) + handleVirtRecipeProviders(tasklist, d) bb.build.add_tasks(tasklist, d) bb.parse.siggen.finalise(fn, d, variant) @@ -444,14 +585,6 @@ def multi_finalize(fn, d): logger.debug("Appending .bbappend file %s to %s", append, fn) bb.parse.BBHandler.handle(append, d, True) - while True: - inherits = d.getVar('__BBDEFINHERITS', False) or [] - if not inherits: - break - inherit, filename, lineno = inherits.pop(0) - d.setVar('__BBDEFINHERITS', inherits) - bb.parse.BBHandler.inherit(inherit, filename, lineno, d, deferred=True) - onlyfinalise = d.getVar("__ONLYFINALISE", False) safe_d = d @@ -487,7 +620,7 @@ def multi_finalize(fn, d): d.setVar("BBEXTENDVARIANT", variantmap[name]) else: d.setVar("PN", "%s-%s" % (pn, name)) - bb.parse.BBHandler.inherit(extendedmap[name], fn, 0, d) + bb.parse.BBHandler.inherit_defer(extendedmap[name], fn, 0, d) safe_d.setVar("BBCLASSEXTEND", extended) _create_variants(datastores, extendedmap.keys(), extendfunc, onlyfinalise) diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py b/bitbake/lib/bb/parse/parse_py/BBHandler.py index c13e4b97..008fec23 100644 --- a/bitbake/lib/bb/parse/parse_py/BBHandler.py +++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py @@ -23,8 +23,8 @@ __func_start_regexp__ = re.compile(r"(((?Ppython(?=(\s|\()))|(?Pfaker __inherit_regexp__ = re.compile(r"inherit\s+(.+)" ) __inherit_def_regexp__ = re.compile(r"inherit_defer\s+(.+)" ) __export_func_regexp__ = re.compile(r"EXPORT_FUNCTIONS\s+(.+)" ) -__addtask_regexp__ = re.compile(r"addtask\s+(?P\w+)\s*((before\s*(?P((.*(?=after))|(.*))))|(after\s*(?P((.*(?=before))|(.*)))))*") -__deltask_regexp__ = re.compile(r"deltask\s+(.+)") +__addtask_regexp__ = re.compile(r"addtask\s+([^#\n]+)(?P#.*|.*?)") +__deltask_regexp__ = re.compile(r"deltask\s+([^#\n]+)(?P#.*|.*?)") __addhandler_regexp__ = re.compile(r"addhandler\s+(.+)" ) __def_regexp__ = re.compile(r"def\s+(\w+).*:" ) __python_func_regexp__ = re.compile(r"(\s+.*)|(^$)|(^#)" ) @@ -42,12 +42,22 @@ def supports(fn, d): """Return True if fn has a supported extension""" return os.path.splitext(fn)[-1] in [".bb", ".bbclass", ".inc"] +def inherit_defer(expression, fn, lineno, d): + inherit = (expression, fn, lineno) + inherits = d.getVar('__BBDEFINHERITS', False) or [] + inherits.append(inherit) + d.setVar('__BBDEFINHERITS', inherits) + def inherit(files, fn, lineno, d, deferred=False): __inherit_cache = d.getVar('__inherit_cache', False) or [] #if "${" in files and not deferred: # bb.warn("%s:%s has non deferred conditional inherit" % (fn, lineno)) files = d.expand(files).split() for file in files: + defer = (d.getVar("BB_DEFER_BBCLASSES") or "").split() + if not deferred and file in defer: + inherit_defer(file, fn, lineno, d) + continue classtype = d.getVar("__bbclasstype", False) origfile = file for t in ["classes-" + classtype, "classes"]: @@ -239,29 +249,38 @@ def feeder(lineno, s, fn, root, statements, eof=False): m = __addtask_regexp__.match(s) if m: - if len(m.group().split()) == 2: - # Check and warn for "addtask task1 task2" - m2 = re.match(r"addtask\s+(?P\w+)(?P.*)", s) - if m2 and m2.group('ignores'): - logger.warning('addtask ignored: "%s"' % m2.group('ignores')) - - # Check and warn for "addtask task1 before task2 before task3", the - # similar to "after" - taskexpression = s.split() - for word in ('before', 'after'): - if taskexpression.count(word) > 1: - logger.warning("addtask contained multiple '%s' keywords, only one is supported" % word) + after = "" + before = "" + + # This code splits on 'before' and 'after' instead of on whitespace so we can defer + # evaluation to as late as possible. + tasks = m.group(1).split(" before ")[0].split(" after ")[0] + + for exp in m.group(1).split(" before "): + exp2 = exp.split(" after ") + if len(exp2) > 1: + after = after + " ".join(exp2[1:]) - # Check and warn for having task with exprssion as part of task name + for exp in m.group(1).split(" after "): + exp2 = exp.split(" before ") + if len(exp2) > 1: + before = before + " ".join(exp2[1:]) + + # Check and warn for having task with a keyword as part of task name + taskexpression = s.split() for te in taskexpression: if any( ( "%s_" % keyword ) in te for keyword in bb.data_smart.__setvar_keyword__ ): raise ParseError("Task name '%s' contains a keyword which is not recommended/supported.\nPlease rename the task not to include the keyword.\n%s" % (te, ("\n".join(map(str, bb.data_smart.__setvar_keyword__)))), fn) - ast.handleAddTask(statements, fn, lineno, m) + + if tasks is not None: + ast.handleAddTask(statements, fn, lineno, tasks, before, after) return m = __deltask_regexp__.match(s) if m: - ast.handleDelTask(statements, fn, lineno, m) + task = m.group(1) + if task is not None: + ast.handleDelTask(statements, fn, lineno, task) return m = __addhandler_regexp__.match(s) diff --git a/bitbake/lib/bb/parse/parse_py/ConfHandler.py b/bitbake/lib/bb/parse/parse_py/ConfHandler.py index 7826dee7..9ddbae12 100644 --- a/bitbake/lib/bb/parse/parse_py/ConfHandler.py +++ b/bitbake/lib/bb/parse/parse_py/ConfHandler.py @@ -20,10 +20,10 @@ from bb.parse import ParseError, resolve_file, ast, logger, handle __config_regexp__ = re.compile( r""" ^ (?Pexport\s+)? - (?P[a-zA-Z0-9\-_+.${}/~:]+?) - (\[(?P[a-zA-Z0-9\-_+.][a-zA-Z0-9\-_+.@]*)\])? + (?P[a-zA-Z0-9\-_+.${}/~:]*?) + (\[(?P[a-zA-Z0-9\-_+.][a-zA-Z0-9\-_+.@/]*)\])? - \s* ( + (?P\s*) ( (?P:=) | (?P\?\?=) | (?P\?=) | @@ -32,7 +32,7 @@ __config_regexp__ = re.compile( r""" (?P=\.) | (?P\.=) | = - ) \s* + ) (?P\s*) (?!'[^']*'[^']*'$) (?!\"[^\"]*\"[^\"]*\"$) @@ -43,10 +43,12 @@ __config_regexp__ = re.compile( r""" """, re.X) __include_regexp__ = re.compile( r"include\s+(.+)" ) __require_regexp__ = re.compile( r"require\s+(.+)" ) +__includeall_regexp__ = re.compile( r"include_all\s+(.+)" ) __export_regexp__ = re.compile( r"export\s+([a-zA-Z0-9\-_+.${}/~]+)$" ) __unset_regexp__ = re.compile( r"unset\s+([a-zA-Z0-9\-_+.${}/~]+)$" ) __unset_flag_regexp__ = re.compile( r"unset\s+([a-zA-Z0-9\-_+.${}/~]+)\[([a-zA-Z0-9\-_+.][a-zA-Z0-9\-_+.@]+)\]$" ) __addpylib_regexp__ = re.compile(r"addpylib\s+(.+)\s+(.+)" ) +__addfragments_regexp__ = re.compile(r"addfragments\s+(.+)\s+(.+)\s+(.+)\s+(.+)" ) def init(data): return @@ -164,6 +166,10 @@ def feeder(lineno, s, fn, statements, baseconfig=False, conffile=True): m = __config_regexp__.match(s) if m: groupd = m.groupdict() + if groupd['var'] == "": + raise ParseError("Empty variable name in assignment: '%s'" % s, fn, lineno); + if not groupd['whitespace'] or not groupd['whitespace2']: + logger.warning("%s:%s has a lack of whitespace around the assignment: '%s'" % (fn, lineno, s)) ast.handleData(statements, fn, lineno, groupd) return @@ -177,6 +183,11 @@ def feeder(lineno, s, fn, statements, baseconfig=False, conffile=True): ast.handleInclude(statements, fn, lineno, m, True) return + m = __includeall_regexp__.match(s) + if m: + ast.handleIncludeAll(statements, fn, lineno, m) + return + m = __export_regexp__.match(s) if m: ast.handleExport(statements, fn, lineno, m) @@ -197,6 +208,11 @@ def feeder(lineno, s, fn, statements, baseconfig=False, conffile=True): ast.handlePyLib(statements, fn, lineno, m) return + m = __addfragments_regexp__.match(s) + if m: + ast.handleAddFragments(statements, fn, lineno, m) + return + raise ParseError("unparsed line: '%s'" % s, fn, lineno); # Add us to the handlers list diff --git a/bitbake/lib/bb/persist_data.py b/bitbake/lib/bb/persist_data.py deleted file mode 100644 index bcca791e..00000000 --- a/bitbake/lib/bb/persist_data.py +++ /dev/null @@ -1,271 +0,0 @@ -"""BitBake Persistent Data Store - -Used to store data in a central location such that other threads/tasks can -access them at some future date. Acts as a convenience wrapper around sqlite, -currently, providing a key/value store accessed by 'domain'. -""" - -# Copyright (C) 2007 Richard Purdie -# Copyright (C) 2010 Chris Larson -# -# SPDX-License-Identifier: GPL-2.0-only -# - -import collections -import collections.abc -import contextlib -import functools -import logging -import os.path -import sqlite3 -import sys -from collections.abc import Mapping - -sqlversion = sqlite3.sqlite_version_info -if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3): - raise Exception("sqlite3 version 3.3.0 or later is required.") - - -logger = logging.getLogger("BitBake.PersistData") - -@functools.total_ordering -class SQLTable(collections.abc.MutableMapping): - class _Decorators(object): - @staticmethod - def retry(*, reconnect=True): - """ - Decorator that restarts a function if a database locked sqlite - exception occurs. If reconnect is True, the database connection - will be closed and reopened each time a failure occurs - """ - def retry_wrapper(f): - def wrap_func(self, *args, **kwargs): - # Reconnect if necessary - if self.connection is None and reconnect: - self.reconnect() - - count = 0 - while True: - try: - return f(self, *args, **kwargs) - except sqlite3.OperationalError as exc: - if count < 500 and ('is locked' in str(exc) or 'locking protocol' in str(exc)): - count = count + 1 - if reconnect: - self.reconnect() - continue - raise - return wrap_func - return retry_wrapper - - @staticmethod - def transaction(f): - """ - Decorator that starts a database transaction and creates a database - cursor for performing queries. If no exception is thrown, the - database results are committed. If an exception occurs, the database - is rolled back. In all cases, the cursor is closed after the - function ends. - - Note that the cursor is passed as an extra argument to the function - after `self` and before any of the normal arguments - """ - def wrap_func(self, *args, **kwargs): - # Context manager will COMMIT the database on success, - # or ROLLBACK on an exception - with self.connection: - # Automatically close the cursor when done - with contextlib.closing(self.connection.cursor()) as cursor: - return f(self, cursor, *args, **kwargs) - return wrap_func - - """Object representing a table/domain in the database""" - def __init__(self, cachefile, table): - self.cachefile = cachefile - self.table = table - - self.connection = None - self._execute_single("CREATE TABLE IF NOT EXISTS %s(key TEXT PRIMARY KEY NOT NULL, value TEXT);" % table) - - @_Decorators.retry(reconnect=False) - @_Decorators.transaction - def _setup_database(self, cursor): - cursor.execute("pragma synchronous = off;") - # Enable WAL and keep the autocheckpoint length small (the default is - # usually 1000). Persistent caches are usually read-mostly, so keeping - # this short will keep readers running quickly - cursor.execute("pragma journal_mode = WAL;") - cursor.execute("pragma wal_autocheckpoint = 100;") - - def reconnect(self): - if self.connection is not None: - self.connection.close() - self.connection = sqlite3.connect(self.cachefile, timeout=5) - self.connection.text_factory = str - self._setup_database() - - @_Decorators.retry() - @_Decorators.transaction - def _execute_single(self, cursor, *query): - """ - Executes a single query and discards the results. This correctly closes - the database cursor when finished - """ - cursor.execute(*query) - - @_Decorators.retry() - def _row_iter(self, f, *query): - """ - Helper function that returns a row iterator. Each time __next__ is - called on the iterator, the provided function is evaluated to determine - the return value - """ - class CursorIter(object): - def __init__(self, cursor): - self.cursor = cursor - - def __iter__(self): - return self - - def __next__(self): - row = self.cursor.fetchone() - if row is None: - self.cursor.close() - raise StopIteration - return f(row) - - def __enter__(self): - return self - - def __exit__(self, typ, value, traceback): - self.cursor.close() - return False - - cursor = self.connection.cursor() - try: - cursor.execute(*query) - return CursorIter(cursor) - except: - cursor.close() - - def __enter__(self): - self.connection.__enter__() - return self - - def __exit__(self, *excinfo): - self.connection.__exit__(*excinfo) - - @_Decorators.retry() - @_Decorators.transaction - def __getitem__(self, cursor, key): - cursor.execute("SELECT * from %s where key=?;" % self.table, [key]) - row = cursor.fetchone() - if row is not None: - return row[1] - raise KeyError(key) - - @_Decorators.retry() - @_Decorators.transaction - def __delitem__(self, cursor, key): - if key not in self: - raise KeyError(key) - cursor.execute("DELETE from %s where key=?;" % self.table, [key]) - - @_Decorators.retry() - @_Decorators.transaction - def __setitem__(self, cursor, key, value): - if not isinstance(key, str): - raise TypeError('Only string keys are supported') - elif not isinstance(value, str): - raise TypeError('Only string values are supported') - - # Ensure the entire transaction (including SELECT) executes under write lock - cursor.execute("BEGIN EXCLUSIVE") - - cursor.execute("SELECT * from %s where key=?;" % self.table, [key]) - row = cursor.fetchone() - if row is not None: - cursor.execute("UPDATE %s SET value=? WHERE key=?;" % self.table, [value, key]) - else: - cursor.execute("INSERT into %s(key, value) values (?, ?);" % self.table, [key, value]) - - @_Decorators.retry() - @_Decorators.transaction - def __contains__(self, cursor, key): - cursor.execute('SELECT * from %s where key=?;' % self.table, [key]) - return cursor.fetchone() is not None - - @_Decorators.retry() - @_Decorators.transaction - def __len__(self, cursor): - cursor.execute("SELECT COUNT(key) FROM %s;" % self.table) - row = cursor.fetchone() - if row is not None: - return row[0] - - def __iter__(self): - return self._row_iter(lambda row: row[0], "SELECT key from %s;" % self.table) - - def __lt__(self, other): - if not isinstance(other, Mapping): - raise NotImplementedError() - - return len(self) < len(other) - - def get_by_pattern(self, pattern): - return self._row_iter(lambda row: row[1], "SELECT * FROM %s WHERE key LIKE ?;" % - self.table, [pattern]) - - def values(self): - return list(self.itervalues()) - - def itervalues(self): - return self._row_iter(lambda row: row[0], "SELECT value FROM %s;" % - self.table) - - def items(self): - return list(self.iteritems()) - - def iteritems(self): - return self._row_iter(lambda row: (row[0], row[1]), "SELECT * FROM %s;" % - self.table) - - @_Decorators.retry() - @_Decorators.transaction - def clear(self, cursor): - cursor.execute("DELETE FROM %s;" % self.table) - - def has_key(self, key): - return key in self - -def persist(domain, d): - """Convenience factory for SQLTable objects based upon metadata""" - import bb.utils - cachedir = (d.getVar("PERSISTENT_DIR") or - d.getVar("CACHE")) - if not cachedir: - logger.critical("Please set the 'PERSISTENT_DIR' or 'CACHE' variable") - sys.exit(1) - - bb.utils.mkdirhier(cachedir) - cachefile = os.path.join(cachedir, "bb_persist_data.sqlite3") - - try: - return SQLTable(cachefile, domain) - except sqlite3.OperationalError: - # Sqlite fails to open database when its path is too long. - # After testing, 504 is the biggest path length that can be opened by - # sqlite. - # Note: This code is called before sanity.bbclass and its path length - # check - max_len = 504 - if len(cachefile) > max_len: - logger.critical("The path of the cache file is too long " - "({0} chars > {1}) to be opened by sqlite! " - "Your cache file is \"{2}\"".format( - len(cachefile), - max_len, - cachefile)) - sys.exit(1) - else: - raise diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index bc7e1817..87063336 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -10,10 +10,12 @@ Handles preparation and execution of a queue of tasks # import copy +import enum import os import sys import stat import errno +import itertools import logging import re import bb @@ -21,14 +23,16 @@ from bb import msg, event from bb import monitordisk import subprocess import pickle -from multiprocessing import Process import shlex import pprint import time +Process = bb.multiprocessing.Process + bblogger = logging.getLogger("BitBake") logger = logging.getLogger("BitBake.RunQueue") hashequiv_logger = logging.getLogger("BitBake.RunQueue.HashEquiv") +psi_logger = logging.getLogger("BitBake.RunQueue.PSI") __find_sha256__ = re.compile( r'(?i)(? PN (fn) table for mc in taskData: @@ -1269,31 +1277,45 @@ class RunQueueData: bb.parse.siggen.tasks_resolved(virtmap, virtpnmap, self.dataCaches[mc]) self.init_progress_reporter.next_stage() - bb.event.check_for_interrupts(self.cooker.data) + bb.event.check_for_interrupts() bb.parse.siggen.set_setscene_tasks(self.runq_setscene_tids) + starttime = time.time() + lasttime = starttime + # Iterate over the task list and call into the siggen code dealtwith = set() todeal = set(self.runtaskentries) while todeal: + ready = set() for tid in todeal.copy(): if not (self.runtaskentries[tid].depends - dealtwith): - dealtwith.add(tid) - todeal.remove(tid) - self.prepare_task_hash(tid) - bb.event.check_for_interrupts(self.cooker.data) + self.runtaskentries[tid].taskhash_deps = bb.parse.siggen.prep_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches) + # get_taskhash for a given tid *must* be called before get_unihash* below + self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches) + ready.add(tid) + unihashes = bb.parse.siggen.get_unihashes(ready) + for tid in ready: + dealtwith.add(tid) + todeal.remove(tid) + self.runtaskentries[tid].unihash = unihashes[tid] + + bb.event.check_for_interrupts() + + if time.time() > (lasttime + 30): + lasttime = time.time() + hashequiv_logger.verbose("Initial setup loop progress: %s of %s in %s" % (len(todeal), len(self.runtaskentries), lasttime - starttime)) + + endtime = time.time() + if (endtime-starttime > 60): + hashequiv_logger.verbose("Initial setup loop took: %s" % (endtime-starttime)) bb.parse.siggen.writeout_file_checksum_cache() #self.dump_data() return len(self.runtaskentries) - def prepare_task_hash(self, tid): - bb.parse.siggen.prep_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches) - self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches) - self.runtaskentries[tid].unihash = bb.parse.siggen.get_unihash(tid) - def dump_data(self): """ Dump some debug information on the internal data structures @@ -1320,13 +1342,13 @@ class RunQueue: self.hashvalidate = cfgData.getVar("BB_HASHCHECK_FUNCTION") or None self.depvalidate = cfgData.getVar("BB_SETSCENE_DEPVALID") or None - self.state = runQueuePrepare + self.state = RunQueueState.PREPARE # For disk space monitor # Invoked at regular time intervals via the bitbake heartbeat event # while the build is running. We generate a unique name for the handler # here, just in case that there ever is more than one RunQueue instance, - # start the handler when reaching runQueueSceneInit, and stop it when + # start the handler when reaching RunQueueState.SCENE_INIT, and stop it when # done with the build. self.dm = monitordisk.diskMonitor(cfgData) self.dm_event_handler_name = '_bb_diskmonitor_' + str(id(self)) @@ -1538,9 +1560,9 @@ class RunQueue: """ retval = True - bb.event.check_for_interrupts(self.cooker.data) + bb.event.check_for_interrupts() - if self.state is runQueuePrepare: + if self.state == RunQueueState.PREPARE: # NOTE: if you add, remove or significantly refactor the stages of this # process then you should recalculate the weightings here. This is quite # easy to do - just change the next line temporarily to pass debug=True as @@ -1551,12 +1573,12 @@ class RunQueue: "Initialising tasks", [43, 967, 4, 3, 1, 5, 3, 7, 13, 1, 2, 1, 1, 246, 35, 1, 38, 1, 35, 2, 338, 204, 142, 3, 3, 37, 244]) if self.rqdata.prepare() == 0: - self.state = runQueueComplete + self.state = RunQueueState.COMPLETE else: - self.state = runQueueSceneInit + self.state = RunQueueState.SCENE_INIT bb.parse.siggen.save_unitaskhashes() - if self.state is runQueueSceneInit: + if self.state == RunQueueState.SCENE_INIT: self.rqdata.init_progress_reporter.next_stage() # we are ready to run, emit dependency info to any UI or class which @@ -1567,24 +1589,29 @@ class RunQueue: if not self.dm_event_handler_registered: res = bb.event.register(self.dm_event_handler_name, - lambda x, y: self.dm.check(self) if self.state in [runQueueRunning, runQueueCleanUp] else False, + lambda x, y: self.dm.check(self) if self.state in [RunQueueState.RUNNING, RunQueueState.CLEAN_UP] else False, ('bb.event.HeartbeatEvent',), data=self.cfgData) self.dm_event_handler_registered = True self.rqdata.init_progress_reporter.next_stage() self.rqexe = RunQueueExecute(self) - dump = self.cooker.configuration.dump_signatures - if dump: + dumpsigs = self.cooker.configuration.dump_signatures + if dumpsigs: self.rqdata.init_progress_reporter.finish() - if 'printdiff' in dump: - invalidtasks = self.print_diffscenetasks() - self.dump_signatures(dump) - if 'printdiff' in dump: - self.write_diffscenetasks(invalidtasks) - self.state = runQueueComplete - - if self.state is runQueueSceneInit: + if 'printdiff' in dumpsigs: + self.invalidtasks_dump = self.print_diffscenetasks() + self.state = RunQueueState.DUMP_SIGS + + if self.state == RunQueueState.DUMP_SIGS: + dumpsigs = self.cooker.configuration.dump_signatures + retval = self.dump_signatures(dumpsigs) + if retval is False: + if 'printdiff' in dumpsigs: + self.write_diffscenetasks(self.invalidtasks_dump) + self.state = RunQueueState.COMPLETE + + if self.state == RunQueueState.SCENE_INIT: self.start_worker(self.rqexe) self.rqdata.init_progress_reporter.finish() @@ -1597,15 +1624,15 @@ class RunQueue: self.rqexe.tasks_notcovered.add(tid) self.rqexe.sqdone = True logger.info('Executing Tasks') - self.state = runQueueRunning + self.state = RunQueueState.RUNNING - if self.state is runQueueRunning: + if self.state == RunQueueState.RUNNING: retval = self.rqexe.execute() - if self.state is runQueueCleanUp: + if self.state == RunQueueState.CLEAN_UP: retval = self.rqexe.finish() - build_done = self.state is runQueueComplete or self.state is runQueueFailed + build_done = self.state in [RunQueueState.COMPLETE, RunQueueState.FAILED] if build_done and self.dm_event_handler_registered: bb.event.remove(self.dm_event_handler_name, None, data=self.cfgData) @@ -1621,10 +1648,10 @@ class RunQueue: # Let's avoid the word "failed" if nothing actually did logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and all succeeded.", self.rqexe.stats.completed, self.rqexe.stats.skipped) - if self.state is runQueueFailed: + if self.state == RunQueueState.FAILED: raise bb.runqueue.TaskFailure(self.rqexe.failed_tids) - if self.state is runQueueComplete: + if self.state == RunQueueState.COMPLETE: # All done return False @@ -1644,7 +1671,7 @@ class RunQueue: self.teardown_workers() except: pass - self.state = runQueueComplete + self.state = RunQueueState.COMPLETE raise except Exception as err: logger.exception("An uncaught exception occurred in runqueue") @@ -1652,12 +1679,12 @@ class RunQueue: self.teardown_workers() except: pass - self.state = runQueueComplete + self.state = RunQueueState.COMPLETE raise def finish_runqueue(self, now = False): if not self.rqexe: - self.state = runQueueComplete + self.state = RunQueueState.COMPLETE return if now: @@ -1672,33 +1699,42 @@ class RunQueue: bb.parse.siggen.dump_sigtask(taskfn, taskname, dataCaches[mc].stamp[taskfn], True) def dump_signatures(self, options): - if bb.cooker.CookerFeatures.RECIPE_SIGGEN_INFO not in self.cooker.featureset: - bb.fatal("The dump signatures functionality needs the RECIPE_SIGGEN_INFO feature enabled") - - bb.note("Writing task signature files") - - max_process = int(self.cfgData.getVar("BB_NUMBER_PARSE_THREADS") or os.cpu_count() or 1) - def chunkify(l, n): - return [l[i::n] for i in range(n)] - tids = chunkify(list(self.rqdata.runtaskentries), max_process) - # We cannot use the real multiprocessing.Pool easily due to some local data - # that can't be pickled. This is a cheap multi-process solution. - launched = [] - while tids: - if len(launched) < max_process: - p = Process(target=self._rq_dump_sigtid, args=(tids.pop(), )) + if not hasattr(self, "dumpsigs_launched"): + if bb.cooker.CookerFeatures.RECIPE_SIGGEN_INFO not in self.cooker.featureset: + bb.fatal("The dump signatures functionality needs the RECIPE_SIGGEN_INFO feature enabled") + + bb.note("Writing task signature files") + + max_process = int(self.cfgData.getVar("BB_NUMBER_PARSE_THREADS") or os.cpu_count() or 1) + def chunkify(l, n): + return [l[i::n] for i in range(n)] + dumpsigs_tids = chunkify(list(self.rqdata.runtaskentries), max_process) + + # We cannot use the real multiprocessing.Pool easily due to some local data + # that can't be pickled. This is a cheap multi-process solution. + self.dumpsigs_launched = [] + + for tids in dumpsigs_tids: + p = Process(target=self._rq_dump_sigtid, args=(tids, )) p.start() - launched.append(p) - for q in launched: - # The finished processes are joined when calling is_alive() - if not q.is_alive(): - launched.remove(q) - for p in launched: + self.dumpsigs_launched.append(p) + + return 1.0 + + for q in self.dumpsigs_launched: + # The finished processes are joined when calling is_alive() + if not q.is_alive(): + self.dumpsigs_launched.remove(q) + + if self.dumpsigs_launched: + return 1.0 + + for p in self.dumpsigs_launched: p.join() bb.parse.siggen.dump_sigs(self.rqdata.dataCaches, options) - return + return False def print_diffscenetasks(self): def get_root_invalid_tasks(task, taskdepends, valid, noexec, visited_invalid): @@ -1971,14 +2007,14 @@ class RunQueueExecute: pass if self.failed_tids: - self.rq.state = runQueueFailed + self.rq.state = RunQueueState.FAILED return - self.rq.state = runQueueComplete + self.rq.state = RunQueueState.COMPLETE return def finish(self): - self.rq.state = runQueueCleanUp + self.rq.state = RunQueueState.CLEAN_UP active = self.stats.active + len(self.sq_live) if active > 0: @@ -1987,10 +2023,10 @@ class RunQueueExecute: return self.rq.active_fds() if self.failed_tids: - self.rq.state = runQueueFailed + self.rq.state = RunQueueState.FAILED return True - self.rq.state = runQueueComplete + self.rq.state = RunQueueState.COMPLETE return True # Used by setscene only @@ -2109,7 +2145,7 @@ class RunQueueExecute: bb.event.fire(runQueueTaskFailed(task, self.stats, exitcode, self.rq, fakeroot_log=("".join(fakeroot_log) or None)), self.cfgData) if self.rqdata.taskData[''].halt: - self.rq.state = runQueueCleanUp + self.rq.state = RunQueueState.CLEAN_UP def task_skip(self, task, reason): self.runq_running.add(task) @@ -2175,12 +2211,20 @@ class RunQueueExecute: if not hasattr(self, "sorted_setscene_tids"): # Don't want to sort this set every execution self.sorted_setscene_tids = sorted(self.rqdata.runq_setscene_tids) + # Resume looping where we left off when we returned to feed the mainloop + self.setscene_tids_generator = itertools.cycle(self.rqdata.runq_setscene_tids) task = None if not self.sqdone and self.can_start_task(): - # Find the next setscene to run - for nexttask in self.sorted_setscene_tids: + loopcount = 0 + # Find the next setscene to run, exit the loop when we've processed all tids or found something to execute + while loopcount < len(self.rqdata.runq_setscene_tids): + loopcount += 1 + nexttask = next(self.setscene_tids_generator) if nexttask in self.sq_buildable and nexttask not in self.sq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values() and nexttask not in self.sq_harddep_deferred: + if nexttask in self.sq_deferred and self.sq_deferred[nexttask] not in self.runq_complete: + # Skip deferred tasks quickly before the 'expensive' tests below - this is key to performant multiconfig builds + continue if nexttask not in self.sqdata.unskippable and self.sqdata.sq_revdeps[nexttask] and \ nexttask not in self.sq_needed_harddeps and \ self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and \ @@ -2206,12 +2250,10 @@ class RunQueueExecute: return True continue # If covered tasks are running, need to wait for them to complete - for t in self.sqdata.sq_covered_tasks[nexttask]: - if t in self.runq_running and t not in self.runq_complete: - continue + if any(t in self.runq_running and t not in self.runq_complete for t in self.sqdata.sq_covered_tasks[nexttask]): + continue if nexttask in self.sq_deferred: - if self.sq_deferred[nexttask] not in self.runq_complete: - continue + # Deferred tasks that were still deferred were skipped above so we now need to process logger.debug("Task %s no longer deferred" % nexttask) del self.sq_deferred[nexttask] valid = self.rq.validate_hashes(set([nexttask]), self.cooker.data, 0, False, summary=False) @@ -2296,17 +2338,17 @@ class RunQueueExecute: err = self.summarise_scenequeue_errors() if err: - self.rq.state = runQueueFailed + self.rq.state = RunQueueState.FAILED return True if self.cooker.configuration.setsceneonly: - self.rq.state = runQueueComplete + self.rq.state = RunQueueState.COMPLETE return True self.sqdone = True if self.stats.total == 0: # nothing to do - self.rq.state = runQueueComplete + self.rq.state = RunQueueState.COMPLETE return True if self.cooker.configuration.setsceneonly: @@ -2373,7 +2415,7 @@ class RunQueueExecute: self.rq.start_fakeworker(self, mc) except OSError as exc: logger.critical("Failed to spawn fakeroot worker to run %s: %s" % (task, str(exc))) - self.rq.state = runQueueFailed + self.rq.state = RunQueueState.FAILED self.stats.taskFailed() return True RunQueue.send_pickled_data(self.rq.fakeworker[mc].process, runtask, "runtask") @@ -2401,7 +2443,7 @@ class RunQueueExecute: return True if self.failed_tids: - self.rq.state = runQueueFailed + self.rq.state = RunQueueState.FAILED return True # Sanity Checks @@ -2418,9 +2460,9 @@ class RunQueueExecute: err = True if err: - self.rq.state = runQueueFailed + self.rq.state = RunQueueState.FAILED else: - self.rq.state = runQueueComplete + self.rq.state = RunQueueState.COMPLETE return True @@ -2438,14 +2480,17 @@ class RunQueueExecute: taskdepdata_cache = {} for task in self.rqdata.runtaskentries: (mc, fn, taskname, taskfn) = split_tid_mcfn(task) - pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn] - deps = self.rqdata.runtaskentries[task].depends - provides = self.rqdata.dataCaches[mc].fn_provides[taskfn] - taskhash = self.rqdata.runtaskentries[task].hash - unihash = self.rqdata.runtaskentries[task].unihash - deps = self.filtermcdeps(task, mc, deps) - hashfn = self.rqdata.dataCaches[mc].hashfn[taskfn] - taskdepdata_cache[task] = [pn, taskname, fn, deps, provides, taskhash, unihash, hashfn] + taskdepdata_cache[task] = bb.TaskData( + pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn], + taskname = taskname, + fn = fn, + deps = self.filtermcdeps(task, mc, self.rqdata.runtaskentries[task].depends), + provides = self.rqdata.dataCaches[mc].fn_provides[taskfn], + taskhash = self.rqdata.runtaskentries[task].hash, + unihash = self.rqdata.runtaskentries[task].unihash, + hashfn = self.rqdata.dataCaches[mc].hashfn[taskfn], + taskhash_deps = self.rqdata.runtaskentries[task].taskhash_deps, + ) self.taskdepdata_cache = taskdepdata_cache @@ -2460,9 +2505,11 @@ class RunQueueExecute: while next: additional = [] for revdep in next: - self.taskdepdata_cache[revdep][6] = self.rqdata.runtaskentries[revdep].unihash + self.taskdepdata_cache[revdep] = self.taskdepdata_cache[revdep]._replace( + unihash=self.rqdata.runtaskentries[revdep].unihash + ) taskdepdata[revdep] = self.taskdepdata_cache[revdep] - for revdep2 in self.taskdepdata_cache[revdep][3]: + for revdep2 in self.taskdepdata_cache[revdep].deps: if revdep2 not in taskdepdata: additional.append(revdep2) next = additional @@ -2531,9 +2578,6 @@ class RunQueueExecute: self.rqdata.runtaskentries[hashtid].unihash = unihash bb.parse.siggen.set_unihash(hashtid, unihash) toprocess.add(hashtid) - if torehash: - # Need to save after set_unihash above - bb.parse.siggen.save_unitaskhashes() # Work out all tasks which depend upon these total = set() @@ -2556,17 +2600,28 @@ class RunQueueExecute: elif self.rqdata.runtaskentries[p].depends.isdisjoint(total): next.add(p) + starttime = time.time() + lasttime = starttime + # When an item doesn't have dependencies in total, we can process it. Drop items from total when handled while next: current = next.copy() next = set() + ready = {} for tid in current: - if self.rqdata.runtaskentries[p].depends and not self.rqdata.runtaskentries[tid].depends.isdisjoint(total): + if self.rqdata.runtaskentries[tid].depends and not self.rqdata.runtaskentries[tid].depends.isdisjoint(total): continue + # get_taskhash for a given tid *must* be called before get_unihash* below + ready[tid] = bb.parse.siggen.get_taskhash(tid, self.rqdata.runtaskentries[tid].depends, self.rqdata.dataCaches) + + unihashes = bb.parse.siggen.get_unihashes(ready.keys()) + + for tid in ready: orighash = self.rqdata.runtaskentries[tid].hash - newhash = bb.parse.siggen.get_taskhash(tid, self.rqdata.runtaskentries[tid].depends, self.rqdata.dataCaches) + newhash = ready[tid] origuni = self.rqdata.runtaskentries[tid].unihash - newuni = bb.parse.siggen.get_unihash(tid) + newuni = unihashes[tid] + # FIXME, need to check it can come from sstate at all for determinism? remapped = False if newuni == origuni: @@ -2587,6 +2642,15 @@ class RunQueueExecute: next |= self.rqdata.runtaskentries[tid].revdeps total.remove(tid) next.intersection_update(total) + bb.event.check_for_interrupts() + + if time.time() > (lasttime + 30): + lasttime = time.time() + hashequiv_logger.verbose("Rehash loop slow progress: %s in %s" % (len(total), lasttime - starttime)) + + endtime = time.time() + if (endtime-starttime > 60): + hashequiv_logger.verbose("Rehash loop took more than 60s: %s" % (endtime-starttime)) if changed: for mc in self.rq.worker: @@ -2628,7 +2692,7 @@ class RunQueueExecute: if dep in self.runq_complete and dep not in self.runq_tasksrun: bb.error("Task %s marked as completed but now needing to rerun? Halting build." % dep) self.failed_tids.append(tid) - self.rq.state = runQueueCleanUp + self.rq.state = RunQueueState.CLEAN_UP return if dep not in self.runq_complete: @@ -2712,8 +2776,12 @@ class RunQueueExecute: logger.debug2("%s was unavailable and is a hard dependency of %s so skipping" % (task, dep)) self.sq_task_failoutright(dep) continue + + # For performance, only compute allcovered once if needed + if self.sqdata.sq_deps[task]: + allcovered = self.scenequeue_covered | self.scenequeue_notcovered for dep in sorted(self.sqdata.sq_deps[task]): - if self.sqdata.sq_revdeps[dep].issubset(self.scenequeue_covered | self.scenequeue_notcovered): + if self.sqdata.sq_revdeps[dep].issubset(allcovered): if dep not in self.sq_buildable: self.sq_buildable.add(dep) @@ -2759,7 +2827,7 @@ class RunQueueExecute: pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn] if not check_setscene_enforce_ignore_tasks(pn, taskname, self.rqdata.setscene_ignore_tasks): logger.error('Task %s.%s failed' % (pn, taskname + "_setscene")) - self.rq.state = runQueueCleanUp + self.rq.state = RunQueueState.CLEAN_UP def sq_task_complete(self, task): bb.event.fire(sceneQueueTaskCompleted(task, self.stats, self.rq), self.cfgData) @@ -2806,13 +2874,19 @@ class RunQueueExecute: additional = [] for revdep in next: (mc, fn, taskname, taskfn) = split_tid_mcfn(revdep) - pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn] deps = getsetscenedeps(revdep) - provides = self.rqdata.dataCaches[mc].fn_provides[taskfn] - taskhash = self.rqdata.runtaskentries[revdep].hash - unihash = self.rqdata.runtaskentries[revdep].unihash - hashfn = self.rqdata.dataCaches[mc].hashfn[taskfn] - taskdepdata[revdep] = [pn, taskname, fn, deps, provides, taskhash, unihash, hashfn] + + taskdepdata[revdep] = bb.TaskData( + pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn], + taskname = taskname, + fn = fn, + deps = deps, + provides = self.rqdata.dataCaches[mc].fn_provides[taskfn], + taskhash = self.rqdata.runtaskentries[revdep].hash, + unihash = self.rqdata.runtaskentries[revdep].unihash, + hashfn = self.rqdata.dataCaches[mc].hashfn[taskfn], + taskhash_deps = self.rqdata.runtaskentries[revdep].taskhash_deps, + ) for revdep2 in deps: if revdep2 not in taskdepdata: additional.append(revdep2) @@ -2964,14 +3038,13 @@ def build_scenequeue_data(sqdata, rqdata, sqrq): rqdata.init_progress_reporter.next_stage(len(rqdata.runtaskentries)) # Sanity check all dependencies could be changed to setscene task references - for taskcounter, tid in enumerate(rqdata.runtaskentries): + for tid in rqdata.runtaskentries: if tid in rqdata.runq_setscene_tids: pass elif sq_revdeps_squash[tid]: bb.msg.fatal("RunQueue", "Something went badly wrong during scenequeue generation, halting. Please report this problem.") else: del sq_revdeps_squash[tid] - rqdata.init_progress_reporter.update(taskcounter) rqdata.init_progress_reporter.next_stage() @@ -3236,6 +3309,12 @@ class taskUniHashUpdate(bb.event.Event): self.unihash = unihash bb.event.Event.__init__(self) +class PSIEvent(bb.event.Event): + def __init__(self, pressure_state, pressure_values): + super().__init__() + self.pressure_state = pressure_state + self.pressure_values = pressure_values + class runQueuePipe(): """ Abstraction for a pipe between a worker thread and the server @@ -3261,7 +3340,7 @@ class runQueuePipe(): start = len(self.queue) try: - self.queue.extend(self.input.read(102400) or b"") + self.queue.extend(self.input.read(512 * 1024) or b"") except (OSError, IOError) as e: if e.errno != errno.EAGAIN: raise diff --git a/bitbake/lib/bb/server/process.py b/bitbake/lib/bb/server/process.py index 76b18929..d0f73590 100644 --- a/bitbake/lib/bb/server/process.py +++ b/bitbake/lib/bb/server/process.py @@ -13,7 +13,7 @@ import bb import bb.event import logging -import multiprocessing +from bb import multiprocessing import threading import array import os @@ -80,9 +80,6 @@ class idleFinish(): self.msg = msg class ProcessServer(): - profile_filename = "profile.log" - profile_processed_filename = "profile.log.processed" - def __init__(self, lock, lockname, sock, sockname, server_timeout, xmlrpcinterface): self.command_channel = False self.command_channel_reply = False @@ -140,23 +137,7 @@ class ProcessServer(): serverlog("Error writing to lock file: %s" % str(e)) pass - if self.cooker.configuration.profile: - try: - import cProfile as profile - except: - import profile - prof = profile.Profile() - - ret = profile.Profile.runcall(prof, self.main) - - prof.dump_stats("profile.log") - bb.utils.process_profilelog("profile.log") - serverlog("Raw profiling information saved to profile.log and processed statistics to profile.log.processed") - - else: - ret = self.main() - - return ret + return bb.utils.profile_function("main" in self.cooker.configuration.profile, self.main, "profile-mainloop.log") def _idle_check(self): return len(self._idlefuns) == 0 and self.cooker.command.currentAsyncCommand is None @@ -321,7 +302,22 @@ class ProcessServer(): bb.warn('Ignoring invalid BB_SERVER_TIMEOUT=%s, must be a float specifying seconds.' % self.timeout) seendata = True - ready = self.idle_commands(.1, fds) + if not self.idle: + self.idle = threading.Thread(target=self.idle_thread) + self.idle.start() + elif self.idle and not self.idle.is_alive(): + serverlog("Idle thread terminated, main thread exiting too") + bb.error("Idle thread terminated, main thread exiting too") + self.quit = True + + nextsleep = 1.0 + if self.xmlrpc: + nextsleep = self.xmlrpc.get_timeout(nextsleep) + try: + ready = select.select(fds,[],[],nextsleep)[0] + except InterruptedError: + # Ignore EINTR + ready = [] if self.idle: self.idle.join() @@ -402,20 +398,7 @@ class ProcessServer(): serverlog("".join(msg)) def idle_thread(self): - if self.cooker.configuration.profile: - try: - import cProfile as profile - except: - import profile - prof = profile.Profile() - - ret = profile.Profile.runcall(prof, self.idle_thread_internal) - - prof.dump_stats("profile-mainloop.log") - bb.utils.process_profilelog("profile-mainloop.log") - serverlog("Raw profiling information saved to profile-mainloop.log and processed statistics to profile-mainloop.log.processed") - else: - self.idle_thread_internal() + bb.utils.profile_function("idle" in self.cooker.configuration.profile, self.idle_thread_internal, "profile-idleloop.log") def idle_thread_internal(self): def remove_idle_func(function): @@ -424,7 +407,7 @@ class ProcessServer(): self.idle_cond.notify_all() while not self.quit: - nextsleep = 0.1 + nextsleep = 1.0 fds = [] with bb.utils.lock_timeout(self._idlefuncsLock): @@ -462,7 +445,7 @@ class ProcessServer(): # Create new heartbeat event? now = time.time() - if bb.event._heartbeat_enabled and now >= self.next_heartbeat: + if items and bb.event._heartbeat_enabled and now >= self.next_heartbeat: # We might have missed heartbeats. Just trigger once in # that case and continue after the usual delay. self.next_heartbeat += self.heartbeat_seconds @@ -485,31 +468,6 @@ class ProcessServer(): if nextsleep is not None: select.select(fds,[],[],nextsleep)[0] - def idle_commands(self, delay, fds=None): - nextsleep = delay - if not fds: - fds = [] - - if not self.idle: - self.idle = threading.Thread(target=self.idle_thread) - self.idle.start() - elif self.idle and not self.idle.is_alive(): - serverlog("Idle thread terminated, main thread exiting too") - bb.error("Idle thread terminated, main thread exiting too") - self.quit = True - - if nextsleep is not None: - if self.xmlrpc: - nextsleep = self.xmlrpc.get_timeout(nextsleep) - try: - return select.select(fds,[],[],nextsleep)[0] - except InterruptedError: - # Ignore EINTR - return [] - else: - return select.select(fds,[],[],0)[0] - - class ServerCommunicator(): def __init__(self, connection, recv): self.connection = connection @@ -642,7 +600,7 @@ class BitBakeServer(object): os.set_inheritable(self.bitbake_lock.fileno(), True) os.set_inheritable(self.readypipein, True) serverscript = os.path.realpath(os.path.dirname(__file__) + "/../../../bin/bitbake-server") - os.execl(sys.executable, sys.executable, serverscript, "decafbad", str(self.bitbake_lock.fileno()), str(self.readypipein), self.logfile, self.bitbake_lock.name, self.sockname, str(self.server_timeout or 0), str(int(self.profile)), str(self.xmlrpcinterface[0]), str(self.xmlrpcinterface[1])) + os.execl(sys.executable, sys.executable, serverscript, "decafbad", str(self.bitbake_lock.fileno()), str(self.readypipein), self.logfile, self.bitbake_lock.name, self.sockname, str(self.server_timeout or 0), str(list(self.profile)), str(self.xmlrpcinterface[0]), str(self.xmlrpcinterface[1])) def execServer(lockfd, readypipeinfd, lockname, sockname, server_timeout, xmlrpcinterface, profile): diff --git a/bitbake/lib/bb/server/xmlrpcserver.py b/bitbake/lib/bb/server/xmlrpcserver.py index 04b0b17d..ebc271ac 100644 --- a/bitbake/lib/bb/server/xmlrpcserver.py +++ b/bitbake/lib/bb/server/xmlrpcserver.py @@ -14,6 +14,8 @@ from xmlrpc.server import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler import bb.server.xmlrpcclient import bb +import bb.cooker +import bb.event # This request handler checks if the request has a "Bitbake-token" header # field (this comes from the client side) and compares it with its internal @@ -54,7 +56,7 @@ class BitBakeXMLRPCServer(SimpleXMLRPCServer): def __init__(self, interface, cooker, parent): # Use auto port configuration - if (interface[1] == -1): + if interface[1] == -1: interface = (interface[0], 0) SimpleXMLRPCServer.__init__(self, interface, requestHandler=BitBakeXMLRPCRequestHandler, @@ -87,11 +89,12 @@ class BitBakeXMLRPCServer(SimpleXMLRPCServer): def handle_requests(self): self._handle_request_noblock() -class BitBakeXMLRPCServerCommands(): +class BitBakeXMLRPCServerCommands: def __init__(self, server): self.server = server self.has_client = False + self.event_handle = None def registerEventHandler(self, host, port): """ @@ -100,8 +103,8 @@ class BitBakeXMLRPCServerCommands(): s, t = bb.server.xmlrpcclient._create_server(host, port) # we don't allow connections if the cooker is running - if (self.server.cooker.state in [bb.cooker.state.parsing, bb.cooker.state.running]): - return None, "Cooker is busy: %s" % bb.cooker.state.get_name(self.server.cooker.state) + if self.server.cooker.state in [bb.cooker.State.PARSING, bb.cooker.State.RUNNING]: + return None, f"Cooker is busy: {self.server.cooker.state.name}" self.event_handle = bb.event.register_UIHhandler(s, True) return self.event_handle, 'OK' diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py index 8ab08ec9..3a203676 100644 --- a/bitbake/lib/bb/siggen.py +++ b/bitbake/lib/bb/siggen.py @@ -43,6 +43,10 @@ def check_siggen_version(siggen): if siggen.find_siginfo_version < siggen.find_siginfo_minversion: bb.fatal("Siggen from metadata (OE-Core?) is too old, please update it (%s vs %s)" % (siggen.find_siginfo_version, siggen.find_siginfo_minversion)) +def check_hashserv_unihash(unihash): + if not hashserv.is_valid_unihash(unihash): + bb.fatal("Hash Equivalence Server returned invalid unihash") + class SetEncoder(json.JSONEncoder): def default(self, obj): if isinstance(obj, set) or isinstance(obj, frozenset): @@ -201,9 +205,6 @@ class SignatureGenerator(object): def save_unitaskhashes(self): return - def copy_unitaskhashes(self, targetdir): - return - def set_setscene_tasks(self, setscene_tasks): return @@ -381,7 +382,7 @@ class SignatureGeneratorBasic(SignatureGenerator): self.taints[tid] = taint logger.warning("%s is tainted from a forced run" % tid) - return + return set(dep for _, dep in self.runtaskdeps[tid]) def get_taskhash(self, tid, deps, dataCaches): @@ -418,9 +419,6 @@ class SignatureGeneratorBasic(SignatureGenerator): def save_unitaskhashes(self): self.unihash_cache.save(self.unitaskhashes) - def copy_unitaskhashes(self, targetdir): - self.unihash_cache.copyfile(targetdir) - def dump_sigtask(self, mcfn, task, stampbase, runtime): tid = mcfn + ":" + task mc = bb.runqueue.mc_from_tid(mcfn) @@ -540,7 +538,7 @@ class SignatureGeneratorUniHashMixIn(object): def __init__(self, data): self.extramethod = {} # NOTE: The cache only tracks hashes that exist. Hashes that don't - # exist are always queries from the server since it is possible for + # exist are always queried from the server since it is possible for # hashes to appear over time, but much less likely for them to # disappear self.unihash_exists_cache = set() @@ -558,11 +556,11 @@ class SignatureGeneratorUniHashMixIn(object): super().__init__(data) def get_taskdata(self): - return (self.server, self.method, self.extramethod, self.max_parallel, self.username, self.password, self.env) + super().get_taskdata() + return (self.server, self.method, self.extramethod, self.username, self.password, self.env) + super().get_taskdata() def set_taskdata(self, data): - self.server, self.method, self.extramethod, self.max_parallel, self.username, self.password, self.env = data[:7] - super().set_taskdata(data[7:]) + self.server, self.method, self.extramethod, self.username, self.password, self.env = data[:6] + super().set_taskdata(data[6:]) def get_hashserv_creds(self): if self.username and self.password: @@ -595,13 +593,6 @@ class SignatureGeneratorUniHashMixIn(object): self._client = hashserv.create_client(self.server, **self.get_hashserv_creds()) yield self._client - @contextmanager - def client_pool(self): - with self._client_env(): - if getattr(self, '_client_pool', None) is None: - self._client_pool = hashserv.client.ClientPool(self.server, self.max_parallel, **self.get_hashserv_creds()) - yield self._client_pool - def reset(self, data): self.__close_clients() return super().reset(data) @@ -678,25 +669,20 @@ class SignatureGeneratorUniHashMixIn(object): if len(query) == 0: return {} - uncached_query = {} + query_keys = [] result = {} for key, unihash in query.items(): if unihash in self.unihash_exists_cache: result[key] = True else: - uncached_query[key] = unihash + query_keys.append(key) - if self.max_parallel <= 1 or len(uncached_query) <= 1: - # No parallelism required. Make the query serially with the single client + if query_keys: with self.client() as client: - uncached_result = { - key: client.unihash_exists(value) for key, value in uncached_query.items() - } - else: - with self.client_pool() as client_pool: - uncached_result = client_pool.unihashes_exist(uncached_query) + query_result = client.unihash_exists_batch(query[k] for k in query_keys) - for key, exists in uncached_result.items(): + for idx, key in enumerate(query_keys): + exists = query_result[idx] if exists: self.unihash_exists_cache.add(query[key]) result[key] = exists @@ -712,29 +698,24 @@ class SignatureGeneratorUniHashMixIn(object): unihash """ result = {} - queries = {} - query_result = {} + query_tids = [] for tid in tids: unihash = self.get_cached_unihash(tid) if unihash: result[tid] = unihash else: - queries[tid] = (self._get_method(tid), self.taskhash[tid]) - - if len(queries) == 0: - return result + query_tids.append(tid) - if self.max_parallel <= 1 or len(queries) <= 1: - # No parallelism required. Make the query serially with the single client - with self.client() as client: - for tid, args in queries.items(): - query_result[tid] = client.get_unihash(*args) - else: - with self.client_pool() as client_pool: - query_result = client_pool.get_unihashes(queries) + if query_tids: + unihashes = [] + try: + with self.client() as client: + unihashes = client.get_unihash_batch((self._get_method(tid), self.taskhash[tid]) for tid in query_tids) + except (ConnectionError, FileNotFoundError, EOFError) as e: + bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e))) - for tid, unihash in query_result.items(): + for idx, tid in enumerate(query_tids): # In the absence of being able to discover a unique hash from the # server, make it be equivalent to the taskhash. The unique "hash" only # really needs to be a unique string (not even necessarily a hash), but @@ -749,7 +730,10 @@ class SignatureGeneratorUniHashMixIn(object): # to the server, there is a better chance that they will agree on # the unique hash. taskhash = self.taskhash[tid] - if unihash: + + if unihashes and unihashes[idx]: + unihash = unihashes[idx] + check_hashserv_unihash(unihash) # A unique hash equal to the taskhash is not very interesting, # so it is reported it at debug level 2. If they differ, that # is much more interesting, so it is reported at debug level 1 @@ -758,7 +742,6 @@ class SignatureGeneratorUniHashMixIn(object): hashequiv_logger.debug2('No reported unihash for %s:%s from %s' % (tid, taskhash, self.server)) unihash = taskhash - self.set_unihash(tid, unihash) self.unihash[tid] = unihash result[tid] = unihash @@ -769,7 +752,7 @@ class SignatureGeneratorUniHashMixIn(object): import importlib taskhash = d.getVar('BB_TASKHASH') - unihash = d.getVar('BB_UNIHASH') + unihash = d.getVar('BB_UNIHASH', expand=False) report_taskdata = d.getVar('SSTATE_HASHEQUIV_REPORT_TASKDATA') == '1' tempdir = d.getVar('T') mcfn = d.getVar('BB_FILENAME') @@ -831,6 +814,7 @@ class SignatureGeneratorUniHashMixIn(object): data = client.report_unihash(taskhash, method, outhash, unihash, extra_data) new_unihash = data['unihash'] + check_hashserv_unihash(new_unihash) if new_unihash != unihash: hashequiv_logger.debug('Task %s unihash changed %s -> %s by server %s' % (taskhash, unihash, new_unihash, self.server)) @@ -839,7 +823,7 @@ class SignatureGeneratorUniHashMixIn(object): d.setVar('BB_UNIHASH', new_unihash) else: hashequiv_logger.debug('Reported task %s as unihash %s to %s' % (taskhash, unihash, self.server)) - except ConnectionError as e: + except (ConnectionError, FileNotFoundError, EOFError) as e: bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e))) finally: if sigfile: @@ -870,6 +854,7 @@ class SignatureGeneratorUniHashMixIn(object): return False finalunihash = data['unihash'] + check_hashserv_unihash(finalunihash) if finalunihash == current_unihash: hashequiv_logger.verbose('Task %s unihash %s unchanged by server' % (tid, finalunihash)) @@ -881,7 +866,7 @@ class SignatureGeneratorUniHashMixIn(object): # TODO: What to do here? hashequiv_logger.verbose('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash)) - except ConnectionError as e: + except (ConnectionError, FileNotFoundError, EOFError) as e: bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e))) return False @@ -895,13 +880,12 @@ class SignatureGeneratorTestEquivHash(SignatureGeneratorUniHashMixIn, SignatureG super().init_rundepcheck(data) self.server = data.getVar('BB_HASHSERVE') self.method = "sstate_output_hash" - self.max_parallel = 1 def clean_checksum_file_path(file_checksum_tuple): f, cs = file_checksum_tuple if "/./" in f: return "./" + f.split("/./")[1] - return f + return os.path.basename(f) def dump_this_task(outfile, d): import bb.parse @@ -1286,7 +1270,10 @@ def dump_sigfile(a): else: output.append("Unable to compute base hash") - computed_taskhash = calc_taskhash(a_data) - output.append("Computed task hash is %s" % computed_taskhash) + if 'runtaskdeps' in a_data: + computed_taskhash = calc_taskhash(a_data) + output.append("Computed task hash is %s" % computed_taskhash) + else: + output.append("Unable to compute task hash") return output diff --git a/bitbake/lib/bb/taskdata.py b/bitbake/lib/bb/taskdata.py index 66545a65..b40afc02 100644 --- a/bitbake/lib/bb/taskdata.py +++ b/bitbake/lib/bb/taskdata.py @@ -138,15 +138,15 @@ class TaskData: rdependids = set() rdepends = dataCache.rundeps[fn] rrecs = dataCache.runrecs[fn] - rdependlist = [] - rreclist = [] + rdependlist = set() + rreclist = set() for package in rdepends: for rdepend in rdepends[package]: - rdependlist.append(rdepend) + rdependlist.add(rdepend) rdependids.add(rdepend) for package in rrecs: for rdepend in rrecs[package]: - rreclist.append(rdepend) + rreclist.add(rdepend) rdependids.add(rdepend) if rdependlist: logger.debug2("Added runtime dependencies %s for %s", str(rdependlist), fn) diff --git a/bitbake/lib/bb/tests/codeparser.py b/bitbake/lib/bb/tests/codeparser.py index f6585fb3..c0d1362a 100644 --- a/bitbake/lib/bb/tests/codeparser.py +++ b/bitbake/lib/bb/tests/codeparser.py @@ -106,6 +106,46 @@ ${D}${libdir}/pkgconfig/*.pc self.parseExpression("foo=$(echo bar)") self.assertExecs(set(["echo"])) + def test_assign_subshell_expansion_quotes(self): + self.parseExpression('foo="$(echo bar)"') + self.assertExecs(set(["echo"])) + + def test_assign_subshell_expansion_nested(self): + self.parseExpression('foo="$(func1 "$(func2 bar$(func3))")"') + self.assertExecs(set(["func1", "func2", "func3"])) + + def test_assign_subshell_expansion_multiple(self): + self.parseExpression('foo="$(func1 "$(func2)") $(func3)"') + self.assertExecs(set(["func1", "func2", "func3"])) + + def test_assign_subshell_expansion_escaped_quotes(self): + self.parseExpression('foo="\\"fo\\"o$(func1)"') + self.assertExecs(set(["func1"])) + + def test_assign_subshell_expansion_empty(self): + self.parseExpression('foo="bar$()foo"') + self.assertExecs(set()) + + def test_assign_subshell_backticks(self): + self.parseExpression("foo=`echo bar`") + self.assertExecs(set(["echo"])) + + def test_assign_subshell_backticks_quotes(self): + self.parseExpression('foo="`echo bar`"') + self.assertExecs(set(["echo"])) + + def test_assign_subshell_backticks_multiple(self): + self.parseExpression('foo="`func1 bar` `func2`"') + self.assertExecs(set(["func1", "func2"])) + + def test_assign_subshell_backticks_escaped_quotes(self): + self.parseExpression('foo="\\"fo\\"o`func1`"') + self.assertExecs(set(["func1"])) + + def test_assign_subshell_backticks_empty(self): + self.parseExpression('foo="bar``foo"') + self.assertExecs(set()) + def test_shell_unexpanded(self): self.setEmptyVars(["QT_BASE_NAME"]) self.parseExpression('echo "${QT_BASE_NAME}"') diff --git a/bitbake/lib/bb/tests/compression.py b/bitbake/lib/bb/tests/compression.py index 95af3f96..16c297b3 100644 --- a/bitbake/lib/bb/tests/compression.py +++ b/bitbake/lib/bb/tests/compression.py @@ -66,8 +66,8 @@ class CompressionTests(object): class LZ4Tests(CompressionTests, unittest.TestCase): def setUp(self): - if shutil.which("lz4c") is None: - self.skipTest("'lz4c' not found") + if shutil.which("lz4") is None: + self.skipTest("'lz4' not found") super().setUp() @contextlib.contextmanager diff --git a/bitbake/lib/bb/tests/data.py b/bitbake/lib/bb/tests/data.py index cbc7c1ec..49177474 100644 --- a/bitbake/lib/bb/tests/data.py +++ b/bitbake/lib/bb/tests/data.py @@ -11,6 +11,7 @@ import unittest import bb import bb.data import bb.parse +import io import logging import os @@ -405,6 +406,31 @@ class TestOverrides(unittest.TestCase): bb.data.expandKeys(self.d) self.assertEqual(self.d.getVar("VERSION"), "2") + # Test an :append whose override name is only resolved by key expansion + def test_append_in_expanded_override(self): + self.d.setVar("MACHINE", "qemux86") + self.d.setVar("PN", "gizmo") + self.d.setVar("OVERRIDES", "gizmo:pn-gizmo-qemux86") + self.d.setVar("TEST:${PN}", "base") + self.d.setVar("TEST:${PN}:append:pn-gizmo-${MACHINE}", " appended") + bb.data.expandKeys(self.d) + self.assertEqual(self.d.getVar("TEST"), "base appended") + + # Test renaming to a key which is itself not expanded yet, as native.bbclass + # does. The dependent override keys must be left for expandKeys() to rename. + def test_rename_to_unexpanded_key_with_override(self): + self.d.setVar("BPN", "gizmo") + self.d.setVar("PN", "gizmo-native") + self.d.setVar("OVERRIDES", "class-target") + self.d.setVar("TEST:${PN}-lib", "base") + self.d.setVar("TEST:${PN}-lib:class-target", "target") + with LogRecord() as logs: + self.d.renameVar("TEST:${PN}-lib", "TEST:${BPN}-lib-native") + bb.data.expandKeys(self.d) + self.assertFalse(logContains("renameVar with equivalent keys", logs)) + self.assertFalse(logContains("replaces original key", logs)) + self.assertEqual(self.d.getVar("TEST:gizmo-lib-native"), "target") + def test_remove_with_override(self): self.d.setVar("TEST:bar", "testvalue2") self.d.setVar("TEST:some_val", "testvalue3 testvalue5") @@ -450,17 +476,64 @@ class TestFlags(unittest.TestCase): self.d = bb.data.init() self.d.setVar("foo", "value of foo") self.d.setVarFlag("foo", "flag1", "value of flag1") + self.d.setVarFlag("foo", "_defaultval_flag_flag1", "default of flag1") self.d.setVarFlag("foo", "flag2", "value of flag2") + self.d.setVarFlag("foo", "_defaultval_flag_flag2", "default of flag2") + self.d.setVarFlag("foo", "flag3", "value of flag3") + self.d.setVarFlag("foo", "_defaultval_flag_flagnovalue", "default of flagnovalue") def test_setflag(self): self.assertEqual(self.d.getVarFlag("foo", "flag1", False), "value of flag1") self.assertEqual(self.d.getVarFlag("foo", "flag2", False), "value of flag2") + self.assertDictEqual( + self.d.getVarFlags("foo"), + { + "flag1": "value of flag1", + "flag2": "value of flag2", + "flag3": "value of flag3", + "flagnovalue": "default of flagnovalue", + } + ) + self.assertDictEqual( + self.d.getVarFlags("foo", internalflags=True), + { + "_content": "value of foo", + "flag1": "value of flag1", + "flag2": "value of flag2", + "flag3": "value of flag3", + "_defaultval_flag_flag1": "default of flag1", + "_defaultval_flag_flag2": "default of flag2", + "_defaultval_flag_flagnovalue": "default of flagnovalue", + } + ) def test_delflag(self): self.d.delVarFlag("foo", "flag2") + self.d.delVarFlag("foo", "flag3") self.assertEqual(self.d.getVarFlag("foo", "flag1", False), "value of flag1") self.assertEqual(self.d.getVarFlag("foo", "flag2", False), None) - + self.assertDictEqual( + self.d.getVarFlags("foo"), + { + "flag1": "value of flag1", + "flagnovalue": "default of flagnovalue", + } + ) + self.assertDictEqual( + self.d.getVarFlags("foo", internalflags=True), + { + "_content": "value of foo", + "flag1": "value of flag1", + "_defaultval_flag_flag1": "default of flag1", + "_defaultval_flag_flagnovalue": "default of flagnovalue", + } + ) + + def test_delvar(self): + self.d.delVar("foo") + self.assertEqual(self.d.getVarFlag("foo", "flag1", False), None) + self.assertEqual(self.d.getVarFlag("foo", "flag2", False), None) + self.assertEqual(self.d.getVarFlags("foo", internalflags=True), None) class Contains(unittest.TestCase): def setUp(self): @@ -570,4 +643,136 @@ class Serialize(unittest.TestCase): self.assertEqual(newd.getVar('HELLO'), 'world') self.assertEqual(newd.getVarFlag('HELLO', 'other'), 'planet') +class EmitVar(unittest.TestCase): + def setUp(self): + self.d = bb.data.init() + + self.d.setVar("foo", "value of foo") + + self.d.setVar("foo_exported", "value of foo_exported") + self.d.setVarFlag("foo_exported", "export", "1") + + self.d.setVar("foo_unexported", "value of foo_unexported") + self.d.setVarFlag("foo_unexported", "unexport", "1") + + self.d.setVar("test_func", "echo test_func_l1\necho test_func_l2") + self.d.setVarFlag("test_func", "func", "1") + + self.d.setVar("bad_chars", 'a"b\nc`d\ne$f') + + @staticmethod + def get_output(out): + lines = [line for line in out.getvalue().splitlines() + if len(line) == 0 or line[0] != '#' + ] + out.truncate(0) + out.seek(0) + return lines + + def test_simple_variable(self): + out = io.StringIO() + + bb.data.emit_var('foo', out, self.d, all=True) + self.assertEqual(self.get_output(out), ['foo="value of foo"']) + + def test_exported(self): + out = io.StringIO() + + bb.data.emit_var('foo_exported', out, self.d) + self.assertEqual(self.get_output(out), + ['export foo_exported="value of foo_exported"']) + + def test_unexported(self): + out = io.StringIO() + + bb.data.emit_var('foo_unexported', out, self.d) + self.assertEqual(self.get_output(out), ['unset foo_unexported']) + + def test_function(self): + out = io.StringIO() + + bb.data.emit_var('test_func', out, self.d) + self.assertEqual(self.get_output(out), ['test_func() {', + 'echo test_func_l1', + 'echo test_func_l2', + '}']) + + def test_all(self): + out = io.StringIO() + + bb.data.emit_var('foo', out, self.d, all=True) + self.assertEqual(self.get_output(out), ['foo="value of foo"']) + + bb.data.emit_var('foo_exported', out, self.d, all=True) + self.assertEqual(self.get_output(out), + ['export foo_exported="value of foo_exported"']) + bb.data.emit_var('foo_unexported', out, self.d, all=True) + + self.assertEqual(self.get_output(out), ['unset foo_unexported']) + + bb.data.emit_var('test_func', out, self.d, all=True) + self.assertEqual(self.get_output(out), ['test_func() {', + 'echo test_func_l1', + 'echo test_func_l2', + '}']) + + def test_not_all(self): + out = io.StringIO() + + bb.data.emit_var('foo', out, self.d) + self.assertEqual(self.get_output(out), []) + + bb.data.emit_var('foo_exported', out, self.d) + self.assertEqual(self.get_output(out), + ['export foo_exported="value of foo_exported"']) + + bb.data.emit_var('foo_unexported', out, self.d) + self.assertEqual(self.get_output(out), ['unset foo_unexported']) + + bb.data.emit_var('test_func', out, self.d) + self.assertEqual(self.get_output(out), ['test_func() {', + 'echo test_func_l1', + 'echo test_func_l2', + '}']) + + def test_not_set(self): + out = io.StringIO() + + bb.data.emit_var('random_name', out, self.d) + self.assertEqual(self.get_output(out), []) + + def test_bad_chars(self): + out = io.StringIO() + + bb.data.emit_var('bad_chars', out, self.d, all=True) + self.assertEqual(self.get_output(out), ['bad_chars="a\\"b \\', + 'c\\`d \\', + 'e\\$f"']) + +class ExportedVars(unittest.TestCase): + def test_expanded_before_returning(self): + # Called while the variable is set, read once it is gone: the value + # has to be the one from the call, not from the read. + d = bb.data.init() + d.setVar("TESTVAR", "${@os.environ.get('BB_TEST_EXPORT', 'gone')}") + d.setVarFlag("TESTVAR", "export", "1") + + with bb.utils.environment(BB_TEST_EXPORT="present"): + exported = bb.data.exported_vars(d) + + self.assertEqual(dict(exported), {"TESTVAR": "present"}) + self.assertIsInstance(exported, list) + + def test_unexpandable_value_warns_and_is_skipped(self): + # The warning belongs to the call, not to a later iteration. + d = bb.data.init() + d.setVar("TESTVAR", "value") + d.setVarFlag("TESTVAR", "export", "1") + d.setVar("TESTBROKEN", "${@int('not a number')}") + d.setVarFlag("TESTBROKEN", "export", "1") + + with LogRecord() as logs: + exported = bb.data.exported_vars(d) + self.assertEqual(dict(exported), {"TESTVAR": "value"}) + self.assertTrue(logContains("Unable to export ${TESTBROKEN}", logs)) diff --git a/bitbake/lib/bb/tests/fetch-testdata/software/miniupnp/download.php b/bitbake/lib/bb/tests/fetch-testdata/software/miniupnp/download.php new file mode 100644 index 00000000..e27ee134 --- /dev/null +++ b/bitbake/lib/bb/tests/fetch-testdata/software/miniupnp/download.php @@ -0,0 +1,3528 @@ + + + + +MiniUPnP download zone + + + + + + + + + + + +

MiniUPnP Project

+ +

+Home | +Downloads | +Compatibility list | +libnatpmp | +MiniSSDPd | +xchat upnp patch | +Search | +Forum +

+

+English | Français +

+ +
+ + +
+ +

MiniUPnP download zone

+

+Find on this page the source of miniupnp and +some related files. You will also find precompiled binaries +of the UPnP client sample program for windows compiled using +MinGW. There are also Windows +binaries (including python module) automatically built using +AppVeyor. +

+

If you just need one of the software installed on your machine, +you probably don't need to download and compile the source files. +It is very likely that a package/port already exists for +your system/distribution. Refer to your system documentation +to find how to search and install a package/port. +Mac OS X does have port systems too : see +MacPorts or +Homebrew or +Fink. +

+

+The miniupnpc (client) sources have been successfully compiled +under Windows XP/vista/7/10/etc. (using +MinGW, +Mingw-w64 +or Cygwin), +Linux, OpenBSD, FreeBSD, NetBSD, DragonFlyBSD, +Solaris, MacOS X and AmigaOS.
+The Makefile of the client is made for GNU make : +check which version your system have +with the command "make --version". On some systems, such as OpenBSD, +you have to use "gmake". Under Windows with MinGW, GNU make is +called "mingw32-make" and a sligthly modified version of the Makefile +should be used : Makefile.mingw. Run "mingw32make.bat" to compile.
+If you have any compatibility problem, please post on the +forum +or contact me by email. +

+ +

Get miniupnpc under AmigaOS 4 on +OS4Depot. +

+

+Dario Meloni has made a Ruby Gem embedding miniupnpc : +https://rubygems.org/gems/mupnp. +

+

+The python module is available on pypi.org : +pip install miniupnpc. +

+

+The daemon (starting in November 2006) compiles with BSD make under BSD +and Solaris.
+To compile the daemon under linux, use "make -f Makefile.linux"
+To compile for OpenWRT +please read the README.openwrt file, or use the packages +miniupnpc and +miniupnpd. + +
+pfSense users are advised to use the +miniupnpd port available for their system. Recent versions of +pfSense include MiniUPnPd in the base system. +
+For Linksys WRT54G +and WRT54GL owners, +Tarifa firmware +is another alternative to get miniUPnPd running on the router. +

+

+Please read README and +LICENCE files included with the distribution for further informations. +

+

+The MiniUPnP daemon (miniupnpd) is working under +OpenBSD, +NetBSD, +FreeBSD, +DragonFlyBSD, +Mac OS X and +(Open)Solaris +with pf, +with IP Filter or +with ipfw. +The linux version uses either libiptc which permits to access +netfilter +rules inside the kernel the same way as +iptables, or +libnftnl +which is the equivalent for +nftables. +

+ +

Releases are now GPG signed with the key A31ACAAF. +Previous signing key was A5C0863C. +Get it from your favorite +key server.

+ +

REST API

+

You can use the REST API to get the latest releases available:

+ + +

You can help !

+

If you make a package/port for your favorite OS distribution, +inform me so I can upload the package here or add a link to your +repository. +

+ +

Latest files

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
namesizedatecomment
miniupnpc-2.3.2.tar.gz14013705/03/2025 10:31MiniUPnP client release source codechangelogSignature
miniupnpd-2.3.7.tar.gz26532922/06/2024 22:31MiniUPnP daemon release source codechangelogSignature
libnatpmp-20230423.tar.gz2650623/04/2023 11:02latest libnatpmp source codechangelogSignature
minissdpd-1.6.0.tar.gz3907722/10/2022 18:41MiniSSDPd release source codechangelogSignature
upnpc-exe-win32-20220515.zip6950315/05/2022 14:31Windows executablechangelog
minissdpd-1.5.20211105.tar.gz3887004/11/2021 23:34latest MiniSSDPd source codechangelogSignature
miniupnpc-2.1.20201016.tar.gz9768215/10/2020 22:31latest MiniUPnP client source codechangelogSignature
miniupnpd-2.1.20200510.tar.gz24542610/05/2020 18:23latest MiniUPnP daemon source codechangelogSignature
xchat-upnp20110811.patch1032911/08/2011 15:18Patch to add UPnP capabilities to xchatchangelog
minidlna_1.0.21.minissdp1.patch759825/07/2011 14:57Patch for MiniDLNA to use miniSSDPDchangelog
miniupnpc-new20060630.tar.gz1484004/11/2006 18:16João Paulo Barraca version of the upnp clientchangelog
+ +

All files

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
namesizedatecomment
miniupnpc-2.3.2.tar.gz14013705/03/2025 10:31:36 +0000MiniUPnP client release source codeSignature
miniupnpc-2.3.1.tar.gz13949923/02/2025 16:44:16 +0000MiniUPnP client release source codeSignature
miniupnpc-2.3.0.tar.gz10507110/01/2025 23:16:45 +0000MiniUPnP client release source codeSignature
miniupnpd-2.3.7.tar.gz26532922/06/2024 22:31:38 +0000MiniUPnP daemon release source codeSignature
miniupnpc-2.2.8.tar.gz10460308/06/2024 22:13:39 +0000MiniUPnP client release source codeSignature
miniupnpd-2.3.6.tar.gz26301819/03/2024 23:39:51 +0000MiniUPnP daemon release source codeSignature
miniupnpc-2.2.7.tar.gz10425819/03/2024 23:25:18 +0000MiniUPnP client release source codeSignature
miniupnpd-2.3.5.tar.gz26195202/03/2024 11:04:07 +0000MiniUPnP daemon release source codeSignature
miniupnpd-2.3.4.tar.gz26081004/01/2024 00:53:17 +0000MiniUPnP daemon release source codeSignature
miniupnpc-2.2.6.tar.gz10394904/01/2024 00:27:14 +0000MiniUPnP client release source codeSignature
miniupnpc-2.2.5.tar.gz10365411/06/2023 23:14:56 +0000MiniUPnP client release source codeSignature
libnatpmp-20230423.tar.gz2650623/04/2023 11:02:09 +0000libnatpmp source codeSignature
miniupnpd-2.3.3.tar.gz26007917/02/2023 03:07:46 +0000MiniUPnP daemon release source codeSignature
miniupnpd-2.3.2.tar.gz25968619/01/2023 23:18:08 +0000MiniUPnP daemon release source codeSignature
minissdpd-1.6.0.tar.gz3907722/10/2022 18:41:54 +0000MiniSSDPd release source codeSignature
miniupnpc-2.2.4.tar.gz10293221/10/2022 21:01:01 +0000MiniUPnP client release source codeSignature
miniupnpd-2.3.1.tar.gz25805016/10/2022 05:58:44 +0000MiniUPnP daemon release source codeSignature
upnpc-exe-win32-20220515.zip6950315/05/2022 14:31:25 +0000Windows executable
hexchat-2.16.patch814719/03/2022 16:52:05 +0000
miniupnpd-2.3.0.tar.gz25606923/01/2022 00:23:32 +0000MiniUPnP daemon release source codeSignature
minissdpd-1.5.20211105.tar.gz3887004/11/2021 23:34:49 +0000MiniSSDPd source codeSignature
miniupnpc-2.2.3.tar.gz10136028/09/2021 21:43:32 +0000MiniUPnP client release source codeSignature
miniupnpd-2.2.3.tar.gz25475221/08/2021 08:35:13 +0000MiniUPnP daemon release source codeSignature
miniupnpd-2.2.2.tar.gz25064913/05/2021 11:30:11 +0000MiniUPnP daemon release source codeSignature
miniupnpc-2.2.2.tar.gz10000802/03/2021 23:44:52 +0000MiniUPnP client release source codeSignature
miniupnpd-2.2.1.tar.gz25002320/12/2020 18:08:08 +0000MiniUPnP daemon release source codeSignature
miniupnpc-2.2.1.tar.gz9959520/12/2020 18:08:02 +0000MiniUPnP client release source codeSignature
miniupnpc-2.2.0.tar.gz9834809/11/2020 19:51:24 +0000MiniUPnP client release source codeSignature
miniupnpd-2.2.0.tar.gz24985831/10/2020 09:20:59 +0000MiniUPnP daemon release source codeSignature
miniupnpd-2.2.0-RC3.tar.gz24987930/10/2020 21:49:49 +0000MiniUPnP daemon release source codeSignature
miniupnpc-2.1.20201016.tar.gz9768215/10/2020 22:31:09 +0000MiniUPnP client source codeSignature
miniupnpd-2.2.0-RC2.tar.gz24875628/09/2020 21:57:22 +0000MiniUPnP daemon release source codeSignature
miniupnpc-2.1.20200928.tar.gz9650828/09/2020 21:56:09 +0000MiniUPnP client source codeSignature
minissdpd-1.5.20200928.tar.gz3786028/09/2020 21:55:40 +0000MiniSSDPd source codeSignature
miniupnpd-2.2.0-RC1.tar.gz24777206/06/2020 18:34:50 +0000MiniUPnP daemon release source codeSignature
miniupnpd-2.2.0-RC0.tar.gz24550716/05/2020 18:03:17 +0000MiniUPnP daemon release source codeSignature
miniupnpd-2.1.20200510.tar.gz24542610/05/2020 18:23:13 +0000MiniUPnP daemon source codeSignature
miniupnpd-2.1.20200329.tar.gz24372529/03/2020 09:11:02 +0000MiniUPnP daemon source codeSignature
miniupnpc-2.1.20191224.tar.gz9474023/12/2019 23:37:32 +0000MiniUPnP client source codeSignature
miniupnpd-2.1.20191006.tar.gz24325506/10/2019 21:02:31 +0000MiniUPnP daemon source codeSignature
miniupnpd-2.1.20191005.tar.gz24410005/10/2019 21:33:08 +0000MiniUPnP daemon source codeSignature
miniupnpd-2.1.20191003.tar.gz24328702/10/2019 22:23:51 +0000MiniUPnP daemon source codeSignature
miniupnpd-2.1.20190924.tar.gz24100824/09/2019 11:58:15 +0000MiniUPnP daemon source codeSignature
miniupnpd-2.1.20190902.tar.gz24074201/09/2019 23:03:03 +0000MiniUPnP daemon source codeSignature
miniupnpd-2.1.20190824.tar.gz24049024/08/2019 09:21:52 +0000MiniUPnP daemon source codeSignature
minissdpd-1.5.20190824.tar.gz3730024/08/2019 09:17:32 +0000MiniSSDPd source codeSignature
miniupnpc-2.1.20190824.tar.gz9456424/08/2019 09:12:50 +0000MiniUPnP client source codeSignature
miniupnpd-2.1.20190630.tar.gz24046630/06/2019 20:27:38 +0000MiniUPnP daemon source codeSignature
miniupnpd-2.1.20190625.tar.gz24012025/06/2019 21:33:49 +0000MiniUPnP daemon source codeSignature
miniupnpc-2.1.20190625.tar.gz9446125/06/2019 21:33:26 +0000MiniUPnP client source codeSignature
miniupnpd-2.1.20190502.tar.gz23605202/05/2019 17:22:23 +0000MiniUPnP daemon source codeSignature
miniupnpc-2.1.20190408.tar.gz9421608/04/2019 12:50:21 +0000MiniUPnP client source codeSignature
miniupnpd-2.1.20190408.tar.gz23598908/04/2019 12:50:01 +0000MiniUPnP daemon source codeSignature
miniupnpc-2.1.20190403.tar.gz9420403/04/2019 15:41:36 +0000MiniUPnP client source codeSignature
miniupnpd-2.1.20190403.tar.gz23590903/04/2019 15:41:17 +0000MiniUPnP daemon source codeSignature
minissdpd-1.5.20190210.tar.gz3722710/02/2019 15:21:49 +0000MiniSSDPd source codeSignature
miniupnpc-2.1.20190210.tar.gz9412510/02/2019 12:46:09 +0000MiniUPnP client source codeSignature
miniupnpd-2.1.20190210.tar.gz23509310/02/2019 11:20:11 +0000MiniUPnP daemon source codeSignature
miniupnpd-2.1.20180706.tar.gz23367506/07/2018 12:44:24 +0000MiniUPnP daemon source codeSignature
miniupnpd-2.1.tar.gz22545808/05/2018 21:50:32 +0000MiniUPnP daemon release source codeSignature
miniupnpc-2.1.tar.gz9191407/05/2018 11:10:59 +0000MiniUPnP client release source codeSignature
miniupnpd-2.0.20180503.tar.gz22545403/05/2018 08:33:10 +0000MiniUPnP daemon source code
miniupnpc-2.0.20180503.tar.gz8820703/05/2018 08:31:22 +0000MiniUPnP client source code
miniupnpd-2.0.20180422.tar.gz22494222/04/2018 19:48:54 +0000MiniUPnP daemon source code
miniupnpd-2.0.20180412.tar.gz22483112/04/2018 08:16:25 +0000MiniUPnP daemon source code
miniupnpd-2.0.20180410.tar.gz22473610/04/2018 07:58:28 +0000MiniUPnP daemon source code
miniupnpc-2.0.20180410.tar.gz8736310/04/2018 07:52:55 +0000MiniUPnP client source code
miniupnpc-2.0.20180406.tar.gz8737406/04/2018 10:55:21 +0000MiniUPnP client source code
minissdpd-1.5.20180223.tar.gz3617923/02/2018 14:24:07 +0000MiniSSDPd source code
miniupnpc-2.0.20180222.tar.gz8701822/02/2018 15:09:24 +0000MiniUPnP client source code
miniupnpd-2.0.20180222.tar.gz22369722/02/2018 15:09:14 +0000MiniUPnP daemon source code
miniupnpd-2.0.20180203.tar.gz22308403/02/2018 22:34:46 +0000MiniUPnP daemon source code
miniupnpc-2.0.20180203.tar.gz8677203/02/2018 22:34:32 +0000MiniUPnP client source code
minissdpd-1.5.20180203.tar.gz3584803/02/2018 22:33:08 +0000MiniSSDPd source code
miniupnpc-2.0.20171212.tar.gz8660712/12/2017 12:03:38 +0000MiniUPnP client source code
miniupnpd-2.0.20171212.tar.gz22261712/12/2017 12:03:32 +0000MiniUPnP daemon source code
miniupnpc-2.0.20171102.tar.gz8636302/11/2017 17:58:34 +0000MiniUPnP client source code
miniupnpc-2.0.20170509.tar.gz8605509/05/2017 10:14:56 +0000MiniUPnP client source code
miniupnpc-2.0.20170421.tar.gz8598421/04/2017 12:02:26 +0000MiniUPnP client source code
miniupnpd-2.0.20170421.tar.gz21919121/04/2017 12:02:06 +0000MiniUPnP daemon source code
miniupnpd-2.0.20161216.tar.gz21811916/12/2016 09:34:08 +0000MiniUPnP daemon source code
miniupnpc-2.0.20161216.tar.gz8578016/12/2016 09:34:03 +0000MiniUPnP client source code
minissdpd-1.5.20161216.tar.gz3507816/12/2016 09:33:59 +0000MiniSSDPd source code
miniupnpd-2.0.tar.gz21780219/04/2016 21:12:01 +0000MiniUPnP daemon release source codeSignature
miniupnpc-2.0.tar.gz8528719/04/2016 21:07:52 +0000MiniUPnP client release source code
minissdpd-1.5.20160301.tar.gz3482701/03/2016 18:08:23 +0000MiniSSDPd source code
miniupnpd-1.9.20160222.tar.gz21754122/02/2016 10:21:40 +0000MiniUPnP daemon source code
miniupnpd-1.9.20160216.tar.gz21700716/02/2016 12:41:44 +0000MiniUPnP daemon source code
miniupnpd-1.9.20160212.tar.gz21586612/02/2016 15:22:04 +0000MiniUPnP daemon source code
miniupnpd-1.9.20160209.tar.gz21341609/02/2016 09:47:03 +0000MiniUPnP daemon source code
miniupnpc-1.9.20160209.tar.gz8526809/02/2016 09:44:50 +0000MiniUPnP client source code
minissdpd-1.5.20160119.tar.gz3471119/01/2016 13:39:51 +0000MiniSSDPd source code
miniupnpd-1.9.20160113.tar.gz21143713/01/2016 16:03:14 +0000MiniUPnP daemon source code
minissdpd-1.5.tar.gz3440413/01/2016 15:26:53 +0000MiniSSDPd release source code
miniupnpd-1.9.20151212.tar.gz21091212/12/2015 10:06:07 +0000MiniUPnP daemon source code
miniupnpd-1.9.20151118.tar.gz21032218/11/2015 08:59:46 +0000MiniUPnP daemon source code
miniupnpc-1.9.20151026.tar.gz8420826/10/2015 17:07:34 +0000MiniUPnP client source code
miniupnpc-1.9.20151008.tar.gz8353808/10/2015 16:22:28 +0000MiniUPnP client source code
miniupnpd-1.9.20150922.tar.gz20870022/09/2015 10:21:50 +0000MiniUPnP daemon source code
upnpc-exe-win32-20150918.zip10000418/09/2015 12:50:51 +0000Windows executable
miniupnpc-1.9.20150917.tar.gz8260917/09/2015 14:09:14 +0000MiniUPnP client source code
upnpc-exe-win32-20150824.zip9952024/08/2015 15:25:18 +0000Windows executable
minissdpd-1.4.tar.gz3201706/08/2015 13:38:37 +0000MiniSSDPd release source code
miniupnpc-1.9.20150730.tar.gz8143129/07/2015 22:10:00 +0000MiniUPnP client source code
miniupnpd-1.9.20150721.tar.gz20756221/07/2015 13:35:51 +0000MiniUPnP daemon source code
miniupnpc-1.9.20150721.tar.gz8052121/07/2015 13:27:00 +0000MiniUPnP client source code
libnatpmp-20150609.tar.gz2439209/06/2015 15:40:28 +0000libnatpmp source code
miniupnpc-1.9.20150609.tar.gz7931109/06/2015 15:39:48 +0000MiniUPnP client source code
miniupnpd-1.9.20150609.tar.gz20708809/06/2015 15:39:36 +0000MiniUPnP daemon source code
minissdpd-1.3.20150527.tar.gz3102527/05/2015 09:17:15 +0000MiniSSDPd source code
miniupnpc-1.9.20150522.tar.gz7908022/05/2015 11:02:27 +0000MiniUPnP client source code
minissdpd-1.3.20150522.tar.gz3033422/05/2015 11:02:04 +0000MiniSSDPd source code
miniupnpd-1.9.20150430.tar.gz20593030/04/2015 09:09:27 +0000MiniUPnP daemon source code
miniupnpc-1.9.20150430.tar.gz7845930/04/2015 08:39:31 +0000MiniUPnP client source code
miniupnpc-1.9.20150427.tar.gz7842427/04/2015 16:08:42 +0000MiniUPnP client source code
miniupnpd-1.9.20150427.tar.gz19115727/04/2015 16:08:27 +0000MiniUPnP daemon source code
miniupnpd-1.9.20150307.tar.gz19091307/03/2015 16:11:51 +0000MiniUPnP daemon source code
miniupnpc-1.9.20150206.tar.gz7686406/02/2015 14:38:00 +0000MiniUPnP client source code
miniupnpd-1.9.20141209.tar.gz19318309/12/2014 09:58:34 +0000MiniUPnP daemon source code
minissdpd-1.3.tar.gz3032609/12/2014 09:57:30 +0000MiniSSDPd release source code
minissdpd-1.2.20141204.tar.gz2697804/12/2014 10:55:26 +0000MiniSSDPd source code
miniupnpd-1.9.20141204.tar.gz19259704/12/2014 10:55:03 +0000MiniUPnP daemon source code
minissdpd-1.2.20141128.tar.gz2679528/11/2014 16:33:10 +0000MiniSSDPd source code
miniupnpd-1.9.20141128.tar.gz19255828/11/2014 13:31:36 +0000MiniUPnP daemon source code
miniupnpc-1.9.20141128.tar.gz7654128/11/2014 13:31:15 +0000MiniUPnP client source code
miniupnpc-1.9.20141117.tar.gz7386517/11/2014 09:51:36 +0000MiniUPnP client source code
miniupnpc-1.9.20141113.tar.gz7285713/11/2014 10:36:44 +0000MiniUPnP client source code
minissdpd-1.2.20141108.tar.gz2200108/11/2014 13:55:41 +0000MiniSSDPd source code
miniupnpc-1.9.20141108.tar.gz7278108/11/2014 13:53:48 +0000MiniUPnP client source code
miniupnpd-1.9.20141108.tar.gz19241308/11/2014 13:53:38 +0000MiniUPnP daemon source code
miniupnpd-1.9.tar.gz19218327/10/2014 16:45:34 +0000MiniUPnP daemon release source code
miniupnpc-1.9.20141027.tar.gz7676327/10/2014 16:45:25 +0000MiniUPnP client source code
miniupnpd-1.8.20141022.tar.gz19163022/10/2014 09:17:41 +0000MiniUPnP daemon source code
miniupnpd-1.8.20141021.tar.gz19127021/10/2014 14:18:58 +0000MiniUPnP daemon source code
miniupnpc-1.9.20140911.tar.gz7685511/09/2014 14:15:23 +0000MiniUPnP client source code
minissdpd-1.2.20140906.tar.gz2195606/09/2014 08:34:10 +0000MiniSSDPd source code
miniupnpd-1.8.20140906.tar.gz19118306/09/2014 08:34:02 +0000MiniUPnP daemon source code
miniupnpc-1.9.20140906.tar.gz7679106/09/2014 08:33:45 +0000MiniUPnP client source code
miniupnpc-1.9.20140701.tar.gz7673501/07/2014 13:06:51 +0000MiniUPnP client source code
miniupnpc-1.9.20140610.tar.gz7667410/06/2014 10:28:27 +0000MiniUPnP client source code
minissdpd-1.2.20140610.tar.gz2190910/06/2014 10:03:29 +0000MiniSSDPd source code
miniupnpd-1.8.20140523.tar.gz19093623/05/2014 15:48:03 +0000MiniUPnP daemon source code
upnpc-exe-win32-20140422.zip9750522/04/2014 10:10:07 +0000Windows executable
miniupnpd-1.8.20140422.tar.gz18722522/04/2014 08:58:56 +0000MiniUPnP daemon source code
miniupnpd-1.8.20140401.tar.gz18313101/04/2014 10:07:20 +0000MiniUPnP daemon source code
miniupnpc-1.9.20140401.tar.gz7470301/04/2014 09:49:46 +0000MiniUPnP client source code
libnatpmp-20140401.tar.gz2330201/04/2014 09:49:44 +0000libnatpmp source code
miniupnpd-1.8.20140313.tar.gz17712013/03/2014 10:39:11 +0000MiniUPnP daemon source code
miniupnpd-1.8.20140310.tar.gz17658509/03/2014 23:16:49 +0000MiniUPnP daemon source code
miniupnpd-1.8.20140225.tar.gz17518325/02/2014 11:01:29 +0000MiniUPnP daemon source code
miniupnpd-1.8.20140203.tar.gz17011203/02/2014 09:56:05 +0000MiniUPnP daemon source code
miniupnpc-1.9.tar.gz7423031/01/2014 13:57:40 +0000MiniUPnP client release source code
miniupnpd-1.8.20140127.tar.gz17046727/01/2014 11:25:34 +0000MiniUPnP daemon source code
upnpc-exe-win32-20140117.zip9727017/01/2014 11:37:53 +0000Windows executable
miniupnpd-1.8.20131216.tar.gz17027716/12/2013 16:15:40 +0000MiniUPnP daemon source code
miniupnpd-1.8.20131213.tar.gz16975313/12/2013 16:18:10 +0000MiniUPnP daemon source code
miniupnpc-1.8.20131209.tar.gz7390009/12/2013 20:52:54 +0000MiniUPnP client source code
libnatpmp-20131126.tar.gz2297226/11/2013 08:51:36 +0000libnatpmp source code
miniupnpc-1.8.20131007.tar.gz7375007/10/2013 10:10:25 +0000MiniUPnP client source code
libnatpmp-20130911.tar.gz1874411/09/2013 07:35:51 +0000libnatpmp source code
libnatpmp-20130910.tar.gz1873410/09/2013 20:15:34 +0000libnatpmp source code
minissdpd-1.2.20130907.tar.gz2023707/09/2013 06:46:31 +0000MiniSSDPd source code
minissdpd-1.2.20130819.tar.gz2077219/08/2013 16:50:29 +0000MiniSSDPd source code
miniupnpc-1.8.20130801.tar.gz7342601/08/2013 21:38:05 +0000MiniUPnP client source code
miniupnpd-1.8.20130730.tar.gz14990430/07/2013 11:37:48 +0000MiniUPnP daemon source code
miniupnpd-1.8.20130607.tar.gz14952107/06/2013 08:46:17 +0000MiniUPnP daemon source code
miniupnpd-1.8.20130521.tar.gz14927621/05/2013 09:01:33 +0000MiniUPnP daemon source code
miniupnpd-1.8.20130503.tar.gz14842003/05/2013 19:27:16 +0000MiniUPnP daemon source code
miniupnpc-1.8.20130503.tar.gz7185803/05/2013 19:27:07 +0000MiniUPnP client source code
miniupnpd-1.8.20130426.tar.gz14789026/04/2013 16:57:20 +0000MiniUPnP daemon source code
miniupnpc-1.8.20130211.tar.gz7072311/02/2013 10:32:44 +0000MiniUPnP client source code
miniupnpd-1.8.20130207.tar.gz14732507/02/2013 12:29:32 +0000MiniUPnP daemon source code
miniupnpc-1.8.tar.gz7062406/02/2013 14:31:06 +0000MiniUPnP client release source code
miniupnpd-1.8.tar.gz14667906/02/2013 14:30:59 +0000MiniUPnP daemon release source code
upnpc-exe-win32-20121009.zip9651309/10/2012 17:54:12 +0000Windows executable
miniupnpd-1.7.20121005.tar.gz14439304/10/2012 22:39:05 +0000MiniUPnP daemon source code
miniupnpc-1.7.20120830.tar.gz7007430/08/2012 08:41:51 +0000MiniUPnP client source code
miniupnpd-1.7.20120824.tar.gz14196024/08/2012 18:15:01 +0000MiniUPnP daemon source code
libnatpmp-20120821.tar.gz1783221/08/2012 17:24:46 +0000libnatpmp source code
miniupnpc-1.7.20120714.tar.gz6957014/07/2012 14:40:47 +0000MiniUPnP client source code
miniupnpc-1.7.20120711.tar.gz6958010/07/2012 22:27:05 +0000MiniUPnP client source code
miniupnpd-1.7.20120711.tar.gz14138010/07/2012 22:26:58 +0000MiniUPnP daemon source code
miniupnpd-1.7.tar.gz13804727/05/2012 23:13:30 +0000MiniUPnP daemon release source code
miniupnpc-1.7.tar.gz6832724/05/2012 18:17:48 +0000MiniUPnP client release source code
minissdpd-1.2.tar.gz1987424/05/2012 18:06:24 +0000MiniSSDPd release source code
miniupnpd-1.6.20120509.tar.gz13714709/05/2012 10:45:44 +0000MiniUPnP daemon source code
miniupnpc-1.6.20120509.tar.gz6820509/05/2012 10:45:41 +0000MiniUPnP client source code
minissdpd-1.1.20120509.tar.gz1812309/05/2012 10:45:39 +0000MiniSSDPd source code
miniupnpd-1.6.20120502.tar.gz13668801/05/2012 22:51:18 +0000MiniUPnP daemon source code
miniupnpc-1.6.20120502.tar.gz6817001/05/2012 22:51:11 +0000MiniUPnP client source code
miniupnpd-1.6.20120426.tar.gz13476426/04/2012 16:24:29 +0000MiniUPnP daemon source code
miniupnpd-1.6.20120424.tar.gz13252223/04/2012 22:43:17 +0000MiniUPnP daemon source code
miniupnpc-1.6.20120424.tar.gz6806723/04/2012 22:43:10 +0000MiniUPnP client source code
miniupnpd-1.6.20120420.tar.gz13197220/04/2012 14:58:57 +0000MiniUPnP daemon source code
miniupnpc-1.6.20120420.tar.gz6806820/04/2012 14:58:39 +0000MiniUPnP client source code
miniupnpd-1.6.20120419.tar.gz13108818/04/2012 23:41:36 +0000MiniUPnP daemon source code
miniupnpd-1.6.20120418.tar.gz13087918/04/2012 21:01:10 +0000MiniUPnP daemon source code
minissdpd-1.1.20120410.tar.gz1805909/04/2012 22:45:38 +0000MiniSSDPd source code
miniupnpc-1.6.20120410.tar.gz6793409/04/2012 22:45:10 +0000MiniUPnP client source code
miniupnpd-1.6.20120406.tar.gz12899206/04/2012 17:52:57 +0000MiniUPnP daemon source code
miniupnpc-1.6.20120320.tar.gz6737420/03/2012 16:55:48 +0000MiniUPnP client source code
miniupnpd-1.6.20120320.tar.gz12796820/03/2012 16:46:07 +0000MiniUPnP daemon source code
miniupnpd-1.6.20120305.tar.gz12698505/03/2012 20:42:01 +0000MiniUPnP daemon source code
miniupnpd-1.6.20120207.tar.gz12742507/02/2012 10:21:16 +0000MiniUPnP daemon source code
miniupnpd-1.6.20120203.tar.gz12659903/02/2012 15:14:13 +0000MiniUPnP daemon source code
miniupnpc-1.6.20120125.tar.gz6735425/01/2012 21:12:28 +0000MiniUPnP client source code
miniupnpc-1.6.20120121.tar.gz6734721/01/2012 14:07:41 +0000MiniUPnP client source code
miniupnpd-1.6.20120121.tar.gz12602121/01/2012 14:07:33 +0000MiniUPnP daemon source code
minissdpd-1.1.20120121.tar.gz1776221/01/2012 14:07:16 +0000MiniSSDPd source code
upnpc-exe-win32-20120121.zip9457521/01/2012 13:59:11 +0000Windows executable
upnpc-exe-win32-20111212.zip9450712/12/2011 12:33:48 +0000Windows executable
miniupnpd-1.6.20111118.tar.gz12568318/11/2011 11:26:12 +0000MiniUPnP daemon source code
minissdpd-1.1.20111007.tar.gz1761107/10/2011 09:47:51 +0000MiniSSDPd source code
xchat-upnp20110811.patch1032911/08/2011 15:18:25 +0000Patch to add UPnP capabilities to xchat
xchat-upnp20110811-2.8.8.patch1152911/08/2011 15:18:23 +0000Patch to add UPnP capabilities to xchat
libnatpmp-20110808.tar.gz1776208/08/2011 21:21:34 +0000libnatpmp source code
libnatpmp-20110730.tar.gz1768730/07/2011 13:19:31 +0000libnatpmp source code
minissdpd-1.1.tar.gz1748130/07/2011 13:17:30 +0000MiniSSDPd release source code
miniupnpd-1.6.20110730.tar.gz12558330/07/2011 13:17:09 +0000MiniUPnP daemon source code
minissdpd-1.0.20110729.tar.gz1589829/07/2011 08:47:26 +0000MiniSSDPd source code
miniupnpc-1.6.tar.gz6645425/07/2011 18:03:09 +0000MiniUPnP client release source code
miniupnpd-1.6.tar.gz12491725/07/2011 16:37:57 +0000MiniUPnP daemon release source code
minidlna_1.0.21.minissdp1.patch759825/07/2011 14:57:50 +0000Patch for MiniDLNA to use miniSSDPD
libnatpmp-20110715.tar.gz1794315/07/2011 08:31:40 +0000libnatpmp source code
miniupnpd-1.5.20110715.tar.gz12451915/07/2011 07:55:17 +0000MiniUPnP daemon source code
upnpc-exe-win32-20110714.zip9423613/07/2011 23:16:01 +0000Windows executable
miniupnpd-1.5.20110623.tar.gz12352922/06/2011 22:29:15 +0000MiniUPnP daemon source code
miniupnpd-1.5.20110620.tar.gz12322120/06/2011 14:11:11 +0000MiniUPnP daemon source code
miniupnpd-1.5.20110618.tar.gz12317617/06/2011 23:29:18 +0000MiniUPnP daemon source code
miniupnpc-1.5.20110618.tar.gz6640117/06/2011 23:29:17 +0000MiniUPnP client source code
libnatpmp-20110618.tar.gz1790117/06/2011 23:29:16 +0000libnatpmp source code
minissdpd-1.0.20110618.tar.gz1519317/06/2011 23:29:16 +0000MiniSSDPd source code
minidlna_cvs20110529_minissdp1.patch29/05/2011 21:19:09 +0000Patch for MiniDLNA to use miniSSDPD
miniupnpd-1.5.20110528.tar.gz12198528/05/2011 09:39:04 +0000MiniUPnP daemon source code
minidlna_1.0.19_minissdp1.patch908027/05/2011 09:55:04 +0000Patch for MiniDLNA to use miniSSDPD
miniupnpd-1.5.20110527.tar.gz12089627/05/2011 08:28:35 +0000MiniUPnP daemon source code
miniupnpc-1.5.20110527.tar.gz6627927/05/2011 08:28:34 +0000MiniUPnP client source code
libnatpmp-20110527.tar.gz1762727/05/2011 08:28:33 +0000libnatpmp source code
minissdpd-1.0.20110523.tar.gz1502423/05/2011 12:55:31 +0000MiniSSDPd source code
miniupnpd-1.5.20110520.tar.gz11922720/05/2011 18:00:41 +0000MiniUPnP daemon source code
miniupnpd-1.5.20110519.tar.gz11473518/05/2011 22:29:06 +0000MiniUPnP daemon source code
miniupnpd-1.5.20110516.tar.gz11334816/05/2011 09:32:51 +0000MiniUPnP daemon source code
miniupnpd-1.5.20110515.tar.gz11313515/05/2011 21:51:29 +0000MiniUPnP daemon source code
miniupnpc-1.5.20110515.tar.gz6611215/05/2011 21:51:28 +0000MiniUPnP client source code
miniupnpd-1.5.20110513.tar.gz11102913/05/2011 14:03:12 +0000MiniUPnP daemon source code
miniupnpc-1.5.20110506.tar.gz6553606/05/2011 16:35:38 +0000MiniUPnP client source code
miniupnpc-1.4-v6.20100505.zip9183318/04/2011 20:14:11 +0000
miniupnpd-1.4-v6.20100823.zip22223518/04/2011 20:14:07 +0000
miniupnpc-1.5.20110418.tar.gz6182018/04/2011 20:09:22 +0000MiniUPnP client source code
upnpc-exe-win32-20110418.zip9418318/04/2011 17:53:26 +0000Windows executable
miniupnpc-1.5.20110314.tar.gz5721014/03/2011 14:27:29 +0000MiniUPnP client source code
miniupnpd-1.5.20110309.tar.gz10007309/03/2011 15:36:12 +0000MiniUPnP daemon source code
miniupnpd-1.5.20110302.tar.gz10075602/03/2011 16:17:44 +0000MiniUPnP daemon source code
miniupnpd-1.5.20110221.tar.gz10009220/02/2011 23:48:17 +0000MiniUPnP daemon source code
upnpc-exe-win32-20110215.zip5540915/02/2011 23:05:00 +0000Windows executable
miniupnpc-1.5.20110215.tar.gz5488015/02/2011 11:16:04 +0000MiniUPnP client source code
miniupnpd-1.5.20110214.tar.gz9962914/02/2011 18:00:43 +0000MiniUPnP daemon source code
minidlna_1.0.18_minissdp1.patch974702/02/2011 15:12:19 +0000Patch for MiniDLNA to use miniSSDPD
miniupnpd-1.5.20110127.tar.gz9742127/01/2011 17:51:25 +0000MiniUPnP daemon source code
miniupnpd-1.5.tar.gz9899304/01/2011 09:45:10 +0000MiniUPnP daemon release source code
miniupnpc-1.5.tar.gz5330904/01/2011 09:45:06 +0000MiniUPnP client release source code
libnatpmp-20110103.tar.gz1752903/01/2011 17:33:16 +0000libnatpmp source code
miniupnpc-1.4.20101221.tar.gz5234221/12/2010 16:15:38 +0000MiniUPnP client source code
upnpc-exe-win32-20101213.zip5235912/12/2010 23:44:01 +0000Windows executable
libnatpmp-20101211.tar.gz1732411/12/2010 17:20:36 +0000libnatpmp source code
miniupnpc-1.4.20101209.tar.gz5190009/12/2010 16:17:30 +0000MiniUPnP client source code
miniupnpd-1.4.20100921.tar.gz9548321/09/2010 15:50:00 +0000MiniUPnP daemon source code
upnpc-exe-win32-20100825.zip5063625/08/2010 08:42:59 +0000Windows executable
miniupnpc-1.4.20100609.tar.gz5039009/06/2010 11:03:11 +0000MiniUPnP client source code
upnpc-exe-win32-20100513.zip5095013/05/2010 16:54:33 +0000Windows executable
miniupnpd-1.4.20100511.tar.gz9328111/05/2010 16:22:33 +0000MiniUPnP daemon source code
upnpc-exe-win32-20100418.zip4075817/04/2010 23:00:37 +0000Windows executable
miniupnpc-1.4.20100418.tar.gz5024517/04/2010 22:18:31 +0000MiniUPnP client source code
miniupnpc-1.4.20100412.tar.gz5014512/04/2010 20:42:53 +0000MiniUPnP client source code
miniupnpc-1.4.20100407.tar.gz4975607/04/2010 10:05:08 +0000MiniUPnP client source code
miniupnpc-1.4.20100405.tar.gz4954905/04/2010 14:34:38 +0000MiniUPnP client source code
miniupnpd-1.4.20100308.tar.gz9288908/03/2010 17:18:00 +0000MiniUPnP daemon source code
libnatpmp-20100202.tar.gz1723102/02/2010 18:41:13 +0000libnatpmp source code
miniupnpc-1.4.20100202.tar.gz4671002/02/2010 18:41:13 +0000MiniUPnP client source code
miniupnpc-1.4.20100106.tar.gz4665906/01/2010 10:08:21 +0000MiniUPnP client source code
miniupnpd-1.4.20091222.tar.gz9099322/12/2009 17:23:48 +0000MiniUPnP daemon source code
libnatpmp-20091219.tar.gz1683919/12/2009 14:35:22 +0000libnatpmp source code
miniupnpc-1.4.20091213.tar.gz4651012/12/2009 23:05:40 +0000MiniUPnP client source code
miniupnpc-1.4.20091211.tar.gz4585211/12/2009 16:43:01 +0000MiniUPnP client source code
upnpc-exe-win32-20091210.zip3866610/12/2009 18:50:27 +0000Windows executable
miniupnpc-1.4.20091208.tar.gz4339208/12/2009 10:58:26 +0000MiniUPnP client source code
miniupnpc-1.4.20091203.tar.gz4204003/12/2009 13:56:28 +0000MiniUPnP client source code
miniupnpd-1.4.20091106.tar.gz9078706/11/2009 21:18:50 +0000MiniUPnP daemon source code
miniupnpd-1.4.tar.gz9007130/10/2009 09:20:05 +0000MiniUPnP daemon release source code
miniupnpc-1.4.tar.gz4179030/10/2009 09:20:04 +0000MiniUPnP client release source code
miniupnpc-20091016.tar.gz4179216/10/2009 09:04:35 +0000MiniUPnP client source code
miniupnpd-20091010.tar.gz9004310/10/2009 19:21:30 +0000MiniUPnP daemon source code
miniupnpc-20091010.tar.gz4167110/10/2009 19:21:28 +0000MiniUPnP client source code
miniupnpd-20090921.tar.gz8947621/09/2009 13:00:04 +0000MiniUPnP daemon source code
miniupnpc-20090921.tar.gz4163021/09/2009 13:00:03 +0000MiniUPnP client source code
miniupnpd-20090904.tar.gz8934404/09/2009 16:24:26 +0000MiniUPnP daemon source code
miniupnpd-20090820.tar.gz8914920/08/2009 09:35:58 +0000MiniUPnP daemon source code
miniupnpc-20090807.tar.gz4128807/08/2009 14:46:11 +0000MiniUPnP client source code
miniupnpc-20090729.tar.gz4058829/07/2009 08:47:43 +0000MiniUPnP client source code
xchat-upnp20061022.patch1025817/07/2009 15:49:46 +0000Patch to add UPnP capabilities to xchat
miniupnpc-20090713.tar.gz4020613/07/2009 08:53:49 +0000MiniUPnP client source code
libnatpmp-20090713.tar.gz1426213/07/2009 08:53:49 +0000libnatpmp source code
miniupnpd-20090605.tar.gz8377404/06/2009 23:32:20 +0000MiniUPnP daemon source code
miniupnpc-20090605.tar.gz4007704/06/2009 23:32:16 +0000MiniUPnP client source code
libnatpmp-20090605.tar.gz1381704/06/2009 23:32:15 +0000libnatpmp source code
miniupnpd-20090516.tar.gz8368916/05/2009 08:47:31 +0000MiniUPnP daemon source code
miniupnpc-1.3.tar.gz4005817/04/2009 21:27:55 +0000MiniUPnP client release source code
miniupnpd-1.3.tar.gz8346417/04/2009 20:11:21 +0000MiniUPnP daemon release source code
libnatpmp-20090310.tar.gz1184710/03/2009 10:19:45 +0000libnatpmp source code
miniupnpd-20090214.tar.gz8292114/02/2009 11:27:03 +0000MiniUPnP daemon source code
miniupnpd-20090213.tar.gz8259413/02/2009 19:48:01 +0000MiniUPnP daemon source code
libnatpmp-20090129.tar.gz1174829/01/2009 21:50:31 +0000libnatpmp source code
miniupnpc-20090129.tar.gz3997629/01/2009 21:50:30 +0000MiniUPnP client source code
miniupnpd-20090129.tar.gz8248729/01/2009 21:50:27 +0000MiniUPnP daemon source code
miniupnpd-20081009.tar.gz8173209/10/2008 12:53:02 +0000MiniUPnP daemon source code
minissdpd-1.0.tar.gz1299607/10/2008 14:03:49 +0000MiniSSDPd release source code
miniupnpc-1.2.tar.gz3878707/10/2008 14:03:47 +0000MiniUPnP client release source code
miniupnpd-1.2.tar.gz8102507/10/2008 14:03:45 +0000MiniUPnP daemon release source code
miniupnpd-20081006.tar.gz8051006/10/2008 15:50:34 +0000MiniUPnP daemon source code
minissdpd-20081006.tar.gz1223006/10/2008 15:50:33 +0000MiniSSDPd source code
libnatpmp-20081006.tar.gz1171006/10/2008 15:50:31 +0000libnatpmp source code
mediatomb_minissdp-20081006.patch06/10/2008 15:48:18 +0000
miniupnpc-20081002.tar.gz3829102/10/2008 09:20:18 +0000MiniUPnP client source code
miniupnpd-20081001.tar.gz7969601/10/2008 13:11:20 +0000MiniUPnP daemon source code
upnpc-exe-win32-20080925.zip3660225/09/2008 06:59:33 +0000Windows executable
miniupnpd-20080710.tar.gz7889810/07/2008 09:38:41 +0000MiniUPnP daemon source code
libnatpmp-20080707.tar.gz1167906/07/2008 22:05:23 +0000libnatpmp source code
miniupnpc-1.1.tar.gz3823504/07/2008 16:45:24 +0000MiniUPnP client release source code
miniupnpc-20080703.tar.gz3820403/07/2008 15:47:37 +0000MiniUPnP client source code
libnatpmp-20080703.tar.gz1157003/07/2008 15:47:25 +0000libnatpmp source code
upnpc-exe-win32-20080703.zip3613702/07/2008 23:35:14 +0000Windows executable
libnatpmp-20080702.tar.gz887302/07/2008 17:32:35 +0000libnatpmp source code
libnatpmp-20080630.tar.gz886430/06/2008 14:20:16 +0000libnatpmp source code
libnatpmp-20080529.tar.gz739729/05/2008 09:06:25 +0000libnatpmp source code
upnpc-exe-win32-20080514.zip1422714/05/2008 20:23:19 +0000Windows executable
libnatpmp-20080428.tar.gz729528/04/2008 03:09:14 +0000libnatpmp source code
miniupnpd-20080427.tar.gz7876527/04/2008 18:16:36 +0000MiniUPnP daemon source code
miniupnpc-20080427.tar.gz3761027/04/2008 18:16:35 +0000MiniUPnP client source code
miniupnpd-1.1.tar.gz7859425/04/2008 17:38:05 +0000MiniUPnP daemon release source code
miniupnpc-20080423.tar.gz3681823/04/2008 11:57:36 +0000MiniUPnP client source code
miniupnpd-20080308.tar.gz7567908/03/2008 11:13:29 +0000MiniUPnP daemon source code
miniupnpd-20080303.tar.gz7420203/03/2008 01:43:16 +0000MiniUPnP daemon source code
miniupnpd-20080224.tar.gz7277324/02/2008 11:23:17 +0000MiniUPnP daemon source code
miniupnpc-1.0.tar.gz3622321/02/2008 13:26:46 +0000MiniUPnP client release source code
miniupnpd-20080221.tar.gz7082321/02/2008 10:23:46 +0000MiniUPnP daemon source code
miniupnpc-20080217.tar.gz3524316/02/2008 23:47:59 +0000MiniUPnP client source code
miniupnpd-20080207.tar.gz7064707/02/2008 21:21:00 +0000MiniUPnP daemon source code
miniupnpc-20080203.tar.gz3492103/02/2008 22:28:11 +0000MiniUPnP client source code
miniupnpd-1.0.tar.gz6942727/01/2008 22:41:25 +0000MiniUPnP daemon release source code
upnpc-exe-win32-20080118.zip1358218/01/2008 11:42:16 +0000Windows executable
miniupnpd-1.0-RC13.tar.gz6789203/01/2008 16:50:21 +0000MiniUPnP daemon release source code
miniupnpc-1.0-RC13.tar.gz3482003/01/2008 16:50:20 +0000MiniUPnP client release source code
miniupnpd-20071220.tar.gz6721120/12/2007 12:08:34 +0000MiniUPnP daemon source code
miniupnpc-20071219.tar.gz3429019/12/2007 18:31:47 +0000MiniUPnP client source code
minissdpd-1.0-RC12.tar.gz995619/12/2007 18:30:12 +0000MiniSSDPd release source code
miniupnpd-1.0-RC12.tar.gz6691114/12/2007 17:39:20 +0000MiniUPnP daemon release source code
miniupnpc-1.0-RC12.tar.gz3254314/12/2007 17:39:19 +0000MiniUPnP client release source code
miniupnpc-20071213.tar.gz3254113/12/2007 17:09:51 +0000MiniUPnP client source code
miniupnpd-20071213.tar.gz6682613/12/2007 16:42:50 +0000MiniUPnP daemon source code
libnatpmp-20071213.tar.gz599713/12/2007 14:56:30 +0000libnatpmp source code
libnatpmp-20071202.tar.gz566402/12/2007 00:15:28 +0000libnatpmp source code
miniupnpd-20071103.tar.gz6574002/11/2007 23:58:38 +0000MiniUPnP daemon source code
miniupnpd-20071102.tar.gz6573302/11/2007 23:05:44 +0000MiniUPnP daemon source code
miniupnpc-20071103.tar.gz3223902/11/2007 23:05:34 +0000MiniUPnP client source code
miniupnpd-1.0-RC11.tar.gz6482825/10/2007 13:27:18 +0000MiniUPnP daemon release source code
miniupnpc-1.0-RC11.tar.gz3216125/10/2007 13:27:17 +0000MiniUPnP client release source code
upnpc-exe-win32-20071025.zip1280924/10/2007 23:15:55 +0000Windows executable
miniupnpd-1.0-RC10.tar.gz6267412/10/2007 08:38:33 +0000MiniUPnP daemon release source code
miniupnpc-1.0-RC10.tar.gz3196212/10/2007 08:38:31 +0000MiniUPnP client release source code
minissdpd-1.0-RC10.tar.gz951712/10/2007 08:38:30 +0000MiniSSDPd release source code
miniupnpc-20071003.tar.gz3119903/10/2007 15:30:13 +0000MiniUPnP client source code
upnpc-exe-win32-20071001.zip1260401/10/2007 17:09:22 +0000Windows executable
miniupnpd-1.0-RC9.tar.gz5477827/09/2007 19:38:36 +0000MiniUPnP daemon release source code
minissdpd-1.0-RC9.tar.gz916327/09/2007 17:00:03 +0000MiniSSDPd release source code
miniupnpc-1.0-RC9.tar.gz3053827/09/2007 17:00:03 +0000MiniUPnP client release source code
miniupnpd-20070924.tar.gz5233824/09/2007 20:26:05 +0000MiniUPnP daemon source code
miniupnpd-20070923.tar.gz5106023/09/2007 21:13:34 +0000MiniUPnP daemon source code
miniupnpc-20070923.tar.gz3024623/09/2007 21:13:33 +0000MiniUPnP client source code
minissdpd-20070923.tar.gz897823/09/2007 21:13:32 +0000MiniSSDPd source code
miniupnpc-20070902.tar.gz3020501/09/2007 23:47:23 +0000MiniUPnP client source code
minissdpd-20070902.tar.gz653901/09/2007 23:47:20 +0000MiniSSDPd source code
miniupnpd-1.0-RC8.tar.gz5095229/08/2007 10:56:09 +0000MiniUPnP daemon release source code
miniupnpc-1.0-RC8.tar.gz2931229/08/2007 10:56:08 +0000MiniUPnP client release source code
miniupnpd-1.0-RC7.tar.gz5061320/07/2007 00:15:45 +0000MiniUPnP daemon release source code
miniupnpd-1.0-RC6.tar.gz4998612/06/2007 17:12:07 +0000MiniUPnP daemon release source code
miniupnpc-1.0-RC6.tar.gz2903212/06/2007 17:12:06 +0000MiniUPnP client release source code
miniupnpd-20070607.tar.gz4976806/06/2007 23:12:00 +0000MiniUPnP daemon source code
miniupnpd-20070605.tar.gz4971005/06/2007 21:01:53 +0000MiniUPnP daemon source code
miniupnpd-20070521.tar.gz4837421/05/2007 13:07:43 +0000MiniUPnP daemon source code
upnpc-exe-win32-20070519.zip1083619/05/2007 13:14:15 +0000Windows executable
miniupnpc-20070515.tar.gz2580215/05/2007 18:15:25 +0000MiniUPnP client source code
miniupnpd-1.0-RC5.tar.gz4806410/05/2007 20:22:48 +0000MiniUPnP daemon release source code
miniupnpc-1.0-RC5.tar.gz2524210/05/2007 20:22:46 +0000MiniUPnP client release source code
miniupnpd-20070412.tar.gz4780712/04/2007 20:21:48 +0000MiniUPnP daemon source code
miniupnpd-1.0-RC4.tar.gz4768717/03/2007 11:43:13 +0000MiniUPnP daemon release source code
miniupnpc-1.0-RC4.tar.gz2508517/03/2007 11:43:10 +0000MiniUPnP client release source code
miniupnpd-20070311.tar.gz4759911/03/2007 00:25:26 +0000MiniUPnP daemon source code
miniupnpd-20070208.tar.gz4508407/02/2007 23:04:06 +0000MiniUPnP daemon source code
miniupnpd-1.0-RC3.tar.gz4468330/01/2007 23:00:44 +0000MiniUPnP daemon release source code
miniupnpc-1.0-RC3.tar.gz2505530/01/2007 23:00:42 +0000MiniUPnP client release source code
miniupnpd-20070130.tar.gz4373529/01/2007 23:26:16 +0000MiniUPnP daemon source code
miniupnpc-20070130.tar.gz2446629/01/2007 23:26:13 +0000MiniUPnP client source code
miniupnpd-20070127.tar.gz4264327/01/2007 16:02:35 +0000MiniUPnP daemon source code
miniupnpc-20070127.tar.gz2424127/01/2007 16:02:33 +0000MiniUPnP client source code
miniupnpd-1.0-RC2.tar.gz4042417/01/2007 16:13:05 +0000MiniUPnP daemon release source code
miniupnpd-20070112.tar.gz4070812/01/2007 13:40:50 +0000MiniUPnP daemon source code
miniupnpd-20070111.tar.gz4065111/01/2007 18:50:21 +0000MiniUPnP daemon source code
miniupnpd-20070108.tar.gz4002508/01/2007 10:02:14 +0000MiniUPnP daemon source code
miniupnpd-20070103.tar.gz4006503/01/2007 14:39:11 +0000MiniUPnP daemon source code
miniupnpc-20061214.tar.gz2410614/12/2006 15:43:54 +0000MiniUPnP client source code
miniupnpd-20061214.tar.gz3975014/12/2006 13:44:51 +0000MiniUPnP daemon source code
miniupnpd-1.0-RC1.tar.gz3957207/12/2006 10:55:31 +0000MiniUPnP daemon release source code
miniupnpc-1.0-RC1.tar.gz2358207/12/2006 10:55:30 +0000MiniUPnP client release source code
upnpc-exe-win32-20061201.zip1037801/12/2006 00:33:08 +0000Windows executable
miniupnpd20061130.tar.gz3718430/11/2006 12:25:25 +0000MiniUPnP daemon source code
miniupnpd20061129.tar.gz3604529/11/2006 00:10:49 +0000MiniUPnP daemon source code
miniupnpd20061127.tar.gz3415526/11/2006 23:15:28 +0000MiniUPnP daemon source code
miniupnpc20061123.tar.gz2100423/11/2006 22:41:46 +0000MiniUPnP client source code
miniupnpd-bin-openwrt20061123.tar.gz23/11/2006 22:41:44 +0000Precompiled binaries for openwrt
miniupnpd20061123.tar.gz3380923/11/2006 22:28:29 +0000MiniUPnP daemon source code
miniupnpc20061119.tar.gz2089719/11/2006 22:50:37 +0000MiniUPnP client source code
miniupnpd20061119.tar.gz3258019/11/2006 22:50:36 +0000MiniUPnP daemon source code
miniupnpd20061117.tar.gz3264617/11/2006 13:29:33 +0000MiniUPnP daemon source code
upnpc-exe-win32-20061112.zip1026212/11/2006 22:41:25 +0000Windows executable
miniupnpd20061112.tar.gz3202312/11/2006 21:30:32 +0000MiniUPnP daemon source code
miniupnpc20061112.tar.gz2104712/11/2006 21:30:31 +0000MiniUPnP client source code
miniupnpd20061110.tar.gz2792609/11/2006 23:35:02 +0000MiniUPnP daemon source code
miniupnpc20061110.tar.gz2100909/11/2006 23:32:19 +0000MiniUPnP client source code
upnpc-exe-win32-20061101.zip1008908/11/2006 20:35:09 +0000Windows executable
upnpc-exe-win32-20061020.zip918308/11/2006 20:35:08 +0000Windows executable
upnpc-exe-win32-20060909.zip999408/11/2006 20:35:07 +0000Windows executable
upnpc-exe-win32-20060801.zip1000208/11/2006 20:35:06 +0000Windows executable
upnpc-exe-win32-20060711.zip1373308/11/2006 20:35:05 +0000Windows executable
upnpc-exe-win32-20060709.zip1371308/11/2006 20:35:04 +0000Windows executable
upnpc-exe-win32-20060704.zip1329708/11/2006 20:35:03 +0000Windows executable
miniupnpc20061107.tar.gz2070806/11/2006 23:36:57 +0000MiniUPnP client source code
miniupnpd20061107.tar.gz2699206/11/2006 23:35:06 +0000MiniUPnP daemon source code
miniupnpc20061106.tar.gz2057506/11/2006 17:02:15 +0000MiniUPnP client source code
miniupnpd20061106.tar.gz2659706/11/2006 15:39:10 +0000MiniUPnP daemon source code
miniupnpc20061101.tar.gz2039504/11/2006 18:16:15 +0000MiniUPnP client source code
miniupnpc20061031.tar.gz2039604/11/2006 18:16:13 +0000MiniUPnP client source code
miniupnpc20061023.tar.gz2010904/11/2006 18:16:12 +0000MiniUPnP client source code
miniupnpc20061020.tar.gz1973904/11/2006 18:16:10 +0000MiniUPnP client source code
miniupnpc20060909.tar.gz1928504/11/2006 18:16:09 +0000MiniUPnP client source code
miniupnpc20060731.tar.gz1903204/11/2006 18:16:07 +0000MiniUPnP client source code
miniupnpc20060711.tar.gz1915104/11/2006 18:16:06 +0000MiniUPnP client source code
miniupnpc20060709.tar.gz1908004/11/2006 18:16:04 +0000MiniUPnP client source code
miniupnpc20060703.tar.gz1790604/11/2006 18:16:03 +0000MiniUPnP client source code
miniupnpc-new20060630.tar.gz1484004/11/2006 18:16:01 +0000João Paulo Barraca version of the upnp client
miniupnpd20061029.tar.gz2419703/11/2006 13:40:30 +0000MiniUPnP daemon source code
miniupnpd20061027.tar.gz2390403/11/2006 13:40:29 +0000MiniUPnP daemon source code
miniupnpd20061028.tar.gz2438303/11/2006 13:40:29 +0000MiniUPnP daemon source code
miniupnpd20061018.tar.gz2305103/11/2006 13:40:28 +0000MiniUPnP daemon source code
miniupnpd20061023.tar.gz2347803/11/2006 13:40:28 +0000MiniUPnP daemon source code
miniupnpd20060930.tar.gz2283203/11/2006 13:40:28 +0000MiniUPnP daemon source code
miniupnpd20060924.tar.gz2203803/11/2006 13:40:27 +0000MiniUPnP daemon source code
miniupnpd20060919.tar.gz2156603/11/2006 13:40:27 +0000MiniUPnP daemon source code
miniupnpd20060729.tar.gz1920203/11/2006 13:40:26 +0000MiniUPnP daemon source code
miniupnpd20060909.tar.gz1995203/11/2006 13:40:26 +0000MiniUPnP daemon source code
+ +

Home

+

Contact: miniupnp _AT_ free _DOT_ fr

+

+Valid XHTML 1.0 Transitional +Valid CSS! + + + +Fork me on GitHub +

+ + + + + + diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py index 85c1f79f..792ad148 100644 --- a/bitbake/lib/bb/tests/fetch.py +++ b/bitbake/lib/bb/tests/fetch.py @@ -7,16 +7,22 @@ # import contextlib +import http.server +import shutil import unittest +import unittest.mock +import urllib.parse import hashlib import tempfile import collections import os import signal +import subprocess import tarfile +import threading from bb.fetch2 import URI -from bb.fetch2 import FetchMethod import bb +import bb.utils from bb.tests.support.httpserver import HTTPService def skipIfNoNetwork(): @@ -24,6 +30,18 @@ def skipIfNoNetwork(): return unittest.skip("network test") return lambda f: f + +@contextlib.contextmanager +def hide_directory(directory): + """Hide the given directory and restore it after the context is left""" + temp_name = directory + ".bak" + os.rename(directory, temp_name) + try: + yield + finally: + os.rename(temp_name, directory) + + class TestTimeout(Exception): # Indicate to pytest that this is not a test suite __test__ = False @@ -142,39 +160,6 @@ class URITest(unittest.TestCase): 'query': {}, 'relative': False }, - "cvs://anoncvs@cvs.handhelds.org/cvs;module=familiar/dist/ipkg" : { - 'uri': 'cvs://anoncvs@cvs.handhelds.org/cvs;module=familiar/dist/ipkg', - 'scheme': 'cvs', - 'hostname': 'cvs.handhelds.org', - 'port': None, - 'hostport': 'cvs.handhelds.org', - 'path': '/cvs', - 'userinfo': 'anoncvs', - 'username': 'anoncvs', - 'password': '', - 'params': { - 'module': 'familiar/dist/ipkg' - }, - 'query': {}, - 'relative': False - }, - "cvs://anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg": { - 'uri': 'cvs://anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg', - 'scheme': 'cvs', - 'hostname': 'cvs.handhelds.org', - 'port': None, - 'hostport': 'cvs.handhelds.org', - 'path': '/cvs', - 'userinfo': 'anoncvs:anonymous', - 'username': 'anoncvs', - 'password': 'anonymous', - 'params': collections.OrderedDict([ - ('tag', 'V0-99-81'), - ('module', 'familiar/dist/ipkg') - ]), - 'query': {}, - 'relative': False - }, "file://example.diff": { # NOTE: Not RFC compliant! 'uri': 'file:example.diff', 'scheme': 'file', @@ -309,8 +294,8 @@ class URITest(unittest.TestCase): 'query': {}, 'relative': True }, - "https://www.innodisk.com/Download_file?9BE0BF6657;downloadfilename=EGPL-T101.zip": { - 'uri': 'https://www.innodisk.com/Download_file?9BE0BF6657;downloadfilename=EGPL-T101.zip', + "https://www.innodisk.com/Download_file?9BE0BF6657;downloadfilename=EGPL-T101.zip;someparam=": { + 'uri': 'https://www.innodisk.com/Download_file?9BE0BF6657;downloadfilename=EGPL-T101.zip;someparam=', 'scheme': 'https', 'hostname': 'www.innodisk.com', 'port': None, @@ -320,9 +305,24 @@ class URITest(unittest.TestCase): 'userinfo': '', 'username': '', 'password': '', - 'params': {"downloadfilename" : "EGPL-T101.zip"}, + 'params': {"downloadfilename" : "EGPL-T101.zip", "someparam" : ""}, 'query': {"9BE0BF6657": None}, 'relative': False + }, + "file://example@.service": { + 'uri': 'file:example%40.service', + 'scheme': 'file', + 'hostname': '', + 'port': None, + 'hostport': '', + 'path': 'example@.service', + 'userinfo': '', + 'userinfo': '', + 'username': '', + 'password': '', + 'params': {}, + 'query': {}, + 'relative': True } } @@ -459,16 +459,16 @@ class FetcherTest(unittest.TestCase): class MirrorUriTest(FetcherTest): replaceuris = { - ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "http://somewhere.org/somedir/") + ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", "git://.*/.*", "http://somewhere.org/somedir/") : "http://somewhere.org/somedir/git2_git.invalid.infradead.org.mtd-utils.git.tar.gz", - ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/([^/]+/)*([^/]*)", "git://somewhere.org/somedir/\\2;protocol=http") - : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", - ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/([^/]+/)*([^/]*)", "git://somewhere.org/somedir/\\2;protocol=http") - : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", - ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/([^/]+/)*([^/]*)", "git://somewhere.org/\\2;protocol=http") - : "git://somewhere.org/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", - ("git://someserver.org/bitbake;tag=1234567890123456789012345678901234567890", "git://someserver.org/bitbake", "git://git.openembedded.org/bitbake") - : "git://git.openembedded.org/bitbake;tag=1234567890123456789012345678901234567890", + ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", "git://.*/([^/]+/)*([^/]*)", "git://somewhere.org/somedir/\\2;protocol=http") + : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master;protocol=http", + ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", "git://.*/([^/]+/)*([^/]*)", "git://somewhere.org/somedir/\\2;protocol=http") + : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master;protocol=http", + ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", "git://.*/([^/]+/)*([^/]*)", "git://somewhere.org/\\2;protocol=http") + : "git://somewhere.org/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master;protocol=http", + ("git://someserver.org/bitbake;tag=1234567890123456789012345678901234567890;branch=master", "git://someserver.org/bitbake", "git://git.openembedded.org/bitbake") + : "git://git.openembedded.org/bitbake;tag=1234567890123456789012345678901234567890;branch=master", ("file://sstate-xyz.tgz", "file://.*", "file:///somewhere/1234/sstate-cache") : "file:///somewhere/1234/sstate-cache/sstate-xyz.tgz", ("file://sstate-xyz.tgz", "file://.*", "file:///somewhere/1234/sstate-cache/") @@ -481,12 +481,12 @@ class MirrorUriTest(FetcherTest): : "http://archive.apache.org/dist/subversion/subversion-1.7.1.tar.bz2", ("http://www.apache.org/dist/subversion/subversion-1.7.1.tar.bz2", "http://.*/.*", "file:///somepath/downloads/") : "file:///somepath/downloads/subversion-1.7.1.tar.bz2", - ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http") - : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", - ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http") - : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", - ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/MIRRORNAME;protocol=http") - : "git://somewhere.org/somedir/git.invalid.infradead.org.foo.mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", + ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http") + : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master;protocol=http", + ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http") + : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master;protocol=http", + ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", "git://.*/.*", "git://somewhere.org/somedir/MIRRORNAME;protocol=http") + : "git://somewhere.org/somedir/git.invalid.infradead.org.foo.mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master;protocol=http", ("http://somewhere.org/somedir1/somedir2/somefile_1.2.3.tar.gz", "http://.*/.*", "http://somewhere2.org") : "http://somewhere2.org/somefile_1.2.3.tar.gz", ("http://somewhere.org/somedir1/somedir2/somefile_1.2.3.tar.gz", "http://.*/.*", "http://somewhere2.org/") @@ -502,6 +502,10 @@ class MirrorUriTest(FetcherTest): : "file:///mirror/example/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz", ("https://somewhere.org/example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz", "https://.*/.*", "file:///mirror/some-example-1.0.0.tgz") : "file:///mirror/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz", + ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", r"git://(?!internal\.git\.server).*/.*", "http://somewhere.org/somedir/") + : "http://somewhere.org/somedir/git2_git.invalid.infradead.org.mtd-utils.git.tar.gz", + ("git://internal.git.server.org/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", r"git://(?!internal\.git\.server).*/.*", "http://somewhere.org/somedir/") + : None, #Renaming files doesn't work #("http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz") : "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz" @@ -510,8 +514,8 @@ class MirrorUriTest(FetcherTest): mirrorvar = "http://.*/.* file:///somepath/downloads/ " \ "git://someserver.org/bitbake git://git.openembedded.org/bitbake " \ - "https://.*/.* file:///someotherpath/downloads/ " \ - "http://.*/.* file:///someotherpath/downloads/" + "https?://.*/.* file:///someotherpath/downloads/ " \ + "svn://svn.server1.com/ svn://svn.server2.com/" def test_urireplace(self): self.d.setVar("FILESPATH", ".") @@ -520,7 +524,7 @@ class MirrorUriTest(FetcherTest): ud.setup_localpath(self.d) mirrors = bb.fetch2.mirror_from_string("%s %s" % (k[1], k[2])) newuris, uds = bb.fetch2.build_mirroruris(ud, mirrors, self.d) - self.assertEqual([v], newuris) + self.assertEqual([v] if v else [], newuris) def test_urilist1(self): fetcher = bb.fetch.FetchData("http://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d) @@ -535,6 +539,13 @@ class MirrorUriTest(FetcherTest): uris, uds = bb.fetch2.build_mirroruris(fetcher, mirrors, self.d) self.assertEqual(uris, ['file:///someotherpath/downloads/bitbake-1.0.tar.gz']) + def test_urilistsvn(self): + # Catch svn:// -> svn:// bug + fetcher = bb.fetch.FetchData("svn://svn.server1.com/isource/svnroot/reponame/tags/tagname;module=path_in_tagnamefolder;protocol=https;rev=2", self.d) + mirrors = bb.fetch2.mirror_from_string(self.mirrorvar) + uris, uds = bb.fetch2.build_mirroruris(fetcher, mirrors, self.d) + self.assertEqual(uris, ['svn://svn.server2.com/isource/svnroot/reponame/tags/tagname;module=path_in_tagnamefolder;protocol=https;rev=2']) + def test_mirror_of_mirror(self): # Test if mirror of a mirror works mirrorvar = self.mirrorvar + " http://.*/.* http://otherdownloads.yoctoproject.org/downloads/" @@ -542,21 +553,21 @@ class MirrorUriTest(FetcherTest): fetcher = bb.fetch.FetchData("http://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d) mirrors = bb.fetch2.mirror_from_string(mirrorvar) uris, uds = bb.fetch2.build_mirroruris(fetcher, mirrors, self.d) - self.assertEqual(uris, ['file:///somepath/downloads/bitbake-1.0.tar.gz', - 'file:///someotherpath/downloads/bitbake-1.0.tar.gz', + self.assertEqual(uris, ['file:///somepath/downloads/bitbake-1.0.tar.gz', + 'file:///someotherpath/downloads/bitbake-1.0.tar.gz', 'http://otherdownloads.yoctoproject.org/downloads/bitbake-1.0.tar.gz', 'http://downloads2.yoctoproject.org/downloads/bitbake-1.0.tar.gz']) - recmirrorvar = "https://.*/[^/]* http://AAAA/A/A/A/ " \ - "https://.*/[^/]* https://BBBB/B/B/B/" + recmirrorvar = "https://.*/[^/]* http://aaaa/A/A/A/ " \ + "https://.*/[^/]* https://bbbb/B/B/B/" def test_recursive(self): fetcher = bb.fetch.FetchData("https://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d) mirrors = bb.fetch2.mirror_from_string(self.recmirrorvar) uris, uds = bb.fetch2.build_mirroruris(fetcher, mirrors, self.d) - self.assertEqual(uris, ['http://AAAA/A/A/A/bitbake/bitbake-1.0.tar.gz', - 'https://BBBB/B/B/B/bitbake/bitbake-1.0.tar.gz', - 'http://AAAA/A/A/A/B/B/bitbake/bitbake-1.0.tar.gz']) + self.assertEqual(uris, ['http://aaaa/A/A/A/bitbake/bitbake-1.0.tar.gz', + 'https://bbbb/B/B/B/bitbake/bitbake-1.0.tar.gz', + 'http://aaaa/A/A/A/B/B/bitbake/bitbake-1.0.tar.gz']) class GitDownloadDirectoryNamingTest(FetcherTest): @@ -564,8 +575,8 @@ class GitDownloadDirectoryNamingTest(FetcherTest): super(GitDownloadDirectoryNamingTest, self).setUp() self.recipe_url = "git://git.openembedded.org/bitbake;branch=master;protocol=https" self.recipe_dir = "git.openembedded.org.bitbake" - self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https;branch=master" - self.mirror_dir = "github.com.openembedded.bitbake.git" + self.mirror_url = "git://git.openembedded.org/bitbake-contrib.git;protocol=https;branch=master" + self.mirror_dir = "git.openembedded.org.bitbake-contrib.git" self.d.setVar('SRCREV', '82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40') @@ -612,8 +623,8 @@ class TarballNamingTest(FetcherTest): super(TarballNamingTest, self).setUp() self.recipe_url = "git://git.openembedded.org/bitbake;branch=master;protocol=https" self.recipe_tarball = "git2_git.openembedded.org.bitbake.tar.gz" - self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https;branch=master" - self.mirror_tarball = "git2_github.com.openembedded.bitbake.git.tar.gz" + self.mirror_url = "git://git.openembedded.org/bitbake-contrib.git;protocol=https;branch=master" + self.mirror_tarball = "git2_git.openembedded.org.bitbake-contrib.git.tar.gz" self.d.setVar('BB_GENERATE_MIRROR_TARBALLS', '1') self.d.setVar('SRCREV', '82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40') @@ -646,8 +657,8 @@ class GitShallowTarballNamingTest(FetcherTest): super(GitShallowTarballNamingTest, self).setUp() self.recipe_url = "git://git.openembedded.org/bitbake;branch=master;protocol=https" self.recipe_tarball = "gitshallow_git.openembedded.org.bitbake_82ea737-1_master.tar.gz" - self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https;branch=master" - self.mirror_tarball = "gitshallow_github.com.openembedded.bitbake.git_82ea737-1_master.tar.gz" + self.mirror_url = "git://git.openembedded.org/bitbake-contrib.git;protocol=https;branch=master" + self.mirror_tarball = "gitshallow_git.openembedded.org.bitbake-contrib.git_82ea737-1_master.tar.gz" self.d.setVar('BB_GIT_SHALLOW', '1') self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1') @@ -679,7 +690,7 @@ class GitShallowTarballNamingTest(FetcherTest): class CleanTarballTest(FetcherTest): def setUp(self): super(CleanTarballTest, self).setUp() - self.recipe_url = "git://git.openembedded.org/bitbake;protocol=https" + self.recipe_url = "git://git.openembedded.org/bitbake;protocol=https;branch=master" self.recipe_tarball = "git2_git.openembedded.org.bitbake.tar.gz" self.d.setVar('BB_GENERATE_MIRROR_TARBALLS', '1') @@ -720,6 +731,7 @@ class FetcherLocalTest(FetcherTest): os.makedirs(self.localsrcdir) touch(os.path.join(self.localsrcdir, 'a')) touch(os.path.join(self.localsrcdir, 'b')) + touch(os.path.join(self.localsrcdir, 'c@d')) os.makedirs(os.path.join(self.localsrcdir, 'dir')) touch(os.path.join(self.localsrcdir, 'dir', 'c')) touch(os.path.join(self.localsrcdir, 'dir', 'd')) @@ -731,6 +743,34 @@ class FetcherLocalTest(FetcherTest): bb.process.run('tar cjf archive.tar.bz2 -C dir .', cwd=self.localsrcdir) self.d.setVar("FILESPATH", self.localsrcdir) + def make_ar_package(self, package_name, data_member="data.tar"): + if not shutil.which("ar"): + self.skipTest("ar not installed") + + workdir = tempfile.mkdtemp(dir=self.tempdir) + payload = os.path.join(workdir, "payload") + with open(payload, "w") as f: + f.write("payload\n") + + data_path = os.path.join(workdir, data_member) + mode = "w:gz" if data_member.endswith(".gz") else "w" + with tarfile.open(data_path, mode) as archive: + archive.add(payload, arcname="payload") + + with open(os.path.join(workdir, "debian-binary"), "w") as f: + f.write("2.0\n") + + control = os.path.join(workdir, "control") + with open(control, "w") as f: + f.write("Package: fetch-test\nVersion: 1\nArchitecture: all\n") + with tarfile.open(os.path.join(workdir, "control.tar"), "w") as archive: + archive.add(control, arcname="control") + + package_path = os.path.join(self.localsrcdir, package_name) + subprocess.check_call(["ar", "r", package_path, "debian-binary", "control.tar", data_member], + cwd=workdir, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + return package_name + def fetchUnpack(self, uris): fetcher = bb.fetch.Fetch(uris, self.d) fetcher.download() @@ -751,6 +791,10 @@ class FetcherLocalTest(FetcherTest): tree = self.fetchUnpack(['file://a', 'file://dir/c']) self.assertEqual(tree, ['a', 'dir/c']) + def test_local_at(self): + tree = self.fetchUnpack(['file://c@d']) + self.assertEqual(tree, ['c@d']) + def test_local_backslash(self): tree = self.fetchUnpack([r'file://backslash\x2dsystemd-unit.device']) self.assertEqual(tree, [r'backslash\x2dsystemd-unit.device']) @@ -800,6 +844,40 @@ class FetcherLocalTest(FetcherTest): tree = self.fetchUnpack(['file://archive.tar.bz2;subdir=bar;striplevel=1']) self.assertEqual(tree, ['bar/c', 'bar/d', 'bar/subdir/e']) + def test_local_deb_quoted_filename(self): + package = self.make_ar_package("archive$(id).deb") + tree = self.fetchUnpack(['file://%s' % package]) + self.assertEqual(tree, ['payload']) + + def test_local_ipk_gz_data_member(self): + package = self.make_ar_package("archive.ipk", data_member="data.tar.gz") + tree = self.fetchUnpack(['file://%s' % package]) + self.assertEqual(tree, ['payload']) + + def test_local_deb_rejects_unknown_data_member_suffix(self): + package = self.make_ar_package("archive.deb", data_member="data.tar.foo") + with self.assertRaises(bb.fetch2.UnpackError) as context: + self.fetchUnpack(['file://%s' % package]) + + self.assertIn("does not contain supported data.tar* file", str(context.exception)) + + def test_local_deb_rejects_unsafe_data_member(self): + package = self.make_ar_package("archive.deb", data_member="data.tar.xz;id") + with self.assertRaises(bb.fetch2.UnpackError) as context: + self.fetchUnpack(['file://%s' % package]) + + self.assertIn("does not contain supported data.tar* file", str(context.exception)) + + def assertInvalidStriplevel(self, value): + with self.assertRaises(bb.fetch2.UnpackError) as context: + self.fetchUnpack(['file://archive.tar;subdir=bar;striplevel=%s' % value]) + self.assertIn("Invalid striplevel parameter", str(context.exception)) + + def test_local_striplevel_rejects_invalid_values(self): + for value in ("abc", "", "-1", "1\n", "1 2"): + with self.subTest(striplevel=repr(value)): + self.assertInvalidStriplevel(value) + def dummyGitTest(self, suffix): # Create dummy local Git repo src_dir = tempfile.mkdtemp(dir=self.tempdir, @@ -1055,12 +1133,6 @@ class FetcherNetworkTest(FetcherTest): url1 = url2 = "git://git.openembedded.org/bitbake;rev=dead05b0b4ba0959fe0624d2a4885d7b70426da5;branch=master;protocol=https" self.assertRaises(bb.fetch.FetchError, self.gitfetcher, url1, url2) - @skipIfNoNetwork() - def test_gitfetch_tagandrev(self): - # SRCREV is set but does not match rev= parameter - url1 = url2 = "git://git.openembedded.org/bitbake;rev=270a05b0b4ba0959fe0624d2a4885d7b70426da5;tag=270a05b0b4ba0959fe0624d2a4885d7b70426da5;protocol=https" - self.assertRaises(bb.fetch.FetchError, self.gitfetcher, url1, url2) - @skipIfNoNetwork() def test_gitfetch_usehead(self): # Since self.gitfetcher() sets SRCREV we expect this to override @@ -1095,7 +1167,7 @@ class FetcherNetworkTest(FetcherTest): @skipIfNoNetwork() def test_gitfetch_finds_local_repository_when_premirror_rewrites_the_recipe_url(self): realurl = "https://git.openembedded.org/bitbake" - recipeurl = "git://someserver.org/bitbake;protocol=https" + recipeurl = "git://someserver.org/bitbake;protocol=https;branch=master" self.sourcedir = self.unpackdir.replace("unpacked", "sourcemirror.git") os.chdir(self.tempdir) self.git(['clone', realurl, self.sourcedir], cwd=self.tempdir) @@ -1107,7 +1179,7 @@ class FetcherNetworkTest(FetcherTest): # URL with ssh submodules url = "gitsm://git.yoctoproject.org/git-submodule-test;branch=ssh-gitsm-tests;rev=049da4a6cb198d7c0302e9e8b243a1443cb809a7;branch=master;protocol=https" # Original URL (comment this if you have ssh access to git.yoctoproject.org) - url = "gitsm://git.yoctoproject.org/git-submodule-test;branch=master;rev=a2885dd7d25380d23627e7544b7bbb55014b16ee;branch=master;protocol=https" + url = "gitsm://git.yoctoproject.org/git-submodule-test;branch=master;rev=38e61644af90dccd73c03ed3acaed98c8dda9294;branch=master;protocol=https" fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() # Previous cwd has been deleted @@ -1127,7 +1199,7 @@ class FetcherNetworkTest(FetcherTest): def test_git_submodule_restricted_network_premirrors(self): # this test is to ensure that premirrors will be tried in restricted network # that is, BB_ALLOWED_NETWORKS does not contain the domain the url uses - url = "gitsm://github.com/grpc/grpc.git;protocol=https;name=grpc;branch=v1.60.x;rev=0ef13a7555dbaadd4633399242524129eef5e231" + url = "gitsm://git.yoctoproject.org/git-submodule-test;branch=master;rev=38e61644af90dccd73c03ed3acaed98c8dda9294;branch=master;protocol=https" # create a download directory to be used as premirror later tempdir = tempfile.mkdtemp(prefix="bitbake-fetch-") dl_premirror = os.path.join(tempdir, "download-premirror") @@ -1146,7 +1218,7 @@ class FetcherNetworkTest(FetcherTest): def test_git_submodule_dbus_broker(self): # The following external repositories have show failures in fetch and unpack operations # We want to avoid regressions! - url = "gitsm://github.com/bus1/dbus-broker;protocol=https;rev=fc874afa0992d0c75ec25acb43d344679f0ee7d2;branch=main" + url = "gitsm://git.yoctoproject.org/bbfetchtests-bus1-dbus-broker;protocol=https;rev=fc874afa0992d0c75ec25acb43d344679f0ee7d2;branch=main" fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() # Previous cwd has been deleted @@ -1162,7 +1234,7 @@ class FetcherNetworkTest(FetcherTest): @skipIfNoNetwork() def test_git_submodule_CLI11(self): - url = "gitsm://github.com/CLIUtils/CLI11;protocol=https;rev=bd4dc911847d0cde7a6b41dfa626a85aab213baf;branch=main" + url = "gitsm://git.yoctoproject.org/bbfetchtests-cliutils-cli11;protocol=https;rev=474872a222f61e57b16c2feb774610a4595a729e;branch=main" fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() # Previous cwd has been deleted @@ -1172,17 +1244,16 @@ class FetcherNetworkTest(FetcherTest): repo_path = os.path.join(self.tempdir, 'unpacked', 'git') self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/googletest/config')), msg='Missing submodule config "extern/googletest"') self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"') - self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"') @skipIfNoNetwork() def test_git_submodule_update_CLI11(self): """ Prevent regression on update detection not finding missing submodule, or modules without needed commits """ - url = "gitsm://github.com/CLIUtils/CLI11;protocol=https;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714;branch=main" + url = "gitsm://git.yoctoproject.org/bbfetchtests-cliutils-cli11;protocol=https;rev=fe0e149a0624db734de37dc827af3328855f51cf;branch=main" fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() # CLI11 that pulls in a newer nlohmann-json - url = "gitsm://github.com/CLIUtils/CLI11;protocol=https;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca;branch=main" + url = "gitsm://git.yoctoproject.org/bbfetchtests-cliutils-cli11;protocol=https;rev=141f9dca45bd5ef42e7b1a06637860b7581370c8;branch=main" fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() # Previous cwd has been deleted @@ -1192,11 +1263,10 @@ class FetcherNetworkTest(FetcherTest): repo_path = os.path.join(self.tempdir, 'unpacked', 'git') self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/googletest/config')), msg='Missing submodule config "extern/googletest"') self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"') - self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"') @skipIfNoNetwork() def test_git_submodule_aktualizr(self): - url = "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=https;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44" + url = "gitsm://git.yoctoproject.org/bbfetchtests-advancedtelematic-aktualizr;branch=master;protocol=https;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44" fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() # Previous cwd has been deleted @@ -1241,7 +1311,7 @@ class FetcherNetworkTest(FetcherTest): @skipIfNoNetwork() def test_git_submodule_reference_to_parent(self): - self.recipe_url = "gitsm://github.com/gflags/gflags.git;protocol=https;branch=master" + self.recipe_url = "gitsm://git.yoctoproject.org/bbfetchtests-gflags;protocol=https;branch=master" self.d.setVar("SRCREV", "14e1138441bbbb584160cb1c0a0426ec1bac35f1") with Timeout(60): fetcher = bb.fetch.Fetch([self.recipe_url], self.d) @@ -1250,7 +1320,6 @@ class FetcherNetworkTest(FetcherTest): class SVNTest(FetcherTest): def skipIfNoSvn(): - import shutil if not shutil.which("svn"): return unittest.skip("svn not installed, tests being skipped") @@ -1373,17 +1442,17 @@ class TrustedNetworksTest(FetcherTest): self.assertFalse(bb.fetch.trusted_network(self.d, url)) class URLHandle(unittest.TestCase): - + # Quote password as per RFC3986 + password = urllib.parse.quote(r"!#$%^&*()-_={}[]\|:?,.<>~`", r"!$&'/()*+,;=") datatable = { "http://www.google.com/index.html" : ('http', 'www.google.com', '/index.html', '', '', {}), - "cvs://anoncvs@cvs.handhelds.org/cvs;module=familiar/dist/ipkg" : ('cvs', 'cvs.handhelds.org', '/cvs', 'anoncvs', '', {'module': 'familiar/dist/ipkg'}), - "cvs://anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg" : ('cvs', 'cvs.handhelds.org', '/cvs', 'anoncvs', 'anonymous', collections.OrderedDict([('tag', 'V0-99-81'), ('module', 'familiar/dist/ipkg')])), "git://git.openembedded.org/bitbake;branch=@foo;protocol=https" : ('git', 'git.openembedded.org', '/bitbake', '', '', {'branch': '@foo', 'protocol' : 'https'}), "file://somelocation;someparam=1": ('file', '', 'somelocation', '', '', {'someparam': '1'}), + "file://example@.service": ('file', '', 'example@.service', '', '', {}), "https://somesite.com/somerepo.git;user=anyUser:idtoken=1234" : ('https', 'somesite.com', '/somerepo.git', '', '', {'user': 'anyUser:idtoken=1234'}), - r'git://s.o-me_ONE:!#$%^&*()-_={}[]\|:?,.<>~`@git.openembedded.org/bitbake;branch=main;protocol=https': ('git', 'git.openembedded.org', '/bitbake', 's.o-me_ONE', r'!#$%^&*()-_={}[]\|:?,.<>~`', {'branch': 'main', 'protocol' : 'https'}), + 'git://s.o-me_ONE:%s@git.openembedded.org/bitbake;branch=main;protocol=https' % password: ('git', 'git.openembedded.org', '/bitbake', 's.o-me_ONE', password, {'branch': 'main', 'protocol' : 'https'}), } - # we require a pathname to encodeurl but users can still pass such urls to + # we require a pathname to encodeurl but users can still pass such urls to # decodeurl and we need to handle them decodedata = datatable.copy() decodedata.update({ @@ -1399,13 +1468,15 @@ class URLHandle(unittest.TestCase): def test_encodeurl(self): for k, v in self.datatable.items(): result = bb.fetch.encodeurl(v) + if result.startswith("file:"): + result = urllib.parse.unquote(result) self.assertEqual(result, k) class FetchLatestVersionTest(FetcherTest): test_git_uris = { # version pattern "X.Y.Z" - ("mx-1.0", "git://github.com/clutter-project/mx.git;branch=mx-1.4;protocol=https", "9b1db6b8060bd00b121a692f942404a24ae2960f", "", "") + ("mx-1.0", "git://git.yoctoproject.org/bbfetchtests-clutter-mx;branch=mx-1.4;protocol=https", "9b1db6b8060bd00b121a692f942404a24ae2960f", "", "") : "1.99.4", # version pattern "vX.Y" # mirror of git.infradead.org since network issues interfered with testing @@ -1419,12 +1490,12 @@ class FetchLatestVersionTest(FetcherTest): ("dtc", "git://git.yoctoproject.org/bbfetchtests-dtc.git;branch=master;protocol=https", "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf", "", "") : "1.4.0", # combination version pattern - ("sysprof", "git://gitlab.gnome.org/GNOME/sysprof.git;protocol=https;branch=master", "cd44ee6644c3641507fb53b8a2a69137f2971219", "", "") + ("sysprof", "git://git.yoctoproject.org/sysprof.git;protocol=https;branch=master", "cd44ee6644c3641507fb53b8a2a69137f2971219", "", "") : "1.2.0", - ("u-boot-mkimage", "git://git.denx.de/u-boot.git;branch=master;protocol=git", "62c175fbb8a0f9a926c88294ea9f7e88eb898f6c", "", "") + ("u-boot-mkimage", "git://git.yoctoproject.org/bbfetchtests-u-boot.git;branch=master;protocol=https", "62c175fbb8a0f9a926c88294ea9f7e88eb898f6c", "", "") : "2014.01", # version pattern "yyyymmdd" - ("mobile-broadband-provider-info", "git://gitlab.gnome.org/GNOME/mobile-broadband-provider-info.git;protocol=https;branch=master", "4ed19e11c2975105b71b956440acdb25d46a347d", "", "") + ("mobile-broadband-provider-info", "git://git.yoctoproject.org/mobile-broadband-provider-info.git;protocol=https;branch=master", "4ed19e11c2975105b71b956440acdb25d46a347d", "", "") : "20120614", # packages with a valid UPSTREAM_CHECK_GITTAGREGEX # mirror of git://anongit.freedesktop.org/xorg/driver/xf86-video-omap since network issues interfered with testing @@ -1432,106 +1503,210 @@ class FetchLatestVersionTest(FetcherTest): : "0.4.3", ("build-appliance-image", "git://git.yoctoproject.org/poky;branch=master;protocol=https", "b37dd451a52622d5b570183a81583cc34c2ff555", r"(?P(([0-9][\.|_]?)+[0-9]))", "") : "11.0.0", - ("chkconfig-alternatives-native", "git://github.com/kergoth/chkconfig;branch=sysroot;protocol=https", "cd437ecbd8986c894442f8fce1e0061e20f04dee", r"chkconfig\-(?P((\d+[\.\-_]*)+))", "") + ("chkconfig-alternatives-native", "git://git.yoctoproject.org/bbfetchtests-kergoth-chkconfig;branch=sysroot;protocol=https", "cd437ecbd8986c894442f8fce1e0061e20f04dee", r"chkconfig\-(?P((\d+[\.\-_]*)+))", "") : "1.3.59", - ("remake", "git://github.com/rocky/remake.git;protocol=https;branch=master", "f05508e521987c8494c92d9c2871aec46307d51d", r"(?P(\d+\.(\d+\.)*\d*(\+dbg\d+(\.\d+)*)*))", "") + ("remake", "git://git.yoctoproject.org/bbfetchtests-rocky-remake;protocol=https;branch=master", "f05508e521987c8494c92d9c2871aec46307d51d", r"(?P(\d+\.(\d+\.)*\d*(\+dbg\d+(\.\d+)*)*))", "") : "3.82+dbg0.9", - ("sysdig", "git://github.com/draios/sysdig.git;branch=dev;protocol=https", "4fb6288275f567f63515df0ff0a6518043ecfa9b", r"^(?P\d+(\.\d+)+)", "10.0.0") + ("sysdig", "git://git.yoctoproject.org/bbfetchtests-draios-sysdig;branch=dev;protocol=https", "4fb6288275f567f63515df0ff0a6518043ecfa9b", r"^(?P\d+(\.\d+)+)", "10.0.0") : "0.28.0", } + WgetTestData = collections.namedtuple("WgetTestData", ["pn", "path", "pv", "check_uri", "check_regex"], defaults=[None, None, None]) test_wget_uris = { # # packages with versions inside directory name # # http://kernel.org/pub/linux/utils/util-linux/v2.23/util-linux-2.24.2.tar.bz2 - ("util-linux", "/pub/linux/utils/util-linux/v2.23/util-linux-2.24.2.tar.bz2", "", "") + WgetTestData("util-linux", "/pub/linux/utils/util-linux/v2.23/util-linux-2.24.2.tar.bz2") : "2.24.2", # http://www.abisource.com/downloads/enchant/1.6.0/enchant-1.6.0.tar.gz - ("enchant", "/downloads/enchant/1.6.0/enchant-1.6.0.tar.gz", "", "") + WgetTestData("enchant", "/downloads/enchant/1.6.0/enchant-1.6.0.tar.gz") : "1.6.0", # http://www.cmake.org/files/v2.8/cmake-2.8.12.1.tar.gz - ("cmake", "/files/v2.8/cmake-2.8.12.1.tar.gz", "", "") + WgetTestData("cmake", "/files/v2.8/cmake-2.8.12.1.tar.gz") : "2.8.12.1", # https://download.gnome.org/sources/libxml2/2.9/libxml2-2.9.14.tar.xz - ("libxml2", "/software/libxml2/2.9/libxml2-2.9.14.tar.xz", "", "") + WgetTestData("libxml2", "/software/libxml2/2.9/libxml2-2.9.14.tar.xz") : "2.10.3", # # packages with versions only in current directory # # https://downloads.yoctoproject.org/releases/eglibc/eglibc-2.18-svnr23787.tar.bz2 - ("eglic", "/releases/eglibc/eglibc-2.18-svnr23787.tar.bz2", "", "") + WgetTestData("eglic", "/releases/eglibc/eglibc-2.18-svnr23787.tar.bz2") : "2.19", # https://downloads.yoctoproject.org/releases/gnu-config/gnu-config-20120814.tar.bz2 - ("gnu-config", "/releases/gnu-config/gnu-config-20120814.tar.bz2", "", "") + WgetTestData("gnu-config", "/releases/gnu-config/gnu-config-20120814.tar.bz2") : "20120814", # # packages with "99" in the name of possible version # # http://freedesktop.org/software/pulseaudio/releases/pulseaudio-4.0.tar.xz - ("pulseaudio", "/software/pulseaudio/releases/pulseaudio-4.0.tar.xz", "", "") + WgetTestData("pulseaudio", "/software/pulseaudio/releases/pulseaudio-4.0.tar.xz") : "5.0", # http://xorg.freedesktop.org/releases/individual/xserver/xorg-server-1.15.1.tar.bz2 - ("xserver-xorg", "/releases/individual/xserver/xorg-server-1.15.1.tar.bz2", "", "") + WgetTestData("xserver-xorg", "/releases/individual/xserver/xorg-server-1.15.1.tar.bz2") : "1.15.1", # # packages with valid UPSTREAM_CHECK_URI and UPSTREAM_CHECK_REGEX # # http://www.cups.org/software/1.7.2/cups-1.7.2-source.tar.bz2 # https://github.com/apple/cups/releases - ("cups", "/software/1.7.2/cups-1.7.2-source.tar.bz2", "/apple/cups/releases", r"(?Pcups\-)(?P((\d+[\.\-_]*)+))\-source\.tar\.gz") + WgetTestData("cups", "/software/1.7.2/cups-1.7.2-source.tar.bz2", check_uri="/apple/cups/releases", check_regex=r"(?Pcups\-)(?P((\d+[\.\-_]*)+))\-source\.tar\.gz") : "2.0.0", # http://download.oracle.com/berkeley-db/db-5.3.21.tar.gz # http://ftp.debian.org/debian/pool/main/d/db5.3/ - ("db", "/berkeley-db/db-5.3.21.tar.gz", "/debian/pool/main/d/db5.3/", r"(?Pdb5\.3_)(?P\d+(\.\d+)+).+\.orig\.tar\.xz") + WgetTestData("db", "/berkeley-db/db-5.3.21.tar.gz", check_uri="/debian/pool/main/d/db5.3/", check_regex=r"(?Pdb5\.3_)(?P\d+(\.\d+)+).+\.orig\.tar\.xz") : "5.3.10", # # packages where the tarball compression changed in the new version # # http://ftp.debian.org/debian/pool/main/m/minicom/minicom_2.7.1.orig.tar.gz - ("minicom", "/debian/pool/main/m/minicom/minicom_2.7.1.orig.tar.gz", "", "") + WgetTestData("minicom", "/debian/pool/main/m/minicom/minicom_2.7.1.orig.tar.gz") : "2.8", + + # + # packages where the path doesn't actually contain the filename, so downloadfilename should be respected + # + WgetTestData("miniupnpd", "/software/miniupnp/download.php?file=miniupnpd_2.1.20191006.tar.gz;downloadfilename=miniupnpd_2.1.20191006.tar.gz", pv="2.1.20191006", check_uri="/software/miniupnp/download.php", check_regex=r"miniupnpd-(?P\d+(\.\d+)+)\.tar") + : "2.3.7", + } + + test_crate_uris = { + # basic example; version pattern "A.B.C+cargo-D.E.F" + ("cargo-c", "crate://crates.io/cargo-c/0.9.18+cargo-0.69") + : "0.9.29" + } + + test_git_stable_uris = { + ("dtc", "git://git.yoctoproject.org/bbfetchtests-dtc.git;branch=master;protocol=https", "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf", "", r"^1\.4\.\d+$") + : ("1.4.0", "1.5.0"), + ("systemd", "git://git.yoctoproject.org/bbfetchtests-systemd;protocol=https;branch=stable/v259-stable", "b3d8fc43e9cb531d958c17ef2cd93b374bc14e8a", "", r"^259\.\d+$") + : ("259.5", "260") + } + + test_wget_stable_uris = { + ("openssh", "https://ftp.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-10.2p1.tar.gz", "10.2p1", "", "", r"^10\.2p\d+$") + : ("10.2p1", "10.3") + } + + test_crate_stable_uris = { + ("cargo-c", "crate://crates.io/cargo-c/0.9.18+cargo-0.69", r"^0\.9\.\d+") + : ("0.9.29", "0.10.0") } @skipIfNoNetwork() def test_git_latest_versionstring(self): for k, v in self.test_git_uris.items(): - self.d.setVar("PN", k[0]) - self.d.setVar("SRCREV", k[2]) - self.d.setVar("UPSTREAM_CHECK_GITTAGREGEX", k[3]) - ud = bb.fetch2.FetchData(k[1], self.d) - pupver= ud.method.latest_versionstring(ud, self.d) - verstring = pupver[0] - self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0]) - r = bb.utils.vercmp_string(v, verstring) - self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s <= %s" % (k[0], v, verstring)) - if k[4]: - r = bb.utils.vercmp_string(verstring, k[4]) - self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s <= %s" % (k[0], verstring, k[4])) + with self.subTest(pn=k[0]): + self.d.setVar("PN", k[0]) + self.d.setVar("SRCREV", k[2]) + self.d.setVar("UPSTREAM_CHECK_GITTAGREGEX", k[3]) + ud = bb.fetch2.FetchData(k[1], self.d) + pupver= ud.method.latest_versionstring(ud, self.d) + verstring = pupver[0] + self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0]) + r = bb.utils.vercmp_string(v, verstring) + self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s <= %s" % (k[0], v, verstring)) + if k[4]: + r = bb.utils.vercmp_string(verstring, k[4]) + self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s <= %s" % (k[0], verstring, k[4])) def test_wget_latest_versionstring(self): testdata = os.path.dirname(os.path.abspath(__file__)) + "/fetch-testdata" - server = HTTPService(testdata) + server = HTTPService(testdata, host="127.0.0.1") server.start() port = server.port try: - for k, v in self.test_wget_uris.items(): + for data, v in self.test_wget_uris.items(): + with self.subTest(pn=data.pn): + self.d.setVar("PN", data.pn) + self.d.setVar("PV", data.pv) + if data.check_uri: + checkuri = "http://127.0.0.1:%s/%s" % (port, data.check_uri) + self.d.setVar("UPSTREAM_CHECK_URI", checkuri) + if data.check_regex: + self.d.setVar("UPSTREAM_CHECK_REGEX", data.check_regex) + + url = "http://127.0.0.1:%s/%s" % (port, data.path) + ud = bb.fetch2.FetchData(url, self.d) + pupver = ud.method.latest_versionstring(ud, self.d) + verstring = pupver[0] + self.assertTrue(verstring, msg="Could not find upstream version for %s" % data.pn) + r = bb.utils.vercmp_string(v, verstring) + self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s <= %s" % (data.pn, v, verstring)) + finally: + server.stop() + + @skipIfNoNetwork() + def test_crate_latest_versionstring(self): + for k, v in self.test_crate_uris.items(): + with self.subTest(pn=k[0]): self.d.setVar("PN", k[0]) - checkuri = "" - if k[2]: - checkuri = "http://localhost:%s/" % port + k[2] - self.d.setVar("UPSTREAM_CHECK_URI", checkuri) - self.d.setVar("UPSTREAM_CHECK_REGEX", k[3]) - url = "http://localhost:%s/" % port + k[1] - ud = bb.fetch2.FetchData(url, self.d) + ud = bb.fetch2.FetchData(k[1], self.d) pupver = ud.method.latest_versionstring(ud, self.d) verstring = pupver[0] self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0]) r = bb.utils.vercmp_string(v, verstring) self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s <= %s" % (k[0], v, verstring)) - finally: - server.stop() + @skipIfNoNetwork() + def test_git_latest_versionstring_stable(self): + for k, v in self.test_git_stable_uris.items(): + with self.subTest(pn=k[0]): + self.d.setVar("PN", k[0]) + self.d.setVar("SRCREV", k[2]) + self.d.setVar("UPSTREAM_CHECK_GITTAGREGEX", k[3]) + filter_regex = k[4] + ud = bb.fetch2.FetchData(k[1], self.d) + pupver= ud.method.latest_versionstring(ud, self.d, filter_regex=filter_regex) + verstring = pupver[0] + self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0]) + v_less_or_equal = v[0] + v_larger = v[1] + r = bb.utils.vercmp_string(v_less_or_equal, verstring) + self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s < %s" % (k[0], v_less_or_equal, verstring)) + r = bb.utils.vercmp_string(verstring, v_larger) + self.assertTrue(r == -1, msg="Package %s, version: %s <= %s" % (k[0], v_larger, verstring)) + + @skipIfNoNetwork() + def test_wget_latest_versionstring_stable(self): + for k, v in self.test_wget_stable_uris.items(): + with self.subTest(pn=k[0]): + self.d.setVar("PN", k[0]) + url = k[1] + self.d.setVar("PV", k[2]) + if k[3]: + self.d.setVar("UPSTREAM_CHECK_URI", k[3]) + if k[4]: + self.d.setVar("UPSTREAM_CHECK_REGEX", k[4]) + filter_regex = k[5] + ud = bb.fetch2.FetchData(url, self.d) + pupver= ud.method.latest_versionstring(ud, self.d, filter_regex=filter_regex) + verstring = pupver[0] + self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0]) + v_less_or_equal = v[0] + v_larger = v[1] + r = bb.utils.vercmp_string(v_less_or_equal, verstring) + self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s < %s" % (k[0], v_less_or_equal, verstring)) + r = bb.utils.vercmp_string(verstring, v_larger) + self.assertTrue(r == -1, msg="Package %s, version: %s <= %s" % (k[0], v_larger, verstring)) + + @skipIfNoNetwork() + def test_crate_latest_versionstring_stable(self): + for k, v in self.test_crate_stable_uris.items(): + with self.subTest(pn=k[0]): + self.d.setVar("PN", k[0]) + ud = bb.fetch2.FetchData(k[1], self.d) + filter_regex = k[2] + pupver = ud.method.latest_versionstring(ud, self.d, filter_regex=filter_regex) + verstring = pupver[0] + self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0]) + v_less_or_equal = v[0] + v_larger = v[1] + r = bb.utils.vercmp_string(v_less_or_equal, verstring) + self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s < %s" % (k[0], v_less_or_equal, verstring)) + r = bb.utils.vercmp_string(verstring, v_larger) + self.assertTrue(r == -1, msg="Package %s, version: %s <= %s" % (k[0], v_larger, verstring)) class FetchCheckStatusTest(FetcherTest): test_wget_uris = ["https://downloads.yoctoproject.org/releases/sato/sato-engine-0.1.tar.gz", @@ -1541,11 +1716,43 @@ class FetchCheckStatusTest(FetcherTest): "https://docs.yoctoproject.org", "https://downloads.yoctoproject.org/releases/opkg/opkg-0.1.7.tar.gz", "https://downloads.yoctoproject.org/releases/opkg/opkg-0.3.0.tar.gz", - "ftp://sourceware.org/pub/libffi/libffi-1.20.tar.gz", - # GitHub releases are hosted on Amazon S3, which doesn't support HEAD - "https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz" ] + def _start_checkstatus_server(self): + class CheckStatusHTTPRequestHandler(http.server.BaseHTTPRequestHandler): + def do_HEAD(self): + self.server.requests.append((self.path, dict(self.headers))) + if self.path == "/a" and self.server.redirect_url: + self.send_response(302) + self.send_header("Location", self.server.redirect_url) + self.end_headers() + return + self.send_response(200) + self.end_headers() + + def log_message(self, format_str, *args): + pass + + server = http.server.HTTPServer(("127.0.0.1", 0), CheckStatusHTTPRequestHandler) + server.redirect_url = None + server.requests = [] + thread = threading.Thread(target=server.serve_forever, kwargs={"poll_interval": 0.05}) + thread.daemon = True + thread.start() + + def stop_server(): + server.shutdown() + thread.join() + server.server_close() + + self.addCleanup(stop_server) + return server + + def _checkstatus(self, url): + fetch = bb.fetch2.Fetch([url], self.d) + ud = fetch.ud[url] + return ud.method.checkstatus(fetch, ud, self.d) + @skipIfNoNetwork() def test_wget_checkstatus(self): fetch = bb.fetch2.Fetch(self.test_wget_uris, self.d) @@ -1573,6 +1780,31 @@ class FetchCheckStatusTest(FetcherTest): connection_cache.close_connections() + def test_wget_checkstatus_same_origin_redirect_keeps_auth(self): + server = self._start_checkstatus_server() + server.redirect_url = "http://127.0.0.1:%s/b" % server.server_port + + url = "http://127.0.0.1:%s/a;user=user;pswd=pass" % server.server_port + self.assertTrue(self._checkstatus(url)) + + self.assertEqual(len(server.requests), 2) + redirected_headers = {k.lower(): v for k, v in server.requests[1][1].items()} + self.assertIn("authorization", redirected_headers) + + def test_wget_checkstatus_different_origin_redirect_drops_auth(self): + origin = self._start_checkstatus_server() + target = self._start_checkstatus_server() + # Same host but different port is a different origin. + origin.redirect_url = "http://127.0.0.1:%s/b" % target.server_port + + url = "http://127.0.0.1:%s/a;user=user;pswd=pass" % origin.server_port + self.assertTrue(self._checkstatus(url)) + + self.assertEqual(len(origin.requests), 1) + self.assertEqual(len(target.requests), 1) + redirected_headers = {k.lower(): v for k, v in target.requests[0][1].items()} + self.assertNotIn("authorization", redirected_headers) + class GitMakeShallowTest(FetcherTest): def setUp(self): @@ -1715,6 +1947,8 @@ class GitShallowTest(FetcherTest): if cwd is None: cwd = self.gitdir actual_refs = self.git(['for-each-ref', '--format=%(refname)'], cwd=cwd).splitlines() + # Resolve references into the same format as the comparision (needed by git 2.48 onwards) + actual_refs = self.git(['rev-parse', '--symbolic-full-name'] + actual_refs, cwd=cwd).splitlines() full_expected = self.git(['rev-parse', '--symbolic-full-name'] + expected_refs, cwd=cwd).splitlines() self.assertEqual(sorted(set(full_expected)), sorted(set(actual_refs))) @@ -1761,7 +1995,6 @@ class GitShallowTest(FetcherTest): def fetch_shallow(self, uri=None, disabled=False, keepclone=False): """Fetch a uri, generating a shallow tarball, then unpack using it""" fetcher, ud = self.fetch_and_unpack(uri) - assert os.path.exists(ud.clonedir), 'Git clone in DLDIR (%s) does not exist for uri %s' % (ud.clonedir, uri) # Confirm that the unpacked repo is unshallow if not disabled: @@ -1769,9 +2002,10 @@ class GitShallowTest(FetcherTest): # fetch and unpack, from the shallow tarball bb.utils.remove(self.gitdir, recurse=True) - bb.process.run('chmod u+w -R "%s"' % ud.clonedir) - bb.utils.remove(ud.clonedir, recurse=True) - bb.utils.remove(ud.clonedir.replace('gitsource', 'gitsubmodule'), recurse=True) + if os.path.exists(ud.clonedir): + bb.process.run('chmod u+w -R "%s"' % ud.clonedir) + bb.utils.remove(ud.clonedir, recurse=True) + bb.utils.remove(ud.clonedir.replace('gitsource', 'gitsubmodule'), recurse=True) # confirm that the unpacked repo is used when no git clone or git # mirror tarball is available @@ -1854,7 +2088,12 @@ class GitShallowTest(FetcherTest): self.add_empty_file('c') self.assertRevCount(3, cwd=self.srcdir) + # Clone without tarball + self.d.setVar('BB_GIT_SHALLOW', '0') + fetcher, ud = self.fetch() + # Clone and generate mirror tarball + self.d.setVar('BB_GIT_SHALLOW', '1') fetcher, ud = self.fetch() # Ensure we have a current mirror tarball, but an out of date clone @@ -1866,6 +2105,7 @@ class GitShallowTest(FetcherTest): fetcher, ud = self.fetch() fetcher.unpack(self.d.getVar('WORKDIR')) self.assertRevCount(1) + assert os.path.exists(os.path.join(self.d.getVar('WORKDIR'), 'git', 'c')) def test_shallow_single_branch_no_merge(self): self.add_empty_file('a') @@ -1963,7 +2203,7 @@ class GitShallowTest(FetcherTest): self.git('submodule update', cwd=self.srcdir) self.git('commit -m submodule -a', cwd=self.srcdir) - uri = 'gitsm://%s;protocol=file;subdir=${S}' % self.srcdir + uri = 'gitsm://%s;protocol=file;subdir=${S};branch=master' % self.srcdir # Fetch once to generate the shallow tarball fetcher, ud = self.fetch(uri) @@ -2004,70 +2244,17 @@ class GitShallowTest(FetcherTest): assert './.git/annex/' in bb.process.run('tar -tzf %s' % os.path.join(self.dldir, ud.mirrortarballs[0]))[0] assert os.path.exists(os.path.join(self.gitdir, 'c')) - def test_shallow_multi_one_uri(self): - # Create initial git repo - self.add_empty_file('a') - self.add_empty_file('b') - self.git('checkout -b a_branch', cwd=self.srcdir) - self.add_empty_file('c') - self.add_empty_file('d') - self.git('checkout master', cwd=self.srcdir) - self.git('tag v0.0 a_branch', cwd=self.srcdir) - self.add_empty_file('e') - self.git('merge --no-ff --no-edit a_branch', cwd=self.srcdir) - self.add_empty_file('f') - self.assertRevCount(7, cwd=self.srcdir) - - uri = self.d.getVar('SRC_URI').split()[0] - uri = '%s;branch=master,a_branch;name=master,a_branch' % uri - - self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0') - self.d.setVar('BB_GIT_SHALLOW_REVS', 'v0.0') - self.d.setVar('SRCREV_master', '${AUTOREV}') - self.d.setVar('SRCREV_a_branch', '${AUTOREV}') - - self.fetch_shallow(uri) - - self.assertRevCount(5) - self.assertRefs(['master', 'origin/master', 'origin/a_branch']) - - def test_shallow_multi_one_uri_depths(self): - # Create initial git repo - self.add_empty_file('a') - self.add_empty_file('b') - self.git('checkout -b a_branch', cwd=self.srcdir) - self.add_empty_file('c') - self.add_empty_file('d') - self.git('checkout master', cwd=self.srcdir) - self.add_empty_file('e') - self.git('merge --no-ff --no-edit a_branch', cwd=self.srcdir) - self.add_empty_file('f') - self.assertRevCount(7, cwd=self.srcdir) - - uri = self.d.getVar('SRC_URI').split()[0] - uri = '%s;branch=master,a_branch;name=master,a_branch' % uri - - self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0') - self.d.setVar('BB_GIT_SHALLOW_DEPTH_master', '3') - self.d.setVar('BB_GIT_SHALLOW_DEPTH_a_branch', '1') - self.d.setVar('SRCREV_master', '${AUTOREV}') - self.d.setVar('SRCREV_a_branch', '${AUTOREV}') - - self.fetch_shallow(uri) - - self.assertRevCount(4, ['--all']) - self.assertRefs(['master', 'origin/master', 'origin/a_branch']) - def test_shallow_clone_preferred_over_shallow(self): self.add_empty_file('a') self.add_empty_file('b') # Fetch once to generate the shallow tarball + self.d.setVar('BB_GIT_SHALLOW', '0') fetcher, ud = self.fetch() - assert os.path.exists(os.path.join(self.dldir, ud.mirrortarballs[0])) # Fetch and unpack with both the clonedir and shallow tarball available bb.utils.remove(self.gitdir, recurse=True) + self.d.setVar('BB_GIT_SHALLOW', '1') fetcher, ud = self.fetch_and_unpack() # The unpacked tree should *not* be shallow @@ -2097,6 +2284,29 @@ class GitShallowTest(FetcherTest): self.fetch_and_unpack() self.assertRevCount(1) + @skipIfNoNetwork() + def test_shallow_mirrors_with_multiple_same_urls(self): + url = "git://git.openembedded.org/bitbake;branch=master;protocol=https;rev=82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40" + + d = self.d.createCopy() + d.delVar('AUTOREV') + d.delVar('SRCREV') + + # prepare premirror + premirror = os.path.join(self.tempdir, "premirror") + os.mkdir(premirror) + d.setVar("DL_DIR", premirror) + fetcher = bb.fetch.Fetch([url], d) + fetcher.download() + + # set PREMIRRORS + d.setVar('PREMIRRORS', 'git://.*/.* file://%s/' % premirror) + + # set DL_DIR back and use the same url multiple times to fetch + d.setVar("DL_DIR", self.dldir) + fetcher2 = bb.fetch.Fetch([url, url], d) + fetcher2.download() + def test_shallow_invalid_depth(self): self.add_empty_file('a') self.add_empty_file('b') @@ -2140,6 +2350,36 @@ class GitShallowTest(FetcherTest): self.assertRefs(['master', 'origin/master', 'v1.0']) self.assertRevCount(1) + def test_shallow_extra_refs_wildcard_shell_quoted(self): + self.add_empty_file('a') + marker = os.path.join(self.tempdir, 'ref-command-marker') + ref = 'refs/tags/poc;touch${IFS}%s' % marker + self.git(['update-ref', ref, 'HEAD'], cwd=self.srcdir) + + self.d.setVar('BB_GIT_SHALLOW_EXTRA_REFS', 'refs/tags/*') + self.fetch_shallow() + + self.assertFalse(os.path.exists(marker)) + self.assertRefs(['master', 'origin/master', ref]) + + def test_shallow_extra_refs_wildcard_fetch_options(self): + self.add_empty_file('a') + marker = os.path.join(self.tempdir, 'ref-option-marker') + helper = os.path.join(self.tempdir, 'upload-pack-helper') + with open(helper, 'w') as f: + f.write('#!/bin/sh\n') + f.write('touch "%s"\n' % marker) + f.write('exec git-upload-pack "$@"\n') + os.chmod(helper, 0o755) + ref = 'refs/tags/--upload-pack=%s' % helper + self.git(['update-ref', ref, 'HEAD'], cwd=self.srcdir) + + self.d.setVar('BB_GIT_SHALLOW_EXTRA_REFS', 'refs/tags/*') + self.fetch_shallow() + + self.assertFalse(os.path.exists(marker)) + self.assertRefs(['master', 'origin/master', ref]) + def test_shallow_missing_extra_refs(self): self.add_empty_file('a') self.add_empty_file('b') @@ -2175,7 +2415,7 @@ class GitShallowTest(FetcherTest): self.fetch_shallow() - self.assertRevCount(5) + self.assertRevCount(2) def test_shallow_invalid_revs(self): self.add_empty_file('a') @@ -2194,7 +2434,10 @@ class GitShallowTest(FetcherTest): self.git('tag v0.0 master', cwd=self.srcdir) self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0') self.d.setVar('BB_GIT_SHALLOW_REVS', 'v0.0') - self.fetch_shallow() + + with self.assertRaises(bb.fetch2.FetchError), self.assertLogs("BitBake.Fetcher", level="ERROR") as cm: + self.fetch_shallow() + self.assertIn("fatal: no commits selected for shallow requests", cm.output[0]) def test_shallow_fetch_missing_revs_fails(self): self.add_empty_file('a') @@ -2207,9 +2450,36 @@ class GitShallowTest(FetcherTest): self.fetch_shallow() self.assertIn("Unable to find revision v0.0 even from upstream", cm.output[0]) + @skipIfNoNetwork() + def test_git_shallow_fetch_premirrors(self): + url = "git://git.openembedded.org/bitbake;branch=master;protocol=https" + + # Create a separate premirror directory within tempdir + premirror = os.path.join(self.tempdir, "premirror") + os.mkdir(premirror) + + # Fetch a non-shallow clone into the premirror subdir + self.d.setVar('BB_GIT_SHALLOW', '0') + self.d.setVar("DL_DIR", premirror) + fetcher, ud = self.fetch(url) + + # Fetch a shallow clone from the premirror subdir with unpacking + # using the original recipe URL and the premirror mapping + self.d.setVar('BB_GIT_SHALLOW', '1') + self.d.setVar("DL_DIR", self.dldir) + self.d.setVar('BB_FETCH_PREMIRRORONLY', '1') + self.d.setVar('BB_NO_NETWORK', '1') + self.d.setVar('BB_GENERATE_MIRROR_TARBALLS', '0') + self.d.setVar("PREMIRRORS", "git://.*/.* git://{0};protocol=file".format(premirror + "/git2/" + ud.host + ud.path.replace("/", "."))) + fetcher = self.fetch_and_unpack(url) + + # Verify that the unpacked sources are shallow clones + self.assertRevCount(1) + assert os.path.exists(os.path.join(self.gitdir, '.git', 'shallow')) + @skipIfNoNetwork() def test_bitbake(self): - self.git('remote add --mirror=fetch origin https://github.com/openembedded/bitbake', cwd=self.srcdir) + self.git('remote add --mirror=fetch origin https://git.openembedded.org/bitbake-contrib', cwd=self.srcdir) self.git('config core.bare true', cwd=self.srcdir) self.git('fetch', cwd=self.srcdir) @@ -2225,7 +2495,7 @@ class GitShallowTest(FetcherTest): revs = len(self.git('rev-list master').splitlines()) self.assertNotEqual(orig_revs, revs) self.assertRefs(['master', 'origin/master']) - self.assertRevCount(orig_revs - 1758) + self.assertRevCount(orig_revs - 1760) def test_that_unpack_throws_an_error_when_the_git_clone_nor_shallow_tarball_exist(self): self.add_empty_file('a') @@ -2239,23 +2509,33 @@ class GitShallowTest(FetcherTest): self.assertIn("No up to date source found", context.exception.msg) self.assertIn("clone directory not available or not up to date", context.exception.msg) - @skipIfNoNetwork() - def test_that_unpack_does_work_when_using_git_shallow_tarball_but_tarball_is_not_available(self): - self.d.setVar('SRCREV', 'e5939ff608b95cdd4d0ab0e1935781ab9a276ac0') - self.d.setVar('BB_GIT_SHALLOW', '1') - self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1') - fetcher = bb.fetch.Fetch(["git://git.yoctoproject.org/fstests;branch=master;protocol=https"], self.d) - fetcher.download() + def test_shallow_check_is_shallow(self): + self.add_empty_file('a') + self.add_empty_file('b') - bb.utils.remove(self.dldir + "/*.tar.gz") - fetcher.unpack(self.unpackdir) + # Fetch and unpack without the clonedir and *only* shallow tarball available + bb.utils.remove(self.gitdir, recurse=True) + fetcher, ud = self.fetch_and_unpack() + + # The unpacked tree *should* be shallow + self.assertRevCount(1) + assert os.path.exists(os.path.join(self.gitdir, '.git', 'shallow')) + + def test_shallow_succeeds_with_tag_containing_slash(self): + self.add_empty_file('a') + self.add_empty_file('b') + self.git('tag t1/t2/t3', cwd=self.srcdir) + self.assertRevCount(2, cwd=self.srcdir) - dir = os.listdir(self.unpackdir + "/git/") - self.assertIn("fstests.doap", dir) + srcrev = self.git('rev-parse HEAD', cwd=self.srcdir).strip() + self.d.setVar('SRCREV', srcrev) + uri = self.d.getVar('SRC_URI').split()[0] + uri = '%s;tag=t1/t2/t3' % uri + self.fetch_shallow(uri) + self.assertRevCount(1) class GitLfsTest(FetcherTest): def skipIfNoGitLFS(): - import shutil if not shutil.which('git-lfs'): return unittest.skip('git-lfs not installed') return lambda f: f @@ -2279,12 +2559,18 @@ class GitLfsTest(FetcherTest): self.git_init(cwd=self.srcdir) self.commit_file('.gitattributes', '*.mp3 filter=lfs -text') - def commit_file(self, filename, content): - with open(os.path.join(self.srcdir, filename), "w") as f: + def commit(self, *, cwd=None): + cwd = cwd or self.srcdir + self.git(["commit", "-m", "Change"], cwd=cwd) + return self.git(["rev-parse", "HEAD"], cwd=cwd).strip() + + def commit_file(self, filename, content, *, cwd=None): + cwd = cwd or self.srcdir + + with open(os.path.join(cwd, filename), "w") as f: f.write(content) - self.git(["add", filename], cwd=self.srcdir) - self.git(["commit", "-m", "Change"], cwd=self.srcdir) - return self.git(["rev-parse", "HEAD"], cwd=self.srcdir).strip() + self.git(["add", filename], cwd=cwd) + return self.commit(cwd=cwd) def fetch(self, uri=None, download=True): uris = self.d.getVar('SRC_URI').split() @@ -2304,26 +2590,113 @@ class GitLfsTest(FetcherTest): unpacked_lfs_file = os.path.join(self.d.getVar('WORKDIR'), 'git', "Cat_poster_1.jpg") return unpacked_lfs_file + @skipIfNoGitLFS() + def test_gitsm_lfs(self): + """Test that the gitsm fetcher caches objects stored via LFS""" + self.git(["lfs", "install", "--local"], cwd=self.srcdir) + + def fetch_and_verify(revision, filename, content): + self.d.setVar('SRCREV', revision) + fetcher, ud = self.fetch() + + with hide_directory(submoduledir), hide_directory(self.srcdir): + workdir = self.d.getVar('WORKDIR') + fetcher.unpack(workdir) + + with open(os.path.join(workdir, "git", filename)) as f: + self.assertEqual(f.read(), content) + + # Create the git repository that will later be used as a submodule + submoduledir = self.tempdir + "/submodule" + bb.utils.mkdirhier(submoduledir) + self.git_init(submoduledir) + self.git(["lfs", "install", "--local"], cwd=submoduledir) + self.commit_file('.gitattributes', '*.mp3 filter=lfs -text', cwd=submoduledir) + + submodule_commit_1 = self.commit_file("a.mp3", "submodule version 1", cwd=submoduledir) + _ = self.commit_file("a.mp3", "submodule version 2", cwd=submoduledir) + + # Add the submodule to the repository at its current HEAD revision + self.git(["-c", "protocol.file.allow=always", "submodule", "add", submoduledir, "submodule"], + cwd=self.srcdir) + base_commit_1 = self.commit() + + # Let the submodule point at a different revision + self.git(["checkout", submodule_commit_1], self.srcdir + "/submodule") + self.git(["add", "submodule"], cwd=self.srcdir) + base_commit_2 = self.commit() + + # Add a LFS file to the repository + base_commit_3 = self.commit_file("a.mp3", "version 1") + # Update the added LFS file + base_commit_4 = self.commit_file("a.mp3", "version 2") + + self.d.setVar('SRC_URI', "gitsm://%s;protocol=file;lfs=1;branch=master" % self.srcdir) + + # Verify that LFS objects referenced from submodules are fetched and checked out + fetch_and_verify(base_commit_1, "submodule/a.mp3", "submodule version 2") + # Verify that the repository inside the download cache of a submodile is extended with any + # additional LFS objects needed when checking out a different revision. + fetch_and_verify(base_commit_2, "submodule/a.mp3", "submodule version 1") + # Verify that LFS objects referenced from the base repository are fetched and checked out + fetch_and_verify(base_commit_3, "a.mp3", "version 1") + # Verify that the cached repository is extended with any additional LFS objects required + # when checking out a different revision. + fetch_and_verify(base_commit_4, "a.mp3", "version 2") + + @skipIfNoGitLFS() + def test_gitsm_lfs_disabled(self): + """Test that the gitsm fetcher does not use LFS when explicitly disabled""" + self.git(["lfs", "install", "--local"], cwd=self.srcdir) + + def fetch_and_verify(revision, filename, content): + self.d.setVar('SRCREV', revision) + fetcher, ud = self.fetch() + + with hide_directory(submoduledir), hide_directory(self.srcdir): + workdir = self.d.getVar('WORKDIR') + fetcher.unpack(workdir) + + with open(os.path.join(workdir, "git", filename)) as f: + # Assume that LFS did not perform smudging when the expected content is + # missing. + self.assertNotEqual(f.read(), content) + + # Create the git repository that will later be used as a submodule + submoduledir = self.tempdir + "/submodule" + bb.utils.mkdirhier(submoduledir) + self.git_init(submoduledir) + self.git(["lfs", "install", "--local"], cwd=submoduledir) + self.commit_file('.gitattributes', '*.mp3 filter=lfs -text', cwd=submoduledir) + + submodule_commit_1 = self.commit_file("a.mp3", "submodule version 1", cwd=submoduledir) + + # Add the submodule to the repository at its current HEAD revision + self.git(["-c", "protocol.file.allow=always", "submodule", "add", submoduledir, "submodule"], + cwd=self.srcdir) + base_commit_1 = self.commit() + + # Add a LFS file to the repository + base_commit_2 = self.commit_file("a.mp3", "version 1") + + self.d.setVar('SRC_URI', "gitsm://%s;protocol=file;lfs=1;branch=master;lfs=0" % self.srcdir) + + # Verify that LFS objects referenced from submodules are not fetched nor checked out + fetch_and_verify(base_commit_1, "submodule/a.mp3", "submodule version 1") + # Verify that the LFS objects referenced from the base repository are not fetched nor + # checked out + fetch_and_verify(base_commit_2, "a.mp3", "version 1") + @skipIfNoGitLFS() def test_fetch_lfs_on_srcrev_change(self): """Test if fetch downloads missing LFS objects when a different revision within an existing repository is requested""" self.git(["lfs", "install", "--local"], cwd=self.srcdir) - @contextlib.contextmanager - def hide_upstream_repository(): - """Hide the upstream repository to make sure that git lfs cannot pull from it""" - temp_name = self.srcdir + ".bak" - os.rename(self.srcdir, temp_name) - try: - yield - finally: - os.rename(temp_name, self.srcdir) - def fetch_and_verify(revision, filename, content): self.d.setVar('SRCREV', revision) fetcher, ud = self.fetch() - with hide_upstream_repository(): + with hide_directory(self.srcdir): workdir = self.d.getVar('WORKDIR') fetcher.unpack(workdir) @@ -2375,8 +2748,6 @@ class GitLfsTest(FetcherTest): @skipIfNoGitLFS() def test_lfs_enabled(self): - import shutil - uri = 'git://%s;protocol=file;lfs=1;branch=master' % self.srcdir self.d.setVar('SRC_URI', uri) @@ -2387,8 +2758,6 @@ class GitLfsTest(FetcherTest): @skipIfNoGitLFS() def test_lfs_disabled(self): - import shutil - uri = 'git://%s;protocol=file;lfs=0;branch=master' % self.srcdir self.d.setVar('SRC_URI', uri) @@ -2397,58 +2766,76 @@ class GitLfsTest(FetcherTest): fetcher, ud = self.fetch() fetcher.unpack(self.d.getVar('WORKDIR')) - def test_lfs_enabled_not_installed(self): - import shutil - + @skipIfNoGitLFS() + def test_lfs_enabled_not_installed_during_unpack(self): uri = 'git://%s;protocol=file;lfs=1;branch=master' % self.srcdir self.d.setVar('SRC_URI', uri) # Careful: suppress initial attempt at downloading fetcher, ud = self.fetch(uri=None, download=False) - # Artificially assert that git-lfs is not installed, so - # we can verify a failure to unpack in it's absence. - old_find_git_lfs = ud.method._find_git_lfs - try: - # If git-lfs cannot be found, the unpack should throw an error - with self.assertRaises(bb.fetch2.FetchError): - fetcher.download() - ud.method._find_git_lfs = lambda d: False + fetcher.download() + # If git-lfs cannot be found, the unpack should throw an error + with self.assertRaises(bb.fetch2.FetchError): + with unittest.mock.patch("shutil.which", return_value=None): shutil.rmtree(self.gitdir, ignore_errors=True) fetcher.unpack(self.d.getVar('WORKDIR')) - finally: - ud.method._find_git_lfs = old_find_git_lfs - - def test_lfs_disabled_not_installed(self): - import shutil - uri = 'git://%s;protocol=file;lfs=0;branch=master' % self.srcdir + def test_lfs_enabled_not_installed(self): + uri = 'git://%s;protocol=file;lfs=1;branch=master' % self.srcdir self.d.setVar('SRC_URI', uri) # Careful: suppress initial attempt at downloading fetcher, ud = self.fetch(uri=None, download=False) - # Artificially assert that git-lfs is not installed, so - # we can verify a failure to unpack in it's absence. - old_find_git_lfs = ud.method._find_git_lfs - try: - # Even if git-lfs cannot be found, the unpack should be successful + # If git-lfs cannot be found, the download should throw an error + with unittest.mock.patch("shutil.which", return_value=None): + with self.assertRaises(bb.fetch2.FetchError): + fetcher.download() + + def test_lfs_disabled_not_installed(self): + uri = 'git://%s;protocol=file;lfs=0;branch=master' % self.srcdir + self.d.setVar('SRC_URI', uri) + + # Careful: suppress initial attempt at downloading + fetcher, ud = self.fetch(uri=None, download=False) + + # Even if git-lfs cannot be found, the download / unpack should be successful + with unittest.mock.patch("shutil.which", return_value=None): + fetcher.download() + shutil.rmtree(self.gitdir, ignore_errors=True) + fetcher.unpack(self.d.getVar('WORKDIR')) + + def test_lfs_enabled_not_installed_but_not_needed(self): + srcdir = os.path.join(self.tempdir, "emptygit") + bb.utils.mkdirhier(srcdir) + self.git_init(srcdir) + self.commit_file("test", "test content", cwd=srcdir) + + uri = 'git://%s;protocol=file;lfs=1;branch=master' % srcdir + self.d.setVar('SRC_URI', uri) + + # Careful: suppress initial attempt at downloading + fetcher, ud = self.fetch(uri=None, download=False) + + # It shouldnt't matter that git-lfs cannot be found as the repository configuration does not + # specify any LFS filters. + with unittest.mock.patch("shutil.which", return_value=None): fetcher.download() - ud.method._find_git_lfs = lambda d: False shutil.rmtree(self.gitdir, ignore_errors=True) fetcher.unpack(self.d.getVar('WORKDIR')) - finally: - ud.method._find_git_lfs = old_find_git_lfs class GitURLWithSpacesTest(FetcherTest): test_git_urls = { "git://tfs-example.org:22/tfs/example%20path/example.git;branch=master" : { 'url': 'git://tfs-example.org:22/tfs/example%20path/example.git;branch=master', + 'repo_url': 'git://tfs-example.org:22/tfs/example%20path/example.git', 'gitsrcname': 'tfs-example.org.22.tfs.example_path.example.git', 'path': '/tfs/example path/example.git' }, "git://tfs-example.org:22/tfs/example%20path/example%20repo.git;branch=master" : { 'url': 'git://tfs-example.org:22/tfs/example%20path/example%20repo.git;branch=master', + 'repo_url': 'git://tfs-example.org:22/tfs/example%20path/example%20repo.git', 'gitsrcname': 'tfs-example.org.22.tfs.example_path.example_repo.git', 'path': '/tfs/example path/example repo.git' } @@ -2471,6 +2858,51 @@ class GitURLWithSpacesTest(FetcherTest): self.assertEqual(ud.lockfile, os.path.join(self.dldir, "git2", ref['gitsrcname'] + '.lock')) self.assertEqual(ud.clonedir, os.path.join(self.dldir, "git2", ref['gitsrcname'])) self.assertEqual(ud.fullmirror, os.path.join(self.dldir, "git2_" + ref['gitsrcname'] + '.tar.gz')) + self.assertEqual(ud.method._get_repo_url(ud), ref['repo_url']) + + +class FetchLocallyMissingTagFromRemote(FetcherTest): + def setUp(self): + FetcherTest.setUp(self) + self.gitdir = os.path.join(self.tempdir, 'git') + self.srcdir = os.path.join(self.tempdir, 'gitsource') + + bb.utils.mkdirhier(self.srcdir) + self.git_init(cwd=self.srcdir) + self.d.setVar('WORKDIR', self.tempdir) + self.d.setVar('S', self.gitdir) + + uri = 'git://%s;protocol=file;subdir=${S};branch=master' % self.srcdir + self.d.setVar('SRC_URI', uri) + + open(os.path.join(self.srcdir, 'dummyfile'), 'w').close() + self.git(['add', 'dummyfile'], self.srcdir) + self.git(['commit', '-m', 'dummymsg', 'dummyfile'], self.srcdir) + + def _fetch_and_unpack(self, uri_to_fetch): + fetcher = bb.fetch2.Fetch([uri_to_fetch], self.d) + fetcher.download() + fetcher.unpack(self.d.getVar('WORKDIR')) + + def test_tag_present_in_remote_but_not_local(self): + # fetch a repo that has no tag in it + # then add a tag to this repo, and fetch it again, without + # changing SRC_REV, but by adding ';tag=tag1` to SRC_URI + # the new tag should be fetched and unpacked + srcrev = self.git('rev-parse HEAD', cwd=self.srcdir).strip() + self.d.setVar('SRCREV', srcrev) + src_uri = self.d.getVar('SRC_URI') + self._fetch_and_unpack(src_uri) + + self.git('tag -m -a tag1', cwd=self.srcdir) + + src_uri = '%s;tag=tag1' % self.d.getVar('SRC_URI').split()[0] + self.d.setVar('SRC_URI', src_uri) + self._fetch_and_unpack(src_uri) + + output = self.git('log --pretty=oneline -n 1 refs/tags/tag1', cwd=self.gitdir) + assert "fatal: ambiguous argument" not in output + class CrateTest(FetcherTest): @skipIfNoNetwork() @@ -2592,7 +3024,7 @@ class CrateTest(FetcherTest): class NPMTest(FetcherTest): def skipIfNoNpm(): - import shutil + return unittest.skip('npm disabled due to security issues') if not shutil.which('npm'): return unittest.skip('npm not installed') return lambda f: f @@ -2600,8 +3032,8 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0' - fetcher = bb.fetch.Fetch([url], self.d) + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] + fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) @@ -2614,9 +3046,9 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_bad_checksum(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0' + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] # Fetch once to get a tarball - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) @@ -2633,9 +3065,9 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_premirrors(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0' + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] # Fetch once to get a tarball - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) @@ -2655,7 +3087,7 @@ class NPMTest(FetcherTest): # while the fetcher object exists, which it does when we rename the # download directory to "mirror" above. Thus we need a new fetcher to go # with the now empty download directory. - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) @@ -2663,9 +3095,9 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_premirrors_with_specified_filename(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0' + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] # Fetch once to get a tarball - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) @@ -2685,8 +3117,8 @@ class NPMTest(FetcherTest): @skipIfNoNetwork() def test_npm_mirrors(self): # Fetch once to get a tarball - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0' - fetcher = bb.fetch.Fetch([url], self.d) + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] + fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) @@ -2710,8 +3142,8 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_destsuffix_downloadfilename(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0;destsuffix=foo/bar;downloadfilename=foo-bar.tgz' - fetcher = bb.fetch.Fetch([url], self.d) + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0;destsuffix=foo/bar;downloadfilename=foo-bar.tgz'] + fetcher = bb.fetch.Fetch(urls, self.d) fetcher.download() self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'foo-bar.tgz'))) fetcher.unpack(self.unpackdir) @@ -2719,18 +3151,19 @@ class NPMTest(FetcherTest): self.assertTrue(os.path.exists(os.path.join(unpackdir, 'package.json'))) def test_npm_no_network_no_tarball(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0' + return unittest.skip('npm disabled due to security issues') + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] self.d.setVar('BB_NO_NETWORK', '1') - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) with self.assertRaises(bb.fetch2.NetworkAccess): fetcher.download() @skipIfNoNpm() @skipIfNoNetwork() def test_npm_no_network_with_tarball(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0' + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] # Fetch once to get a tarball - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) fetcher.download() # Disable network access self.d.setVar('BB_NO_NETWORK', '1') @@ -2743,8 +3176,8 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_registry_alternate(self): - url = 'npm://skimdb.npmjs.com;package=@savoirfairelinux/node-server-example;version=1.0.0' - fetcher = bb.fetch.Fetch([url], self.d) + urls = ['npm://skimdb.npmjs.com;package=@savoirfairelinux/node-server-example;version=1.0.0'] + fetcher = bb.fetch.Fetch(urls, self.d) fetcher.download() fetcher.unpack(self.unpackdir) unpackdir = os.path.join(self.unpackdir, 'npm') @@ -2753,8 +3186,8 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_version_latest(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=latest' - fetcher = bb.fetch.Fetch([url], self.d) + url = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=latest'] + fetcher = bb.fetch.Fetch(url, self.d) fetcher.download() fetcher.unpack(self.unpackdir) unpackdir = os.path.join(self.unpackdir, 'npm') @@ -2763,46 +3196,46 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_registry_invalid(self): - url = 'npm://registry.invalid.org;package=@savoirfairelinux/node-server-example;version=1.0.0' - fetcher = bb.fetch.Fetch([url], self.d) + urls = ['npm://registry.invalid.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] + fetcher = bb.fetch.Fetch(urls, self.d) with self.assertRaises(bb.fetch2.FetchError): fetcher.download() @skipIfNoNpm() @skipIfNoNetwork() def test_npm_package_invalid(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/invalid;version=1.0.0' - fetcher = bb.fetch.Fetch([url], self.d) + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/invalid;version=1.0.0'] + fetcher = bb.fetch.Fetch(urls, self.d) with self.assertRaises(bb.fetch2.FetchError): fetcher.download() @skipIfNoNpm() @skipIfNoNetwork() def test_npm_version_invalid(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=invalid' + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=invalid'] with self.assertRaises(bb.fetch2.ParameterError): - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) @skipIfNoNpm() @skipIfNoNetwork() def test_npm_registry_none(self): - url = 'npm://;package=@savoirfairelinux/node-server-example;version=1.0.0' + urls = ['npm://;package=@savoirfairelinux/node-server-example;version=1.0.0'] with self.assertRaises(bb.fetch2.MalformedUrl): - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) @skipIfNoNpm() @skipIfNoNetwork() def test_npm_package_none(self): - url = 'npm://registry.npmjs.org;version=1.0.0' + urls = ['npm://registry.npmjs.org;version=1.0.0'] with self.assertRaises(bb.fetch2.MissingParameterError): - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) @skipIfNoNpm() @skipIfNoNetwork() def test_npm_version_none(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example' + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example'] with self.assertRaises(bb.fetch2.MissingParameterError): - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) def create_shrinkwrap_file(self, data): import json @@ -2811,32 +3244,31 @@ class NPMTest(FetcherTest): bb.utils.mkdirhier(datadir) with open(swfile, 'w') as f: json.dump(data, f) - # Also configure the S directory - self.sdir = os.path.join(self.unpackdir, 'S') - self.d.setVar('S', self.sdir) return swfile - @skipIfNoNpm() @skipIfNoNetwork() def test_npmsw(self): + return unittest.skip('npm disabled due to security issues') swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=', 'dependencies': { - 'content-type': { - 'version': 'https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz', - 'integrity': 'sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==', - 'dependencies': { - 'cookie': { - 'version': 'git+https://github.com/jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09', - 'from': 'git+https://github.com/jshttp/cookie.git' - } - } - } + 'content-type': "1.0.4" + } + }, + 'node_modules/array-flatten/node_modules/content-type': { + 'version': '1.0.4', + 'resolved': 'https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz', + 'integrity': 'sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==', + 'dependencies': { + 'cookie': 'git+https://git.yoctoproject.org/bbfetchtests-jshttp-cookie#aec1177c7da67e3b3273df96cf476824dbc9ae09' } + }, + 'node_modules/array-flatten/node_modules/content-type/node_modules/cookie': { + 'resolved': 'git+https://git.yoctoproject.org/bbfetchtests-jshttp-cookie#aec1177c7da67e3b3273df96cf476824dbc9ae09' } } }) @@ -2844,63 +3276,38 @@ class NPMTest(FetcherTest): fetcher.download() self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'array-flatten-1.1.1.tgz'))) self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'content-type-1.0.4.tgz'))) - self.assertTrue(os.path.exists(os.path.join(self.dldir, 'git2', 'github.com.jshttp.cookie.git'))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, 'git2', 'git.yoctoproject.org.bbfetchtests-jshttp-cookie'))) fetcher.unpack(self.unpackdir) - self.assertTrue(os.path.exists(os.path.join(self.sdir, 'npm-shrinkwrap.json'))) - self.assertTrue(os.path.exists(os.path.join(self.sdir, 'node_modules', 'array-flatten', 'package.json'))) - self.assertTrue(os.path.exists(os.path.join(self.sdir, 'node_modules', 'array-flatten', 'node_modules', 'content-type', 'package.json'))) - self.assertTrue(os.path.exists(os.path.join(self.sdir, 'node_modules', 'array-flatten', 'node_modules', 'content-type', 'node_modules', 'cookie', 'package.json'))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'npm-shrinkwrap.json'))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'node_modules', 'array-flatten', 'package.json'))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'node_modules', 'array-flatten', 'node_modules', 'content-type', 'package.json'))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'node_modules', 'array-flatten', 'node_modules', 'content-type', 'node_modules', 'cookie', 'package.json'))) - @skipIfNoNpm() @skipIfNoNetwork() def test_npmsw_git(self): + return unittest.skip('npm disabled due to security issues') swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'cookie': { - 'version': 'github:jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09', - 'from': 'github:jshttp/cookie.git' + 'packages': { + 'node_modules/cookie': { + 'resolved': 'git+https://git.yoctoproject.org/bbfetchtests-jshttp-cookie#aec1177c7da67e3b3273df96cf476824dbc9ae09' } } }) fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) fetcher.download() - self.assertTrue(os.path.exists(os.path.join(self.dldir, 'git2', 'github.com.jshttp.cookie.git'))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, 'git2', 'git.yoctoproject.org.bbfetchtests-jshttp-cookie'))) - swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'cookie': { - 'version': 'jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09', - 'from': 'jshttp/cookie.git' - } - } - }) - fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) - fetcher.download() - self.assertTrue(os.path.exists(os.path.join(self.dldir, 'git2', 'github.com.jshttp.cookie.git'))) - - swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'nodejs': { - 'version': 'gitlab:gitlab-examples/nodejs.git#892a1f16725e56cc3a2cb0d677be42935c8fc262', - 'from': 'gitlab:gitlab-examples/nodejs' - } - } - }) - fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) - fetcher.download() - self.assertTrue(os.path.exists(os.path.join(self.dldir, 'git2', 'gitlab.com.gitlab-examples.nodejs.git'))) - - @skipIfNoNpm() @skipIfNoNetwork() def test_npmsw_dev(self): + return unittest.skip('npm disabled due to security issues') swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' }, - 'content-type': { + 'node_modules/content-type': { 'version': '1.0.4', 'resolved': 'https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz', 'integrity': 'sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==', @@ -2919,12 +3326,12 @@ class NPMTest(FetcherTest): self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'array-flatten-1.1.1.tgz'))) self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'content-type-1.0.4.tgz'))) - @skipIfNoNpm() @skipIfNoNetwork() def test_npmsw_destsuffix(self): + return unittest.skip('npm disabled due to security issues') swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' @@ -2937,9 +3344,10 @@ class NPMTest(FetcherTest): self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'foo', 'bar', 'node_modules', 'array-flatten', 'package.json'))) def test_npmsw_no_network_no_tarball(self): + return unittest.skip('npm disabled due to security issues') swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' @@ -2961,8 +3369,8 @@ class NPMTest(FetcherTest): self.d.setVar('BB_NO_NETWORK', '1') # Fetch again swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' @@ -2972,15 +3380,15 @@ class NPMTest(FetcherTest): fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) fetcher.download() fetcher.unpack(self.unpackdir) - self.assertTrue(os.path.exists(os.path.join(self.sdir, 'node_modules', 'array-flatten', 'package.json'))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'node_modules', 'array-flatten', 'package.json'))) - @skipIfNoNpm() @skipIfNoNetwork() def test_npmsw_npm_reusability(self): + return unittest.skip('npm disabled due to security issues') # Fetch once with npmsw swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' @@ -2997,13 +3405,13 @@ class NPMTest(FetcherTest): fetcher.unpack(self.unpackdir) self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'npm', 'package.json'))) - @skipIfNoNpm() @skipIfNoNetwork() def test_npmsw_bad_checksum(self): + return unittest.skip('npm disabled due to security issues') # Try to fetch with bad checksum swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-gfNEp2hqgLTFKT6P3AsBYMgsBqg=' @@ -3015,8 +3423,8 @@ class NPMTest(FetcherTest): fetcher.download() # Fetch correctly to get a tarball swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' @@ -3054,8 +3462,8 @@ class NPMTest(FetcherTest): # Fetch again self.assertFalse(os.path.exists(ud.localpath)) swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' @@ -3082,8 +3490,8 @@ class NPMTest(FetcherTest): # Fetch again with invalid url self.assertFalse(os.path.exists(ud.localpath)) swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://invalid', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' @@ -3094,6 +3502,29 @@ class NPMTest(FetcherTest): fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) + @skipIfNoNetwork() + def test_npmsw_bundled(self): + return unittest.skip('npm disabled due to security issues') + swfile = self.create_shrinkwrap_file({ + 'packages': { + 'node_modules/array-flatten': { + 'version': '1.1.1', + 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', + 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' + }, + 'node_modules/content-type': { + 'version': '1.0.4', + 'resolved': 'https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz', + 'integrity': 'sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==', + 'inBundle': True + } + } + }) + fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) + fetcher.download() + self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'array-flatten-1.1.1.tgz'))) + self.assertFalse(os.path.exists(os.path.join(self.dldir, 'npm2', 'content-type-1.0.4.tgz'))) + class GitSharedTest(FetcherTest): def setUp(self): super(GitSharedTest, self).setUp() @@ -3121,6 +3552,72 @@ class GitSharedTest(FetcherTest): alt = os.path.join(self.unpackdir, 'git/.git/objects/info/alternates') self.assertFalse(os.path.exists(alt)) +class GitTagVerificationTests(FetcherTest): + + @skipIfNoNetwork() + def test_tag_rev_match(self): + # Test a url with rev= and tag= set works + fetcher = bb.fetch.Fetch(["git://git.openembedded.org/bitbake;branch=2.12;protocol=https;rev=5b4e20377eea8d428edf1aeb2187c18f82ca6757;tag=2.12.0"], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + def test_annotated_tag_rev_match(self): + # Test a url with rev= and tag= set works + # rev is the annotated tag revision in this case + fetcher = bb.fetch.Fetch(["git://git.openembedded.org/bitbake;branch=2.12;protocol=https;rev=fa30183549bd09f33fd4eebf56771ca5393526a6;tag=2.12.0"], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + @skipIfNoNetwork() + def test_tag_rev_match2(self): + # Test a url with SRCREV and tag= set works + self.d.setVar('SRCREV', '5b4e20377eea8d428edf1aeb2187c18f82ca6757') + fetcher = bb.fetch.Fetch(["git://git.openembedded.org/bitbake;branch=2.12;protocol=https;tag=2.12.0"], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + @skipIfNoNetwork() + def test_tag_rev_match3(self): + # Test a url with SRCREV, rev= and tag= set works + self.d.setVar('SRCREV', '5b4e20377eea8d428edf1aeb2187c18f82ca6757') + fetcher = bb.fetch.Fetch(["git://git.openembedded.org/bitbake;branch=2.12;protocol=https;rev=5b4e20377eea8d428edf1aeb2187c18f82ca6757;tag=2.12.0"], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + @skipIfNoNetwork() + def test_tag_rev_match4(self): + # Test a url with SRCREV and rev= mismatching errors + self.d.setVar('SRCREV', 'bade540fc31a1c26839efd2c7785a751ce24ebfb') + with self.assertRaises(bb.fetch2.FetchError): + fetcher = bb.fetch.Fetch(["git://git.openembedded.org/bitbake;branch=2.12;protocol=https;rev=5b4e20377eea8d428edf1aeb2187c18f82ca6757;tag=2.12.0"], self.d) + + @skipIfNoNetwork() + def test_tag_rev_match5(self): + # Test a url with SRCREV, rev= and tag= set works when using shallow clones + self.d.setVar('BB_GIT_SHALLOW', '1') + self.d.setVar('SRCREV', '5b4e20377eea8d428edf1aeb2187c18f82ca6757') + fetcher = bb.fetch.Fetch(["git://git.openembedded.org/bitbake;branch=2.12;protocol=https;rev=5b4e20377eea8d428edf1aeb2187c18f82ca6757;tag=2.12.0"], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + @skipIfNoNetwork() + def test_tag_rev_match6(self): + # Test a url with SRCREV, rev= and a mismatched tag= when using shallow clones + self.d.setVar('BB_GIT_SHALLOW', '1') + fetcher = bb.fetch.Fetch(["git://git.openembedded.org/bitbake;branch=2.12;protocol=https;rev=5b4e20377eea8d428edf1aeb2187c18f82ca6757;tag=2.8.0"], self.d) + fetcher.download() + with self.assertRaises(bb.fetch2.FetchError): + fetcher.unpack(self.unpackdir) + + @skipIfNoNetwork() + def test_tag_rev_match7(self): + # Test a url with SRCREV, rev= and a mismatched tag= + self.d.setVar('SRCREV', '5b4e20377eea8d428edf1aeb2187c18f82ca6757') + fetcher = bb.fetch.Fetch(["git://git.openembedded.org/bitbake;branch=2.12;protocol=https;rev=5b4e20377eea8d428edf1aeb2187c18f82ca6757;tag=2.8.0"], self.d) + fetcher.download() + with self.assertRaises(bb.fetch2.FetchError): + fetcher.unpack(self.unpackdir) + class FetchPremirroronlyLocalTest(FetcherTest): @@ -3203,58 +3700,6 @@ class FetchPremirroronlyLocalTest(FetcherTest): with self.assertRaises(bb.fetch2.NetworkAccess): fetcher.download() - def test_mirror_tarball_multiple_branches(self): - """ - test if PREMIRRORS can handle multiple name/branches correctly - both branches have required revisions - """ - self.make_git_repo() - branch1rev = self.git_new_branch("testbranch1") - branch2rev = self.git_new_branch("testbranch2") - self.recipe_url = "git://git.fake.repo/bitbake;branch=testbranch1,testbranch2;protocol=https;name=branch1,branch2" - self.d.setVar("SRCREV_branch1", branch1rev) - self.d.setVar("SRCREV_branch2", branch2rev) - fetcher = bb.fetch.Fetch([self.recipe_url], self.d) - self.assertTrue(os.path.exists(self.mirrorfile), "Mirror file doesn't exist") - fetcher.download() - fetcher.unpack(os.path.join(self.tempdir, "unpacked")) - unpacked = os.path.join(self.tempdir, "unpacked", "git", self.testfilename) - self.assertTrue(os.path.exists(unpacked), "Repo has not been unpackaged properly!") - with open(unpacked, 'r') as f: - content = f.read() - ## We expect to see testbranch1 in the file, not master, not testbranch2 - self.assertTrue(content.find("testbranch1") != -1, "Wrong branch has been checked out!") - - def test_mirror_tarball_multiple_branches_nobranch(self): - """ - test if PREMIRRORS can handle multiple name/branches correctly - Unbalanced name/branches raises ParameterError - """ - self.make_git_repo() - branch1rev = self.git_new_branch("testbranch1") - branch2rev = self.git_new_branch("testbranch2") - self.recipe_url = "git://git.fake.repo/bitbake;branch=testbranch1;protocol=https;name=branch1,branch2" - self.d.setVar("SRCREV_branch1", branch1rev) - self.d.setVar("SRCREV_branch2", branch2rev) - with self.assertRaises(bb.fetch2.ParameterError): - fetcher = bb.fetch.Fetch([self.recipe_url], self.d) - - def test_mirror_tarball_multiple_branches_norev(self): - """ - test if PREMIRRORS can handle multiple name/branches correctly - one of the branches specifies non existing SRCREV - """ - self.make_git_repo() - branch1rev = self.git_new_branch("testbranch1") - branch2rev = self.git_new_branch("testbranch2") - self.recipe_url = "git://git.fake.repo/bitbake;branch=testbranch1,testbranch2;protocol=https;name=branch1,branch2" - self.d.setVar("SRCREV_branch1", branch1rev) - self.d.setVar("SRCREV_branch2", "0"*40) - fetcher = bb.fetch.Fetch([self.recipe_url], self.d) - self.assertTrue(os.path.exists(self.mirrorfile), "Mirror file doesn't exist") - with self.assertRaises(bb.fetch2.NetworkAccess): - fetcher.download() - class FetchPremirroronlyNetworkTest(FetcherTest): @@ -3265,16 +3710,17 @@ class FetchPremirroronlyNetworkTest(FetcherTest): self.reponame = "fstests" self.clonedir = os.path.join(self.tempdir, "git") self.gitdir = os.path.join(self.tempdir, "git", "{}.git".format(self.reponame)) - self.recipe_url = "git://git.yoctoproject.org/fstests;protocol=https" + self.giturl = "https://git.yoctoproject.org/fstests" + self.recipe_url = "git://git.yoctoproject.org/fstests;protocol=https;branch=master" self.d.setVar("BB_FETCH_PREMIRRORONLY", "1") self.d.setVar("BB_NO_NETWORK", "0") self.d.setVar("PREMIRRORS", self.recipe_url + " " + "file://{}".format(self.mirrordir) + " \n") def make_git_repo(self): - import shutil self.mirrorname = "git2_git.yoctoproject.org.fstests.tar.gz" os.makedirs(self.clonedir) - self.git("clone --bare --shallow-since=\"01.01.2013\" {}".format(self.recipe_url), self.clonedir) + self.git("clone --bare {}".format(self.giturl), self.clonedir) + self.git("update-ref HEAD 15413486df1f5a5b5af699b6f3ba5f0984e52a9f", self.gitdir) bb.process.run('tar -czvf {} .'.format(os.path.join(self.mirrordir, self.mirrorname)), cwd = self.gitdir) shutil.rmtree(self.clonedir) @@ -3282,7 +3728,7 @@ class FetchPremirroronlyNetworkTest(FetcherTest): def test_mirror_tarball_updated(self): self.make_git_repo() ## Upstream commit is in the mirror - self.d.setVar("SRCREV", "49d65d53c2bf558ae6e9185af0f3af7b79d255ec") + self.d.setVar("SRCREV", "15413486df1f5a5b5af699b6f3ba5f0984e52a9f") fetcher = bb.fetch.Fetch([self.recipe_url], self.d) fetcher.download() @@ -3290,7 +3736,7 @@ class FetchPremirroronlyNetworkTest(FetcherTest): def test_mirror_tarball_outdated(self): self.make_git_repo() ## Upstream commit not in the mirror - self.d.setVar("SRCREV", "15413486df1f5a5b5af699b6f3ba5f0984e52a9f") + self.d.setVar("SRCREV", "49d65d53c2bf558ae6e9185af0f3af7b79d255ec") fetcher = bb.fetch.Fetch([self.recipe_url], self.d) with self.assertRaises(bb.fetch2.NetworkAccess): fetcher.download() @@ -3300,7 +3746,6 @@ class FetchPremirroronlyMercurialTest(FetcherTest): the test covers also basic hg:// clone (see fetch_and_create_tarball """ def skipIfNoHg(): - import shutil if not shutil.which('hg'): return unittest.skip('Mercurial not installed') return lambda f: f @@ -3347,7 +3792,7 @@ class FetchPremirroronlyBrokenTarball(FetcherTest): os.mkdir(self.mirrordir) self.reponame = "bitbake" self.gitdir = os.path.join(self.tempdir, "git", self.reponame) - self.recipe_url = "git://git.fake.repo/bitbake;protocol=https" + self.recipe_url = "git://git.fake.repo/bitbake;protocol=https;branch=master" self.d.setVar("BB_FETCH_PREMIRRORONLY", "1") self.d.setVar("BB_NO_NETWORK", "1") self.d.setVar("PREMIRRORS", self.recipe_url + " " + "file://{}".format(self.mirrordir) + " \n") @@ -3356,10 +3801,782 @@ class FetchPremirroronlyBrokenTarball(FetcherTest): targz.write("This is not tar.gz file!") def test_mirror_broken_download(self): - import sys self.d.setVar("SRCREV", "0"*40) fetcher = bb.fetch.Fetch([self.recipe_url], self.d) with self.assertRaises(bb.fetch2.FetchError), self.assertLogs() as logs: fetcher.download() output = "".join(logs.output) self.assertFalse(" not a git repository (or any parent up to mount point /)" in output) + +class GoModTest(FetcherTest): + + @skipIfNoNetwork() + def test_gomod_url(self): + urls = ['gomod://github.com/Azure/azure-sdk-for-go/sdk/storage/azblob;version=v1.0.0;' + 'sha256sum=9bb69aea32f1d59711701f9562d66432c9c0374205e5009d1d1a62f03fb4fdad'] + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.url, 'https://proxy.golang.org/github.com/%21azure/azure-sdk-for-go/sdk/storage/azblob/%40v/v1.0.0.zip') + self.assertEqual(ud.parm['downloadfilename'], 'github.com.Azure.azure-sdk-for-go.sdk.storage.azblob@v1.0.0.zip') + self.assertEqual(ud.parm['name'], 'github.com/Azure/azure-sdk-for-go/sdk/storage/azblob@v1.0.0') + + fetcher.download() + fetcher.unpack(self.unpackdir) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.zip'))) + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.mod'))) + self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.mod')), + '7873b8544842329b4f385a3aa6cf82cc2bc8defb41a04fa5291c35fd5900e873') + + @skipIfNoNetwork() + def test_gomod_url_go_mod_only(self): + urls = ['gomod://github.com/Azure/azure-sdk-for-go/sdk/storage/azblob;version=v1.0.0;mod=1;' + 'sha256sum=7873b8544842329b4f385a3aa6cf82cc2bc8defb41a04fa5291c35fd5900e873'] + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.url, 'https://proxy.golang.org/github.com/%21azure/azure-sdk-for-go/sdk/storage/azblob/%40v/v1.0.0.mod') + self.assertEqual(ud.parm['downloadfilename'], 'github.com.Azure.azure-sdk-for-go.sdk.storage.azblob@v1.0.0.mod') + self.assertEqual(ud.parm['name'], 'github.com/Azure/azure-sdk-for-go/sdk/storage/azblob@v1.0.0') + + fetcher.download() + fetcher.unpack(self.unpackdir) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.mod'))) + + @skipIfNoNetwork() + def test_gomod_url_sha256sum_varflag(self): + urls = ['gomod://gopkg.in/ini.v1;version=v1.67.0'] + self.d.setVarFlag('SRC_URI', 'gopkg.in/ini.v1@v1.67.0.sha256sum', 'bd845dfc762a87a56e5a32a07770dc83e86976db7705d7f89c5dbafdc60b06c6') + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.url, 'https://proxy.golang.org/gopkg.in/ini.v1/%40v/v1.67.0.zip') + self.assertEqual(ud.parm['downloadfilename'], 'gopkg.in.ini.v1@v1.67.0.zip') + self.assertEqual(ud.parm['name'], 'gopkg.in/ini.v1@v1.67.0') + + fetcher.download() + fetcher.unpack(self.unpackdir) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.zip'))) + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod'))) + self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod')), + '13aedd85db8e555104108e0e613bb7e4d1242af7f27c15423dd9ab63b60b72a1') + + @skipIfNoNetwork() + def test_gomod_url_no_go_mod_in_module(self): + urls = ['gomod://gopkg.in/ini.v1;version=v1.67.0;' + 'sha256sum=bd845dfc762a87a56e5a32a07770dc83e86976db7705d7f89c5dbafdc60b06c6'] + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.url, 'https://proxy.golang.org/gopkg.in/ini.v1/%40v/v1.67.0.zip') + self.assertEqual(ud.parm['downloadfilename'], 'gopkg.in.ini.v1@v1.67.0.zip') + self.assertEqual(ud.parm['name'], 'gopkg.in/ini.v1@v1.67.0') + + fetcher.download() + fetcher.unpack(self.unpackdir) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.zip'))) + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod'))) + self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod')), + '13aedd85db8e555104108e0e613bb7e4d1242af7f27c15423dd9ab63b60b72a1') + + @skipIfNoNetwork() + def test_gomod_url_host_only(self): + urls = ['gomod://go.opencensus.io;version=v0.24.0;' + 'sha256sum=203a767d7f8e7c1ebe5588220ad168d1e15b14ae70a636de7ca9a4a88a7e0d0c'] + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.url, 'https://proxy.golang.org/go.opencensus.io/%40v/v0.24.0.zip') + self.assertEqual(ud.parm['downloadfilename'], 'go.opencensus.io@v0.24.0.zip') + self.assertEqual(ud.parm['name'], 'go.opencensus.io@v0.24.0') + + fetcher.download() + fetcher.unpack(self.unpackdir) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'go.opencensus.io/@v/v0.24.0.zip'))) + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'go.opencensus.io/@v/v0.24.0.mod'))) + self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'go.opencensus.io/@v/v0.24.0.mod')), + '0dc9ccc660ad21cebaffd548f2cc6efa27891c68b4fbc1f8a3893b00f1acec96') + +class GoModGitTest(FetcherTest): + + @skipIfNoNetwork() + def test_gomodgit_url_repo(self): + urls = ['gomodgit://golang.org/x/net;version=v0.9.0;' + 'repo=go.googlesource.com/net;' + 'srcrev=694cff8668bac64e0864b552bffc280cd27f21b1'] + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.host, 'go.googlesource.com') + self.assertEqual(ud.path, '/net') + self.assertEqual(ud.name, 'golang.org/x/net@v0.9.0') + self.assertEqual(self.d.getVar('SRCREV_golang.org/x/net@v0.9.0'), '694cff8668bac64e0864b552bffc280cd27f21b1') + + fetcher.download() + self.assertTrue(os.path.exists(ud.localpath)) + + fetcher.unpack(self.unpackdir) + vcsdir = os.path.join(self.unpackdir, 'pkg/mod/cache/vcs') + self.assertTrue(os.path.exists(os.path.join(vcsdir, 'ed42bd05533fd84ae290a5d33ebd3695a0a2b06131beebd5450825bee8603aca'))) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'golang.org/x/net/@v/v0.9.0.zip'))) + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'golang.org/x/net/@v/v0.9.0.mod'))) + self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'golang.org/x/net/@v/v0.9.0.mod')), + 'c5d6851ede50ec1c001afb763040194b68961bf06997e2605e8bf06dcd2aeb2e') + + @skipIfNoNetwork() + def test_gomodgit_url_subdir(self): + urls = ['gomodgit://github.com/Azure/azure-sdk-for-go/sdk/storage/azblob;version=v1.0.0;' + 'repo=github.com/Azure/azure-sdk-for-go;subdir=sdk/storage/azblob;' + 'srcrev=ec928e0ed34db682b3f783d3739d1c538142e0c3'] + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.host, 'github.com') + self.assertEqual(ud.path, '/Azure/azure-sdk-for-go') + self.assertEqual(ud.parm['subpath'], 'sdk/storage/azblob') + self.assertEqual(ud.name, 'github.com/Azure/azure-sdk-for-go/sdk/storage/azblob@v1.0.0') + self.assertEqual(self.d.getVar('SRCREV_github.com/Azure/azure-sdk-for-go/sdk/storage/azblob@v1.0.0'), 'ec928e0ed34db682b3f783d3739d1c538142e0c3') + + fetcher.download() + self.assertTrue(os.path.exists(ud.localpath)) + + fetcher.unpack(self.unpackdir) + vcsdir = os.path.join(self.unpackdir, 'pkg/mod/cache/vcs') + self.assertTrue(os.path.exists(os.path.join(vcsdir, 'd31d6145676ed3066ce573a8198f326dea5be45a43b3d8f41ce7787fd71d66b3'))) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.zip'))) + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.mod'))) + self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.mod')), + '7873b8544842329b4f385a3aa6cf82cc2bc8defb41a04fa5291c35fd5900e873') + + @skipIfNoNetwork() + def test_gomodgit_url_srcrev_var(self): + urls = ['gomodgit://gopkg.in/ini.v1;version=v1.67.0'] + self.d.setVar('SRCREV_gopkg.in/ini.v1@v1.67.0', 'b2f570e5b5b844226bbefe6fb521d891f529a951') + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.host, 'gopkg.in') + self.assertEqual(ud.path, '/ini.v1') + self.assertEqual(ud.name, 'gopkg.in/ini.v1@v1.67.0') + self.assertEqual(ud.parm['srcrev'], 'b2f570e5b5b844226bbefe6fb521d891f529a951') + + fetcher.download() + fetcher.unpack(self.unpackdir) + vcsdir = os.path.join(self.unpackdir, 'pkg/mod/cache/vcs') + self.assertTrue(os.path.exists(os.path.join(vcsdir, 'b7879a4be9ba8598851b8278b14c4f71a8316be64913298d1639cce6bde59bc3'))) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.zip'))) + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod'))) + self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod')), + '13aedd85db8e555104108e0e613bb7e4d1242af7f27c15423dd9ab63b60b72a1') + + @skipIfNoNetwork() + def test_gomodgit_url_no_go_mod_in_module(self): + urls = ['gomodgit://gopkg.in/ini.v1;version=v1.67.0;' + 'srcrev=b2f570e5b5b844226bbefe6fb521d891f529a951'] + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.host, 'gopkg.in') + self.assertEqual(ud.path, '/ini.v1') + self.assertEqual(ud.name, 'gopkg.in/ini.v1@v1.67.0') + self.assertEqual(self.d.getVar('SRCREV_gopkg.in/ini.v1@v1.67.0'), 'b2f570e5b5b844226bbefe6fb521d891f529a951') + + fetcher.download() + fetcher.unpack(self.unpackdir) + vcsdir = os.path.join(self.unpackdir, 'pkg/mod/cache/vcs') + self.assertTrue(os.path.exists(os.path.join(vcsdir, 'b7879a4be9ba8598851b8278b14c4f71a8316be64913298d1639cce6bde59bc3'))) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.zip'))) + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod'))) + self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod')), + '13aedd85db8e555104108e0e613bb7e4d1242af7f27c15423dd9ab63b60b72a1') + + @skipIfNoNetwork() + def test_gomodgit_url_host_only(self): + urls = ['gomodgit://go.opencensus.io;version=v0.24.0;' + 'repo=git.yoctoproject.org/bbfetchtests-census-instrumentation-opencensus-go;' + 'srcrev=b1a01ee95db0e690d91d7193d037447816fae4c5'] + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.host, 'git.yoctoproject.org') + self.assertEqual(ud.path, '/bbfetchtests-census-instrumentation-opencensus-go') + self.assertEqual(ud.name, 'go.opencensus.io@v0.24.0') + self.assertEqual(self.d.getVar('SRCREV_go.opencensus.io@v0.24.0'), 'b1a01ee95db0e690d91d7193d037447816fae4c5') + + fetcher.download() + fetcher.unpack(self.unpackdir) + vcsdir = os.path.join(self.unpackdir, 'pkg/mod/cache/vcs') + self.assertTrue(os.path.exists(os.path.join(vcsdir, '24e4a8be150f9e7343b72e40000115a175fb2f19d015542f54abb6ccdbd52f14'))) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'go.opencensus.io/@v/v0.24.0.zip'))) + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'go.opencensus.io/@v/v0.24.0.mod'))) + self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'go.opencensus.io/@v/v0.24.0.mod')), + '0dc9ccc660ad21cebaffd548f2cc6efa27891c68b4fbc1f8a3893b00f1acec96') + + +class GitUnpackUpdateTest(FetcherTest): + """Test the unpack_update functionality for git fetcher. + + Intended workflow + 1. First-time setup: + 1. download() - clones the upstream repo into DL_DIR/git2/... (clonedir). + 2. unpack() - clones from clonedir into the workspace (S/workdir) and + registers a 'dldir' git remote pointing at + file://DL_DIR/git2/... for later offline use. + + 2. Subsequent updates (what unpack_update is designed for): + 1. The user works in the unpacked source tree. + 2. Upstream advances - SRCREV changes in the recipe. + 3. download() - fetches the new revision into the local clonedir. + 4. unpack_update() - instead of wiping the workspace and re-cloning: + * fetches the new revision from the local 'dldir' remote + * rebases the user's local commits on top of the new SRCREV + * raises LocalModificationsError if uncommitted changes block the + update, RebaseError if local commits cannot be rebased, or a + plain UnpackError for other failures (shallow clone, stale dldir); + in all cases the caller (e.g. bitbake-setup) can fall back to + backup + re-clone. + + Key design constraints: + * unpack_update() never deletes existing data (unlike unpack()). + * Only staged/modified tracked files block the update; untracked files and + committed local work are handled gracefully. + * The 'dldir' remote is intentionally visible to users outside the + fetcher (e.g. for manual 'git log dldir/master'). + * Currently only git is supported. + """ + + def setUp(self): + """Set up a local bare git source repository with two commits on 'master'. + + self.initial_rev - the first commit (testfile.txt: 'initial content') + self.updated_rev - the second commit (testfile.txt: 'updated content') + + SRCREV is initialised to self.initial_rev so individual tests can + advance it to self.updated_rev (or create further commits) as needed. + """ + FetcherTest.setUp(self) + + self.gitdir = os.path.join(self.tempdir, 'gitrepo') + self.srcdir = os.path.join(self.tempdir, 'gitsource') + + self.d.setVar('WORKDIR', self.tempdir) + self.d.setVar('S', self.gitdir) + self.d.delVar('PREMIRRORS') + self.d.delVar('MIRRORS') + + # Create a source git repository + bb.utils.mkdirhier(self.srcdir) + self.git_init(cwd=self.srcdir) + + # Create initial commit + with open(os.path.join(self.srcdir, 'testfile.txt'), 'w') as f: + f.write('initial content\n') + self.git(['add', 'testfile.txt'], cwd=self.srcdir) + self.git(['commit', '-m', 'Initial commit'], cwd=self.srcdir) + self.initial_rev = self.git(['rev-parse', 'HEAD'], cwd=self.srcdir).strip() + + # Create a second commit + with open(os.path.join(self.srcdir, 'testfile.txt'), 'w') as f: + f.write('updated content\n') + self.git(['add', 'testfile.txt'], cwd=self.srcdir) + self.git(['commit', '-m', 'Update commit'], cwd=self.srcdir) + self.updated_rev = self.git(['rev-parse', 'HEAD'], cwd=self.srcdir).strip() + + self.d.setVar('SRCREV', self.initial_rev) + self.d.setVar('SRC_URI', 'git://%s;branch=master;protocol=file' % self.srcdir) + + def test_unpack_update_full_clone(self): + """Test that unpack_update updates an existing checkout in place for a full clone. + + Steps: + 1. Fetch and unpack at self.initial_rev - verify 'initial content'. + 2. Advance SRCREV to self.updated_rev and re-download. + 3. Call unpack_update() instead of unpack() - the existing checkout + must be updated via 'git fetch dldir' + 'git rebase' without + re-cloning the directory. + 4. Verify testfile.txt now contains 'updated content'. + """ + # First fetch at initial revision + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + # Verify initial state + unpack_path = os.path.join(self.unpackdir, 'git') + self.assertTrue(os.path.exists(os.path.join(unpack_path, 'testfile.txt'))) + with open(os.path.join(unpack_path, 'testfile.txt'), 'r') as f: + self.assertEqual(f.read(), 'initial content\n') + + # Update to new revision + self.d.setVar('SRCREV', self.updated_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + # Use unpack_update + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + git_fetcher = ud.method + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + # Verify updated state + with open(os.path.join(unpack_path, 'testfile.txt'), 'r') as f: + self.assertEqual(f.read(), 'updated content\n') + + def test_unpack_update_dldir_remote_setup(self): + """Test that unpack() adds a 'dldir' git remote pointing at ud.clonedir. + + The 'dldir' remote is used by subsequent unpack_update() calls to fetch + new commits from the local download cache (${DL_DIR}/git2/…) without + requiring network access. After a normal unpack the remote must exist + and its URL must be 'file://'. + """ + # First fetch + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + + # Check that dldir remote exists + remotes = self.git(['remote'], cwd=unpack_path).strip().split('\n') + self.assertIn('dldir', remotes) + + # Verify it points to the clonedir + dldir_url = self.git(['remote', 'get-url', 'dldir'], cwd=unpack_path).strip() + self.assertEqual(dldir_url, 'file://{}'.format(ud.clonedir)) + + def test_unpack_update_ff_with_local_changes(self): + """Test that unpack_update rebases local commits fast forward. + + Full workflow: + 1. Fetch + unpack at initial_rev - verify 'dldir' remote is created + pointing at ud.clonedir. + 2. Add a local commit touching localfile.txt. + 3. Advance SRCREV to updated_rev and call download() - verify that + ud.clonedir (the dldir bare clone) now contains updated_rev. + 4. Call unpack_update() - it fetches updated_rev from dldir into the + working tree and rebases the local commit on top. + 5. Verify the final commit graph: HEAD's parent is updated_rev, and + both testfile.txt ('updated content') and localfile.txt ('local + change') are present. + + Note: git rebase operates the same way regardless of whether HEAD is + detached or on a named branch (e.g. 'master' or a local feature branch), + so this test covers those scenarios implicitly. + """ + # Step 1 - fetch + unpack at initial_rev + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + unpack_path = os.path.join(self.unpackdir, 'git') + + # The normal unpack must have set up the 'dldir' remote pointing at + # ud.clonedir so that subsequent unpack_update() calls work offline. + dldir_url = self.git(['remote', 'get-url', 'dldir'], cwd=unpack_path).strip() + self.assertEqual(dldir_url, 'file://{}'.format(ud.clonedir)) + + # Step 2 - add a local commit that touches a new file + with open(os.path.join(unpack_path, 'localfile.txt'), 'w') as f: + f.write('local change\n') + self.git(['add', 'localfile.txt'], cwd=unpack_path) + self.git(['commit', '-m', 'Local commit'], cwd=unpack_path) + local_commit = self.git(['rev-parse', 'HEAD'], cwd=unpack_path).strip() + + # Step 3 - advance SRCREV and download; clonedir must now contain + # updated_rev so that unpack_update can fetch it without network access. + self.d.setVar('SRCREV', self.updated_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + ud = fetcher.ud[uri] + clonedir_refs = self.git(['rev-parse', self.updated_rev], cwd=ud.clonedir).strip() + self.assertEqual(clonedir_refs, self.updated_rev, + "clonedir must contain updated_rev after download()") + + # Step 4 - unpack_update fetches from dldir and rebases + git_fetcher = ud.method + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + # Step 5 - verify the commit graph and working tree content + # HEAD is the rebased local commit; its parent must be updated_rev + head_rev = self.git(['rev-parse', 'HEAD'], cwd=unpack_path).strip() + parent_rev = self.git(['rev-parse', 'HEAD^'], cwd=unpack_path).strip() + self.assertNotEqual(head_rev, local_commit, + "local commit should have a new SHA after rebase") + self.assertEqual(parent_rev, self.updated_rev, + "HEAD's parent must be updated_rev after fast-forward rebase") + + with open(os.path.join(unpack_path, 'testfile.txt'), 'r') as f: + self.assertEqual(f.read(), 'updated content\n') + with open(os.path.join(unpack_path, 'localfile.txt'), 'r') as f: + self.assertEqual(f.read(), 'local change\n') + + def test_unpack_update_already_at_target_revision(self): + """Test that unpack_update is a no-op when the checkout is already at SRCREV. + + Calling unpack_update() without advancing SRCREV must succeed and leave + the working tree unchanged. No rebase should be attempted because the + checkout already points at ud.revision. + """ + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + with open(os.path.join(unpack_path, 'testfile.txt')) as f: + self.assertEqual(f.read(), 'initial content\n') + + # Call unpack_update with SRCREV still at initial_rev - no upstream change + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + git_fetcher = ud.method + result = git_fetcher.unpack_update(ud, self.unpackdir, self.d) + self.assertTrue(result) + + # Content must be unchanged + with open(os.path.join(unpack_path, 'testfile.txt')) as f: + self.assertEqual(f.read(), 'initial content\n') + + def test_unpack_update_with_untracked_file(self): + """Test that unpack_update succeeds when the checkout has an untracked file. + + The status check uses '--untracked-files=no', so untracked files are not + detected and do not trigger the fallback path. git rebase also leaves + untracked files untouched, so both the upstream update and the untracked + file must be present after the call. + """ + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + + # Create an untracked file (not staged, not committed) + untracked = os.path.join(unpack_path, 'untracked.txt') + with open(untracked, 'w') as f: + f.write('untracked content\n') + + # Update to new upstream revision + self.d.setVar('SRCREV', self.updated_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + git_fetcher = ud.method + + # --untracked-files=no means the status check passes; rebase preserves the file + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + with open(os.path.join(unpack_path, 'testfile.txt'), 'r') as f: + self.assertEqual(f.read(), 'updated content\n') + + # Untracked file must survive the rebase + self.assertTrue(os.path.exists(untracked)) + with open(untracked, 'r') as f: + self.assertEqual(f.read(), 'untracked content\n') + + def test_unpack_update_with_staged_changes(self): + """Test that unpack_update fails when the checkout has staged (but not committed) changes. + + The rebase is run with --no-autostash so git refuses to rebase over a + dirty index. The caller (bitbake-setup) is expected to catch the + resulting LocalModificationsError and fall back to backup + re-fetch. + """ + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + + # Stage a new file without committing it + staged = os.path.join(unpack_path, 'staged.txt') + with open(staged, 'w') as f: + f.write('staged content\n') + self.git(['add', 'staged.txt'], cwd=unpack_path) + + # Update to new upstream revision + self.d.setVar('SRCREV', self.updated_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + git_fetcher = ud.method + + # Should fail - git rebase refuses to run with a dirty index + with self.assertRaises(bb.fetch2.LocalModificationsError): + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + def test_unpack_update_with_modified_tracked_file(self): + """Test that unpack_update fails when a tracked file has unstaged modifications. + + 'git status --untracked-files=no --porcelain' reports unstaged modifications + to tracked files (output line ' M filename'), which must block the update so + the caller can fall back to backup + re-fetch rather than silently discarding + work in progress. + """ + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + + # Modify a tracked file without staging or committing + with open(os.path.join(unpack_path, 'testfile.txt'), 'w') as f: + f.write('locally modified content\n') + + # Update to new upstream revision + self.d.setVar('SRCREV', self.updated_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + git_fetcher = ud.method + + # Should fail - unstaged modification to tracked file is detected by + # 'git status --untracked-files=no --porcelain' + with self.assertRaises(bb.fetch2.LocalModificationsError): + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + def test_unpack_update_conflict_raises_rebase_error(self): + """Test that unpack_update raises RebaseError on a rebase conflict. + + When a local commit modifies the same lines as an incoming upstream commit, + git rebase cannot resolve the conflict automatically. unpack_update must + abort the failed rebase and raise RebaseError so the caller can fall back + to a backup + re-fetch. + """ + # Fetch and unpack at the initial revision + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + + # Make a local commit that edits the same lines as the upcoming upstream commit + with open(os.path.join(unpack_path, 'testfile.txt'), 'w') as f: + f.write('conflicting local content\n') + self.git(['add', 'testfile.txt'], cwd=unpack_path) + self.git(['commit', '-m', 'Local conflicting commit'], cwd=unpack_path) + + # Add a third upstream commit that also edits testfile.txt differently + with open(os.path.join(self.srcdir, 'testfile.txt'), 'w') as f: + f.write('conflicting upstream content\n') + self.git(['add', 'testfile.txt'], cwd=self.srcdir) + self.git(['commit', '-m', 'Upstream conflicting commit'], cwd=self.srcdir) + conflict_rev = self.git(['rev-parse', 'HEAD'], cwd=self.srcdir).strip() + + # Update SRCREV to the new upstream commit + self.d.setVar('SRCREV', conflict_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + git_fetcher = ud.method + + # unpack_update must fail and clean up (rebase --abort) rather than + # leaving the repo in a mid-rebase state + with self.assertRaises(bb.fetch2.RebaseError): + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + # Verify the repo is not left in a conflicted / mid-rebase state + rebase_merge = os.path.join(unpack_path, '.git', 'rebase-merge') + rebase_apply = os.path.join(unpack_path, '.git', 'rebase-apply') + self.assertFalse(os.path.exists(rebase_merge), + "rebase-merge dir should not exist after failed unpack_update") + self.assertFalse(os.path.exists(rebase_apply), + "rebase-apply dir should not exist after failed unpack_update") + + def test_unpack_update_untracked_file_overwritten_by_upstream(self): + """Test that unpack_update raises RebaseError when an untracked file would be + overwritten by an incoming upstream commit. + + We skip untracked files in the pre-check (git rebase doesn't touch harmless + untracked files), but git itself refuses to rebase when an untracked file would + be overwritten by the incoming changes. The resulting FetchError must be caught + and re-raised as RebaseError without leaving the repo in a mid-rebase state. + + Two sub-cases are covered: + - top-level untracked file clashing with an incoming upstream file + - untracked file inside a subdirectory (xxx/somefile) clashing with an + upstream commit that adds the same path + """ + def _run_case(upstream_path, local_rel_path, commit_msg): + """ + Add upstream_path to self.srcdir, create local_rel_path as an + untracked file in the checkout, then assert that unpack_update + raises RebaseError and leaves no mid-rebase state, and that the + local file is untouched. + """ + # Fresh fetch + unpack at the current SRCREV + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + + # Upstream adds the file (potentially inside a subdirectory) + full_upstream = os.path.join(self.srcdir, upstream_path) + os.makedirs(os.path.dirname(full_upstream), exist_ok=True) + with open(full_upstream, 'w') as f: + f.write('upstream content\n') + self.git(['add', upstream_path], cwd=self.srcdir) + self.git(['commit', '-m', commit_msg], cwd=self.srcdir) + new_rev = self.git(['rev-parse', 'HEAD'], cwd=self.srcdir).strip() + + # Create the clashing untracked file in the checkout + full_local = os.path.join(unpack_path, local_rel_path) + os.makedirs(os.path.dirname(full_local), exist_ok=True) + with open(full_local, 'w') as f: + f.write('local untracked content\n') + + self.d.setVar('SRCREV', new_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + git_fetcher = ud.method + + # git rebase refuses because the untracked file would be overwritten + with self.assertRaises(bb.fetch2.RebaseError): + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + # Repo must not be left in a mid-rebase state + self.assertFalse(os.path.exists(os.path.join(unpack_path, '.git', 'rebase-merge'))) + self.assertFalse(os.path.exists(os.path.join(unpack_path, '.git', 'rebase-apply'))) + + # The local untracked file must be untouched + self.assertTrue(os.path.exists(full_local)) + with open(full_local) as f: + self.assertEqual(f.read(), 'local untracked content\n') + + # Reset unpackdir for the next sub-case + import shutil as _shutil + _shutil.rmtree(self.unpackdir) + os.makedirs(self.unpackdir) + + # Sub-case 1: top-level file clash + _run_case('newfile.txt', 'newfile.txt', + 'Upstream adds newfile.txt') + + # Sub-case 2: file inside a subdirectory (xxx/somefile) + _run_case('xxx/somefile.txt', 'xxx/somefile.txt', + 'Upstream adds xxx/somefile.txt') + + def test_unpack_update_shallow_clone_fails(self): + """Test that unpack_update raises UnpackError for shallow-tarball checkouts. + + Shallow clones lack full history, which makes an in-place rebase impossible + without network access. After fetching with BB_GIT_SHALLOW=1 the clonedir + is deleted so that unpack() is forced to use the shallow tarball. + A subsequent call to unpack_update() must raise UnpackError and the message + must mention 'shallow clone' so callers can distinguish this case. + """ + self.d.setVar('BB_GIT_SHALLOW', '1') + self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1') + + # First fetch at initial revision + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + # Remove clonedir to force use of shallow tarball + clonedir = os.path.join(self.dldir, 'git2') + if os.path.exists(clonedir): + shutil.rmtree(clonedir) + + fetcher.unpack(self.unpackdir) + + # Update to new revision + self.d.setVar('SRCREV', self.updated_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + # unpack_update should fail for shallow clones + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + git_fetcher = ud.method + + with self.assertRaises(bb.fetch2.UnpackError) as context: + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + self.assertIn("shallow clone", str(context.exception).lower()) + + def test_unpack_update_stale_dldir_remote(self): + """Test that unpack_update raises UnpackError when the dldir remote URL is stale. + + If the clonedir has been removed after the initial unpack (e.g. DL_DIR was + cleaned) the 'dldir' remote URL no longer resolves. The fetch inside + update_mode will fail with a FetchError which must be re-raised as + UnpackError so the caller can fall back to a full re-fetch. + """ + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + + # Advance SRCREV to trigger update_mode + self.d.setVar('SRCREV', self.updated_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + + # Delete the clonedir and corrupt the dldir remote URL so that + # 'git fetch dldir' fails, simulating a missing or relocated DL_DIR. + shutil.rmtree(ud.clonedir) + self.git(['remote', 'set-url', 'dldir', 'file://' + ud.clonedir], + cwd=unpack_path) + + git_fetcher = ud.method + with self.assertRaises(bb.fetch2.UnpackError): + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + def test_fetch_unpack_update_toplevel_api(self): + """Test that the top-level Fetch.unpack_update() dispatches to Git.unpack_update(). + + Callers such as bitbake-setup use fetcher.unpack_update(root) rather than + calling the method on the Git fetcher directly. Verify that the public API + works end-to-end: fetch at initial_rev, unpack, advance to updated_rev, + fetch again, then call fetcher.unpack_update(root) and confirm the content + is updated. + """ + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + with open(os.path.join(unpack_path, 'testfile.txt')) as f: + self.assertEqual(f.read(), 'initial content\n') + + self.d.setVar('SRCREV', self.updated_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + # Use the public Fetch.unpack_update() rather than the method directly + fetcher.unpack_update(self.unpackdir) + + with open(os.path.join(unpack_path, 'testfile.txt')) as f: + self.assertEqual(f.read(), 'updated content\n') diff --git a/bitbake/lib/bb/tests/filter.py b/bitbake/lib/bb/tests/filter.py new file mode 100644 index 00000000..245df7b2 --- /dev/null +++ b/bitbake/lib/bb/tests/filter.py @@ -0,0 +1,88 @@ +# +# Copyright (C) 2025 Garmin Ltd. or its subsidiaries +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import unittest +import bb.filter + + +class BuiltinFilterTest(unittest.TestCase): + def test_disallowed_builtins(self): + with self.assertRaises(NameError): + val = bb.filter.apply_filters("1", ["open('foo.txt', 'rb')"]) + + def test_prefix(self): + val = bb.filter.apply_filters("1 2 3", ["prefix(val, 'a')"]) + self.assertEqual(val, "a1 a2 a3") + + val = bb.filter.apply_filters("", ["prefix(val, 'a')"]) + self.assertEqual(val, "") + + def test_suffix(self): + val = bb.filter.apply_filters("1 2 3", ["suffix(val, 'b')"]) + self.assertEqual(val, "1b 2b 3b") + + val = bb.filter.apply_filters("", ["suffix(val, 'b')"]) + self.assertEqual(val, "") + + def test_sort(self): + val = bb.filter.apply_filters("z y x", ["sort(val)"]) + self.assertEqual(val, "x y z") + + val = bb.filter.apply_filters("", ["sort(val)"]) + self.assertEqual(val, "") + + def test_identity(self): + val = bb.filter.apply_filters("1 2 3", ["val"]) + self.assertEqual(val, "1 2 3") + + val = bb.filter.apply_filters("123", ["val"]) + self.assertEqual(val, "123") + + def test_empty(self): + val = bb.filter.apply_filters("1 2 3", ["", "prefix(val, 'a')", ""]) + self.assertEqual(val, "a1 a2 a3") + + def test_nested(self): + val = bb.filter.apply_filters("1 2 3", ["prefix(prefix(val, 'a'), 'b')"]) + self.assertEqual(val, "ba1 ba2 ba3") + + val = bb.filter.apply_filters("1 2 3", ["prefix(prefix(val, 'b'), 'a')"]) + self.assertEqual(val, "ab1 ab2 ab3") + + def test_filter_order(self): + val = bb.filter.apply_filters("1 2 3", ["prefix(val, 'a')", "prefix(val, 'b')"]) + self.assertEqual(val, "ba1 ba2 ba3") + + val = bb.filter.apply_filters("1 2 3", ["prefix(val, 'b')", "prefix(val, 'a')"]) + self.assertEqual(val, "ab1 ab2 ab3") + + val = bb.filter.apply_filters("1 2 3", ["prefix(val, 'a')", "suffix(val, 'b')"]) + self.assertEqual(val, "a1b a2b a3b") + + val = bb.filter.apply_filters("1 2 3", ["suffix(val, 'b')", "prefix(val, 'a')"]) + self.assertEqual(val, "a1b a2b a3b") + + def test_remove(self): + val = bb.filter.apply_filters("1 2 3", ["remove(val, ['2'])"]) + self.assertEqual(val, "1 3") + + val = bb.filter.apply_filters("1,2,3", ["remove(val, ['2'], ',')"]) + self.assertEqual(val, "1,3") + + val = bb.filter.apply_filters("1 2 3", ["remove(val, ['4'])"]) + self.assertEqual(val, "1 2 3") + + val = bb.filter.apply_filters("1 2 3", ["remove(val, ['1', '2'])"]) + self.assertEqual(val, "3") + + val = bb.filter.apply_filters("1 2 3", ["remove(val, '2')"]) + self.assertEqual(val, "1 3") + + val = bb.filter.apply_filters("1 2 3", ["remove(val, '4')"]) + self.assertEqual(val, "1 2 3") + + val = bb.filter.apply_filters("1 2 3", ["remove(val, '1 2')"]) + self.assertEqual(val, "3") diff --git a/bitbake/lib/bb/tests/parse-tests/classes/base.bbclass b/bitbake/lib/bb/tests/parse-tests/classes/base.bbclass new file mode 100644 index 00000000..db8898e1 --- /dev/null +++ b/bitbake/lib/bb/tests/parse-tests/classes/base.bbclass @@ -0,0 +1,5 @@ +# At least one task is required for bitbake to parse +do_fetch() { + : +} +addtask do_fetch diff --git a/bitbake/lib/bb/tests/parse-tests/classes/recipe-file-class.bbclass b/bitbake/lib/bb/tests/parse-tests/classes/recipe-file-class.bbclass new file mode 100644 index 00000000..4682dc6c --- /dev/null +++ b/bitbake/lib/bb/tests/parse-tests/classes/recipe-file-class.bbclass @@ -0,0 +1,2 @@ +BBCLASS_RECIPE_FILE := "${@os.path.basename(d.getVar('__BB_RECIPE_FILE'))}" +BBCLASS_FILE := "${@os.path.basename(d.getVar('FILE'))}" diff --git a/bitbake/lib/bb/tests/parse-tests/conf/bitbake.conf b/bitbake/lib/bb/tests/parse-tests/conf/bitbake.conf new file mode 100644 index 00000000..e0362506 --- /dev/null +++ b/bitbake/lib/bb/tests/parse-tests/conf/bitbake.conf @@ -0,0 +1,15 @@ +CACHE = "${TOPDIR}/cache" +THISDIR = "${@os.path.dirname(d.getVar('FILE'))}" +COREBASE := "${@os.path.normpath(os.path.dirname(d.getVar('FILE')+'/../../'))}" +EXTRA_BBFILES ?= "" +BBFILES = "${COREBASE}/recipes/*.bb ${COREBASE}/recipes/*.bbappend ${EXTRA_BBFILES}" +PROVIDES = "${PN}" +PN = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0]}" +PF = "${BB_CURRENT_MC}:${PN}" +export PATH +TMPDIR ??= "${TOPDIR}" +STAMP = "${TMPDIR}/stamps/${PN}" +T = "${TMPDIR}/workdir/${PN}/temp" +BB_NUMBER_THREADS = "4" + +BB_BASEHASH_IGNORE_VARS = "BB_CURRENT_MC BB_HASHSERVE TMPDIR TOPDIR SLOWTASKS SSTATEVALID FILE BB_CURRENTTASK" diff --git a/bitbake/lib/bb/tests/parse.py b/bitbake/lib/bb/tests/parse.py index 72d1962e..6ac2137e 100644 --- a/bitbake/lib/bb/tests/parse.py +++ b/bitbake/lib/bb/tests/parse.py @@ -11,6 +11,8 @@ import tempfile import logging import bb import os +import subprocess +import textwrap logger = logging.getLogger('BitBake.TestParse') @@ -39,7 +41,6 @@ C = "3" os.chdir(self.origdir) def parsehelper(self, content, suffix = ".bb"): - f = tempfile.NamedTemporaryFile(suffix = suffix) f.write(bytes(content, "utf-8")) f.flush() @@ -47,17 +48,17 @@ C = "3" return f def test_parse_simple(self): - f = self.parsehelper(self.testfile) - d = bb.parse.handle(f.name, self.d)[''] + with self.parsehelper(self.testfile) as f: + d = bb.parse.handle(f.name, self.d)[''] self.assertEqual(d.getVar("A"), "1") self.assertEqual(d.getVar("B"), "2") self.assertEqual(d.getVar("C"), "3") def test_parse_incomplete_function(self): testfileB = self.testfile.replace("}", "") - f = self.parsehelper(testfileB) - with self.assertRaises(bb.parse.ParseError): - d = bb.parse.handle(f.name, self.d)[''] + with self.parsehelper(testfileB) as f: + with self.assertRaises(bb.parse.ParseError): + d = bb.parse.handle(f.name, self.d)[''] unsettest = """ A = "1" @@ -69,12 +70,65 @@ unset B[flag] """ def test_parse_unset(self): - f = self.parsehelper(self.unsettest) - d = bb.parse.handle(f.name, self.d)[''] + with self.parsehelper(self.unsettest) as f: + d = bb.parse.handle(f.name, self.d)[''] self.assertEqual(d.getVar("A"), None) self.assertEqual(d.getVarFlag("A","flag"), None) self.assertEqual(d.getVar("B"), "2") + defaulttest = """ +A = "set value" +A ??= "default value" + +A[flag_set_vs_question] = "set flag" +A[flag_set_vs_question] ?= "question flag" + +A[flag_set_vs_default] = "set flag" +A[flag_set_vs_default] ??= "default flag" + +A[flag_question] ?= "question flag" + +A[flag_default] ??= "default flag" + +A[flag_question_vs_default] ?= "question flag" +A[flag_question_vs_default] ??= "default flag" + +A[flag_default_vs_question] ??= "default flag" +A[flag_default_vs_question] ?= "question flag" + +A[flag_set_question_default] = "set flag" +A[flag_set_question_default] ?= "question flag" +A[flag_set_question_default] ??= "default flag" + +A[flag_set_default_question] = "set flag" +A[flag_set_default_question] ??= "default flag" +A[flag_set_default_question] ?= "question flag" + +A[flag_set_twice] = "set flag first" +A[flag_set_twice] = "set flag second" + +A[flag_question_twice] ?= "question flag first" +A[flag_question_twice] ?= "question flag second" + +A[flag_default_twice] ??= "default flag first" +A[flag_default_twice] ??= "default flag second" +""" + def test_parse_defaulttest(self): + with self.parsehelper(self.defaulttest) as f: + d = bb.parse.handle(f.name, self.d)[''] + self.assertEqual(d.getVar("A"), "set value") + self.assertEqual(d.getVarFlag("A","flag_set_vs_question"), "set flag") + self.assertEqual(d.getVarFlag("A","flag_set_vs_default"), "set flag") + self.assertEqual(d.getVarFlag("A","flag_question"), "question flag") + self.assertEqual(d.getVarFlag("A","flag_default"), "default flag") + self.assertEqual(d.getVarFlag("A","flag_question_vs_default"), "question flag") + self.assertEqual(d.getVarFlag("A","flag_default_vs_question"), "question flag") + self.assertEqual(d.getVarFlag("A","flag_set_question_default"), "set flag") + self.assertEqual(d.getVarFlag("A","flag_set_default_question"), "set flag") + self.assertEqual(d.getVarFlag("A","flag_set_twice"), "set flag second") + self.assertEqual(d.getVarFlag("A","flag_question_twice"), "question flag first") + self.assertEqual(d.getVarFlag("A","flag_default_twice"), "default flag second") + exporttest = """ A = "a" export B = "b" @@ -83,8 +137,8 @@ exportD = "d" """ def test_parse_exports(self): - f = self.parsehelper(self.exporttest) - d = bb.parse.handle(f.name, self.d)[''] + with self.parsehelper(self.exporttest) as f: + d = bb.parse.handle(f.name, self.d)[''] self.assertEqual(d.getVar("A"), "a") self.assertIsNone(d.getVarFlag("A", "export")) self.assertEqual(d.getVar("B"), "b") @@ -96,7 +150,6 @@ exportD = "d" self.assertEqual(d.getVar("exportD"), "d") self.assertIsNone(d.getVarFlag("exportD", "export")) - overridetest = """ RRECOMMENDS:${PN} = "a" RRECOMMENDS:${PN}:libc = "b" @@ -105,8 +158,8 @@ PN = "gtk+" """ def test_parse_overrides(self): - f = self.parsehelper(self.overridetest) - d = bb.parse.handle(f.name, self.d)[''] + with self.parsehelper(self.overridetest) as f: + d = bb.parse.handle(f.name, self.d)[''] self.assertEqual(d.getVar("RRECOMMENDS"), "b") bb.data.expandKeys(d) self.assertEqual(d.getVar("RRECOMMENDS"), "b") @@ -120,8 +173,8 @@ EXTRA_OECONF:append = " c" """ def test_parse_overrides2(self): - f = self.parsehelper(self.overridetest2) - d = bb.parse.handle(f.name, self.d)[''] + with self.parsehelper(self.overridetest2) as f: + d = bb.parse.handle(f.name, self.d)[''] d.appendVar("EXTRA_OECONF", " d") d.setVar("OVERRIDES", "class-target") self.assertEqual(d.getVar("EXTRA_OECONF"), "b c d") @@ -133,8 +186,8 @@ PN = "bc" """ def test_parse_combinations(self): - f = self.parsehelper(self.overridetest3) - d = bb.parse.handle(f.name, self.d)[''] + with self.parsehelper(self.overridetest3) as f: + d = bb.parse.handle(f.name, self.d)[''] bb.data.expandKeys(d) self.assertEqual(d.getVar("DESCRIPTION:bc-dev"), "A B") d.setVar("DESCRIPTION", "E") @@ -142,7 +195,6 @@ PN = "bc" d.setVar("OVERRIDES", "bc-dev") self.assertEqual(d.getVar("DESCRIPTION"), "C D") - classextend = """ VAR_var:override1 = "B" EXTRA = ":override1" @@ -160,24 +212,36 @@ python () { # # Test based upon a real world data corruption issue. One # data store changing a variable poked through into a different data - # store. This test case replicates that issue where the value 'B' would + # store. This test case replicates that issue where the value 'B' would # become unset/disappear. # def test_parse_classextend_contamination(self): self.d.setVar("__bbclasstype", "recipe") - cls = self.parsehelper(self.classextend_bbclass, suffix=".bbclass") - #clsname = os.path.basename(cls.name).replace(".bbclass", "") - self.classextend = self.classextend.replace("###CLASS###", cls.name) - f = self.parsehelper(self.classextend) - alldata = bb.parse.handle(f.name, self.d) - d1 = alldata[''] - d2 = alldata[cls.name] - self.assertEqual(d1.getVar("VAR_var"), "B") - self.assertEqual(d2.getVar("VAR_var"), None) + with self.parsehelper(self.classextend_bbclass, suffix=".bbclass") as cls: + #clsname = os.path.basename(cls.name).replace(".bbclass", "") + self.classextend = self.classextend.replace("###CLASS###", cls.name) + with self.parsehelper(self.classextend) as f: + alldata = bb.parse.handle(f.name, self.d) + d1 = alldata[''] + d2 = alldata[cls.name] + self.assertEqual(d1.getVar("VAR_var"), "B") + self.assertEqual(d2.getVar("VAR_var"), None) addtask_deltask = """ addtask do_patch after do_foo after do_unpack before do_configure before do_compile -addtask do_fetch do_patch +addtask do_fetch2 do_patch2 + +addtask do_myplaintask +addtask do_myplaintask2 +deltask do_myplaintask2 +addtask do_mytask# comment +addtask do_mytask2 # comment2 +addtask do_mytask3 +deltask do_mytask3# comment +deltask do_mytask4 # comment2 + +# Ensure a missing task prefix on after works +addtask do_mytask5 after mytask MYVAR = "do_patch" EMPTYVAR = "" @@ -185,17 +249,10 @@ deltask do_fetch ${MYVAR} ${EMPTYVAR} deltask ${EMPTYVAR} """ def test_parse_addtask_deltask(self): - import sys - - with self.assertLogs() as logs: - f = self.parsehelper(self.addtask_deltask) + with self.parsehelper(self.addtask_deltask) as f: d = bb.parse.handle(f.name, self.d)[''] - - output = "".join(logs.output) - self.assertTrue("addtask contained multiple 'before' keywords" in output) - self.assertTrue("addtask contained multiple 'after' keywords" in output) - self.assertTrue('addtask ignored: " do_patch"' in output) - #self.assertTrue('dependent task do_foo for do_patch does not exist' in output) + self.assertSequenceEqual(['do_fetch2', 'do_patch2', 'do_myplaintask', 'do_mytask', 'do_mytask2', 'do_mytask5'], bb.build.listtasks(d)) + self.assertEqual(['do_mytask'], d.getVarFlag("do_mytask5", "deps")) broken_multiline_comment = """ # First line of comment \\ @@ -203,10 +260,9 @@ deltask ${EMPTYVAR} """ def test_parse_broken_multiline_comment(self): - f = self.parsehelper(self.broken_multiline_comment) - with self.assertRaises(bb.BBHandledException): - d = bb.parse.handle(f.name, self.d)[''] - + with self.parsehelper(self.broken_multiline_comment) as f: + with self.assertRaises(bb.BBHandledException): + d = bb.parse.handle(f.name, self.d)[''] comment_in_var = """ VAR = " \\ @@ -216,10 +272,9 @@ VAR = " \\ " """ def test_parse_comment_in_var(self): - f = self.parsehelper(self.comment_in_var) - with self.assertRaises(bb.BBHandledException): - d = bb.parse.handle(f.name, self.d)[''] - + with self.parsehelper(self.comment_in_var) as f: + with self.assertRaises(bb.BBHandledException): + d = bb.parse.handle(f.name, self.d)[''] at_sign_in_var_flag = """ A[flag@.service] = "nonet" @@ -229,8 +284,8 @@ C[f] = "flag" unset A[flag@.service] """ def test_parse_at_sign_in_var_flag(self): - f = self.parsehelper(self.at_sign_in_var_flag) - d = bb.parse.handle(f.name, self.d)[''] + with self.parsehelper(self.at_sign_in_var_flag) as f: + d = bb.parse.handle(f.name, self.d)[''] self.assertEqual(d.getVar("A"), None) self.assertEqual(d.getVar("B"), None) self.assertEqual(d.getVarFlag("A","flag@.service"), None) @@ -239,9 +294,9 @@ unset A[flag@.service] def test_parse_invalid_at_sign_in_var_flag(self): invalid_at_sign = self.at_sign_in_var_flag.replace("B[f", "B[@f") - f = self.parsehelper(invalid_at_sign) - with self.assertRaises(bb.parse.ParseError): - d = bb.parse.handle(f.name, self.d)[''] + with self.parsehelper(invalid_at_sign) as f: + with self.assertRaises(bb.parse.ParseError): + d = bb.parse.handle(f.name, self.d)[''] export_function_recipe = """ inherit someclass @@ -296,14 +351,11 @@ EXPORT_FUNCTIONS do_compile do_compilepython os.makedirs(tempdir + "/classes") with open(tempdir + "/classes/someclass.bbclass", "w") as f: f.write(self.export_function_class) - f.flush() with open(tempdir + "/classes/secondclass.bbclass", "w") as f: f.write(self.export_function_class2) - f.flush() with open(recipename, "w") as f: f.write(self.export_function_recipe) - f.flush() os.chdir(tempdir) d = bb.parse.handle(recipename, bb.data.createCopy(self.d))[''] self.assertIn("someclass_do_compile", d.getVar("do_compile")) @@ -313,7 +365,6 @@ EXPORT_FUNCTIONS do_compile do_compilepython recipename2 = tempdir + "/recipe2.bb" with open(recipename2, "w") as f: f.write(self.export_function_recipe2) - f.flush() d = bb.parse.handle(recipename2, bb.data.createCopy(self.d))[''] self.assertNotIn("someclass_do_compile", d.getVar("do_compile")) @@ -324,10 +375,8 @@ EXPORT_FUNCTIONS do_compile do_compilepython with open(recipename, "a+") as f: f.write("\ninherit secondclass\n") - f.flush() with open(recipename2, "a+") as f: f.write("\ninherit secondclass\n") - f.flush() d = bb.parse.handle(recipename, bb.data.createCopy(self.d))[''] self.assertIn("secondclass_do_compile", d.getVar("do_compile")) @@ -341,3 +390,251 @@ EXPORT_FUNCTIONS do_compile do_compilepython self.assertIn("else", d.getVar("do_compilepython")) check_function_flags(d) + export_function_unclosed_tab = """ +do_compile () { + bb.note("Something") +\t} +""" + export_function_unclosed_space = """ +do_compile () { + bb.note("Something") + } +""" + export_function_residue = """ +do_compile () { + bb.note("Something") +} + +include \\ +""" + + def test_unclosed_functions(self): + def test_helper(content, expected_error): + with tempfile.TemporaryDirectory() as tempdir: + recipename = tempdir + "/recipe_unclosed.bb" + with open(recipename, "w") as f: + f.write(content) + os.chdir(tempdir) + with self.assertRaises(bb.parse.ParseError) as error: + bb.parse.handle(recipename, bb.data.createCopy(self.d)) + self.assertIn(expected_error, str(error.exception)) + + with tempfile.TemporaryDirectory() as tempdir: + test_helper(self.export_function_unclosed_tab, "Unparsed lines from unclosed function") + test_helper(self.export_function_unclosed_space, "Unparsed lines from unclosed function") + test_helper(self.export_function_residue, "Unparsed lines") + + recipename_closed = tempdir + "/recipe_closed.bb" + with open(recipename_closed, "w") as in_file: + lines = self.export_function_unclosed_tab.split("\n") + lines[3] = "}" + in_file.write("\n".join(lines)) + bb.parse.handle(recipename_closed, bb.data.createCopy(self.d)) + + special_character_assignment = """ +A+="a" +A+ = "b" ++ = "c" +""" + ambigous_assignment = """ ++= "d" +""" + def test_parse_special_character_assignment(self): + with self.parsehelper(self.special_character_assignment) as f: + d = bb.parse.handle(f.name, self.d)[''] + self.assertEqual(d.getVar("A"), " a") + self.assertEqual(d.getVar("A+"), "b") + self.assertEqual(d.getVar("+"), "c") + + with self.parsehelper(self.ambigous_assignment) as f: + with self.assertRaises(bb.parse.ParseError) as error: + bb.parse.handle(f.name, self.d) + self.assertIn("Empty variable name in assignment", str(error.exception)) + + someconf1 = """ +EXTRA_OECONF:append = " foo" +""" + + someconf2 = """ +EXTRA_OECONF:append = " bar" +""" + + someconf3 = """ +EXTRA_OECONF:append = " foobar" +""" + + def test_include_and_require(self): + def test_helper(content, result): + with self.parsehelper(content) as f: + if isinstance(result, type) and issubclass(result, Exception): + with self.assertRaises(result): + d = bb.parse.handle(f.name, bb.data.createCopy(self.d))[''] + else: + d = bb.parse.handle(f.name, bb.data.createCopy(self.d))[''] + self.assertEqual(d.getVar("EXTRA_OECONF"), result) + + with tempfile.TemporaryDirectory() as tempdir: + os.makedirs(tempdir + "/conf1") + os.makedirs(tempdir + "/conf2") + + with open(tempdir + "/conf1/some.conf", "w") as f: + f.write(self.someconf1) + with open(tempdir + "/conf2/some.conf", "w") as f: + f.write(self.someconf2) + with open(tempdir + "/conf2/some3.conf", "w") as f: + f.write(self.someconf3) + + self.d.setVar("BBPATH", tempdir + "/conf1" + ":" + tempdir + "/conf2") + + test_helper("include some.conf", " foo") + test_helper("include someother.conf", None) + test_helper("include some3.conf", " foobar") + test_helper("include ${@''}", None) + test_helper("include " + tempdir + "/conf2/some.conf", " bar") + + test_helper("require some.conf", " foo") + test_helper("require someother.conf", bb.parse.ParseError) + test_helper("require some3.conf", " foobar") + test_helper("require ${@''}", None) + test_helper("require " + tempdir + "/conf2/some.conf", " bar") + + test_helper("include_all some.conf", " foo bar") + test_helper("include_all someother.conf", None) + test_helper("include_all some3.conf", " foobar") + + self.d.setVar("BBPATH", tempdir + "/conf2" + ":" + tempdir + "/conf1") + + test_helper("include some.conf", " bar") + test_helper("include some3.conf", " foobar") + test_helper("require some.conf", " bar") + test_helper("require some3.conf", " foobar") + test_helper("include_all some.conf", " bar foo") + test_helper("include_all some3.conf", " foobar") + + def test_file_variables(self): + # Tests the values of FILE and __BB_RECIPE_FILE in different + # combinations of bbappends, includes, and inherits + + def write_file(path, data): + with open(path, "w") as f: + f.write(textwrap.dedent(data)) + + def run_bitbake(cmd, builddir, extraenv={}): + env = os.environ.copy() + env["BBPATH"] = os.path.realpath(os.path.join(os.path.dirname(__file__), "parse-tests")) + env["BB_ENV_PASSTHROUGH_ADDITIONS"] = "TOPDIR" + env["TOPDIR"] = builddir + for k, v in extraenv.items(): + env[k] = v + env["BB_ENV_PASSTHROUGH_ADDITIONS"] = env["BB_ENV_PASSTHROUGH_ADDITIONS"] + " " + k + try: + return subprocess.check_output(cmd, env=env, stderr=subprocess.STDOUT, universal_newlines=True, cwd=builddir) + except subprocess.CalledProcessError as e: + self.fail("Command %s failed with %s" % (cmd, e.output)) + + with tempfile.TemporaryDirectory(prefix="parserecipes") as recipes, tempfile.TemporaryDirectory(prefix="parsetest") as builddir: + extraenv = { + "EXTRA_BBFILES": f"{recipes}/*.bb {recipes}/*.bbappend", + } + + inc_path = f"{recipes}/recipe-file.inc" + bbappend_path = f"{recipes}/recipe-%.bbappend" + recipe_path = f"{recipes}/recipe-file1.bb" + + # __BB_RECIPE_FILE should always be the name of .bb file, even + # when set in a bbappend. FILE is the name of the bbappend + write_file(recipe_path, "") + write_file(bbappend_path, + """\ + BBAPPEND_RECIPE_FILE := "${@os.path.basename(d.getVar('__BB_RECIPE_FILE'))}" + BBAPPEND_FILE := "${@os.path.basename(d.getVar('FILE'))}" + """ + ) + output = run_bitbake(["bitbake", "-e", "recipe-file1"], builddir, extraenv).splitlines() + self.assertIn('BBAPPEND_FILE="recipe-%.bbappend"', output) + self.assertIn(f'BBAPPEND_RECIPE_FILE="recipe-file1.bb"', output) + + # __BB_RECIPE_FILE should always be the name of .bb file, even when + # set in an include file. FILE is the name of the include + write_file(recipe_path, + """\ + require recipe-file.inc + """ + ) + write_file(inc_path, + """\ + INC_RECIPE_FILE := "${@os.path.basename(d.getVar('__BB_RECIPE_FILE'))}" + INC_FILE := "${@os.path.basename(d.getVar('FILE'))}" + """ + ) + output = run_bitbake(["bitbake", "-e", "recipe-file1"], builddir, extraenv).splitlines() + self.assertIn('INC_FILE="recipe-file.inc"', output) + self.assertIn(f'INC_RECIPE_FILE="recipe-file1.bb"', output) + + # Test when the include file is included from a bbappend + write_file(recipe_path, "") + write_file(bbappend_path, + """\ + require recipe-file.inc + """ + ) + output = run_bitbake(["bitbake", "-e", "recipe-file1"], builddir, extraenv).splitlines() + self.assertIn('INC_FILE="recipe-file.inc"', output) + self.assertIn(f'INC_RECIPE_FILE="recipe-file1.bb"', output) + + # Test the variables in a bbclass when inherited directly in the + # recipe. Note that FILE still refers to the recipe in a bbclass + write_file(recipe_path, + """\ + inherit recipe-file-class + """ + ) + output = run_bitbake(["bitbake", "-e", "recipe-file1"], builddir, extraenv).splitlines() + self.assertIn('BBCLASS_FILE="recipe-file1.bb"', output) + self.assertIn(f'BBCLASS_RECIPE_FILE="recipe-file1.bb"', output) + + # Test the variables when the inherit is in a bbappend. In this + # case, FILE is the bbappend + write_file(recipe_path, "") + write_file(bbappend_path, + """\ + inherit recipe-file-class + """ + ) + output = run_bitbake(["bitbake", "-e", "recipe-file1"], builddir, extraenv).splitlines() + self.assertIn('BBCLASS_FILE="recipe-%.bbappend"', output) + self.assertIn(f'BBCLASS_RECIPE_FILE="recipe-file1.bb"', output) + + # Test the variables when the inherit is in a include. In this + # case, FILE is the include file + write_file(recipe_path, + """\ + require recipe-file.inc + """ + ) + write_file(bbappend_path, "") + write_file(inc_path, + """\ + inherit recipe-file-class + """ + ) + output = run_bitbake(["bitbake", "-e", "recipe-file1"], builddir, extraenv).splitlines() + self.assertIn('BBCLASS_FILE="recipe-file.inc"', output) + self.assertIn(f'BBCLASS_RECIPE_FILE="recipe-file1.bb"', output) + + # Test the variables when the inherit is in a include included from + # a bbappend. In this case, FILE is the include file + write_file(recipe_path, "") + write_file(bbappend_path, + """\ + require recipe-file.inc + """ + ) + write_file(inc_path, + """\ + inherit recipe-file-class + """ + ) + output = run_bitbake(["bitbake", "-e", "recipe-file1"], builddir, extraenv).splitlines() + self.assertIn('BBCLASS_FILE="recipe-file.inc"', output) + self.assertIn(f'BBCLASS_RECIPE_FILE="recipe-file1.bb"', output) diff --git a/bitbake/lib/bb/tests/persist_data.py b/bitbake/lib/bb/tests/persist_data.py deleted file mode 100644 index f641b5ac..00000000 --- a/bitbake/lib/bb/tests/persist_data.py +++ /dev/null @@ -1,129 +0,0 @@ -# -# BitBake Test for lib/bb/persist_data/ -# -# Copyright (C) 2018 Garmin Ltd. -# -# SPDX-License-Identifier: GPL-2.0-only -# - -import unittest -import bb.data -import bb.persist_data -import tempfile -import threading - -class PersistDataTest(unittest.TestCase): - def _create_data(self): - return bb.persist_data.persist('TEST_PERSIST_DATA', self.d) - - def setUp(self): - self.d = bb.data.init() - self.tempdir = tempfile.TemporaryDirectory() - self.d['PERSISTENT_DIR'] = self.tempdir.name - self.data = self._create_data() - self.items = { - 'A1': '1', - 'B1': '2', - 'C2': '3' - } - self.stress_count = 10000 - self.thread_count = 5 - - for k,v in self.items.items(): - self.data[k] = v - - def tearDown(self): - self.tempdir.cleanup() - - def _iter_helper(self, seen, iterator): - with iter(iterator): - for v in iterator: - self.assertTrue(v in seen) - seen.remove(v) - self.assertEqual(len(seen), 0, '%s not seen' % seen) - - def test_get(self): - for k, v in self.items.items(): - self.assertEqual(self.data[k], v) - - self.assertIsNone(self.data.get('D')) - with self.assertRaises(KeyError): - self.data['D'] - - def test_set(self): - for k, v in self.items.items(): - self.data[k] += '-foo' - - for k, v in self.items.items(): - self.assertEqual(self.data[k], v + '-foo') - - def test_delete(self): - self.data['D'] = '4' - self.assertEqual(self.data['D'], '4') - del self.data['D'] - self.assertIsNone(self.data.get('D')) - with self.assertRaises(KeyError): - self.data['D'] - - def test_contains(self): - for k in self.items: - self.assertTrue(k in self.data) - self.assertTrue(self.data.has_key(k)) - self.assertFalse('NotFound' in self.data) - self.assertFalse(self.data.has_key('NotFound')) - - def test_len(self): - self.assertEqual(len(self.data), len(self.items)) - - def test_iter(self): - self._iter_helper(set(self.items.keys()), self.data) - - def test_itervalues(self): - self._iter_helper(set(self.items.values()), self.data.itervalues()) - - def test_iteritems(self): - self._iter_helper(set(self.items.items()), self.data.iteritems()) - - def test_get_by_pattern(self): - self._iter_helper({'1', '2'}, self.data.get_by_pattern('_1')) - - def _stress_read(self, data): - for i in range(self.stress_count): - for k in self.items: - data[k] - - def _stress_write(self, data): - for i in range(self.stress_count): - for k, v in self.items.items(): - data[k] = v + str(i) - - def _validate_stress(self): - for k, v in self.items.items(): - self.assertEqual(self.data[k], v + str(self.stress_count - 1)) - - def test_stress(self): - self._stress_read(self.data) - self._stress_write(self.data) - self._validate_stress() - - def test_stress_threads(self): - def read_thread(): - data = self._create_data() - self._stress_read(data) - - def write_thread(): - data = self._create_data() - self._stress_write(data) - - threads = [] - for i in range(self.thread_count): - threads.append(threading.Thread(target=read_thread)) - threads.append(threading.Thread(target=write_thread)) - - for t in threads: - t.start() - self._stress_read(self.data) - for t in threads: - t.join() - self._validate_stress() - diff --git a/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass b/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass index b57650d5..3a3db55d 100644 --- a/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass +++ b/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass @@ -9,7 +9,7 @@ def stamptask(d): with open(stampname, "a+") as f: f.write(d.getVar("BB_UNIHASH") + "\n") - if d.getVar("BB_CURRENT_MC") != "default": + if d.getVar("BB_CURRENT_MC") != "": thistask = d.expand("${BB_CURRENT_MC}:${PN}:${BB_CURRENTTASK}") if thistask in d.getVar("SLOWTASKS").split(): bb.note("Slowing task %s" % thistask) @@ -98,7 +98,7 @@ do_package_qa[rdeptask] = "do_packagedata" do_populate_lic_deploy[recrdeptask] += "do_populate_lic do_deploy" DEBIANRDEP = "do_packagedata" -oo_package_write_ipk[rdeptask] = "${DEBIANRDEP}" +do_package_write_ipk[rdeptask] = "${DEBIANRDEP}" do_package_write_rpm[rdeptask] = "${DEBIANRDEP}" addtask fetch diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb b/bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb new file mode 100644 index 00000000..3c7dca02 --- /dev/null +++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb @@ -0,0 +1,2 @@ +do_build[mcdepends] = "mc::mc-1:h1:do_invalid" + diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/h1.bb b/bitbake/lib/bb/tests/runqueue-tests/recipes/h1.bb new file mode 100644 index 00000000..e69de29b diff --git a/bitbake/lib/bb/tests/runqueue.py b/bitbake/lib/bb/tests/runqueue.py index cc87e8d6..74f5ded2 100644 --- a/bitbake/lib/bb/tests/runqueue.py +++ b/bitbake/lib/bb/tests/runqueue.py @@ -26,7 +26,7 @@ class RunQueueTests(unittest.TestCase): a1_sstatevalid = "a1:do_package a1:do_package_qa a1:do_packagedata a1:do_package_write_ipk a1:do_package_write_rpm a1:do_populate_lic a1:do_populate_sysroot" b1_sstatevalid = "b1:do_package b1:do_package_qa b1:do_packagedata b1:do_package_write_ipk b1:do_package_write_rpm b1:do_populate_lic b1:do_populate_sysroot" - def run_bitbakecmd(self, cmd, builddir, sstatevalid="", slowtasks="", extraenv=None, cleanup=False): + def run_bitbakecmd(self, cmd, builddir, sstatevalid="", slowtasks="", extraenv=None, cleanup=False, allowfailure=False): env = os.environ.copy() env["BBPATH"] = os.path.realpath(os.path.join(os.path.dirname(__file__), "runqueue-tests")) env["BB_ENV_PASSTHROUGH_ADDITIONS"] = "SSTATEVALID SLOWTASKS TOPDIR" @@ -41,6 +41,8 @@ class RunQueueTests(unittest.TestCase): output = subprocess.check_output(cmd, env=env, stderr=subprocess.STDOUT,universal_newlines=True, cwd=builddir) print(output) except subprocess.CalledProcessError as e: + if allowfailure: + return e.output self.fail("Command %s failed with %s" % (cmd, e.output)) tasks = [] tasklog = builddir + "/task.log" @@ -314,6 +316,13 @@ class RunQueueTests(unittest.TestCase): ["mc_2:a1:%s" % t for t in rerun_tasks] self.assertEqual(set(tasks), set(expected)) + # Check that a multiconfig that doesn't exist rasies a correct error message + error_output = self.run_bitbakecmd(["bitbake", "g1"], tempdir, "", extraenv=extraenv, cleanup=True, allowfailure=True) + self.assertIn("non-existent task", error_output) + # If the word 'Traceback' or 'KeyError' is in the output we've regressed + self.assertNotIn("Traceback", error_output) + self.assertNotIn("KeyError", error_output) + self.shutdown(tempdir) def test_hashserv_single(self): diff --git a/bitbake/lib/bb/tests/setup.py b/bitbake/lib/bb/tests/setup.py new file mode 100644 index 00000000..638d56d3 --- /dev/null +++ b/bitbake/lib/bb/tests/setup.py @@ -0,0 +1,728 @@ +# +# Copyright BitBake Contributors +# +# SPDX-License-Identifier: GPL-2.0-only +# + +from bb.tests.fetch import FetcherTest +import bb +import bb.process +import glob +import hashlib +import json +import os +import stat +from bb.tests.support.httpserver import HTTPService + +class BitbakeSetupTest(FetcherTest): + def setUp(self): + super(BitbakeSetupTest, self).setUp() + + self.registrypath = os.path.join(self.tempdir, "bitbake-setup-configurations") + + os.makedirs(self.registrypath) + self.git_init(cwd=self.registrypath) + self.git('commit --allow-empty -m "Initial commit"', cwd=self.registrypath) + + self.testrepopath = os.path.join(self.tempdir, "test-repo") + os.makedirs(self.testrepopath) + self.git_init(cwd=self.testrepopath) + self.git('commit --allow-empty -m "Initial commit"', cwd=self.testrepopath) + + oeinitbuildenv = """BBPATH=$1 +export BBPATH +PATH={}:$PATH +""".format(os.path.join(self.testrepopath, 'scripts')) + self.add_file_to_testrepo('oe-init-build-env',oeinitbuildenv, script=True) + + oesetupbuild = """#!/usr/bin/env python3 +import getopt +import sys +import os +import shutil +opts, args = getopt.getopt(sys.argv[2:], "c:b:", ["no-shell"]) +for option, value in opts: + if option == '-c': + template = value + if option == '-b': + builddir = value +confdir = os.path.join(builddir, 'conf') +os.makedirs(confdir, exist_ok=True) +with open(os.path.join(confdir, 'conf-summary.txt'), 'w') as f: + f.write(template) +shutil.copy(os.path.join(os.path.dirname(__file__), 'test-repo/test-file'), confdir) +with open(os.path.join(builddir, 'init-build-env'), 'w') as f: + f.write("BBPATH={}\\nexport BBPATH\\nPATH={}:$PATH".format(builddir, os.path.join(os.path.dirname(__file__), 'test-repo/scripts'))) +""" + self.add_file_to_testrepo('scripts/oe-setup-build', oesetupbuild, script=True) + + installbuildtools = """#!/usr/bin/env python3 +import getopt +import sys +import os + +opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory="]) +for option, value in opts: + if option == '-d': + installdir = value + +print("Buildtools installed into {}".format(installdir)) +os.makedirs(installdir) +""" + self.add_file_to_testrepo('scripts/install-buildtools', installbuildtools, script=True) + + bitbakeconfigbuild = """#!/usr/bin/env python3 +import os +import sys +confdir = os.path.join(os.environ['BBPATH'], 'conf') +fragment = sys.argv[2] +with open(os.path.join(confdir, fragment), 'w') as f: + f.write('') +""" + self.add_file_to_testrepo('scripts/bitbake-config-build', bitbakeconfigbuild, script=True) + + sometargetexecutable_template = """#!/usr/bin/env python3 +import os +print("This is {}") +print("BBPATH is {{}}".format(os.environ["BBPATH"])) +""" + for e_name in ("some-target-executable-1", "some-target-executable-2"): + sometargetexecutable = sometargetexecutable_template.format(e_name) + self.add_file_to_testrepo('scripts/{}'.format(e_name), sometargetexecutable, script=True) + + def runbbsetup(self, cmd): + bbsetup = os.path.abspath(os.path.dirname(__file__) + "/../../../bin/bitbake-setup") + return bb.process.run("{} --global-settings {} {}".format(bbsetup, os.path.join(self.tempdir, 'global-config'), cmd)) + + + def _add_json_config_to_registry_helper(self, name, sources): + config = """ +{ + "sources": { +%s + }, + "description": "Test configuration", + "bitbake-setup": { + "configurations": [ + { + "name": "gadget", + "description": "Gadget configuration", + "oe-template": "test-configuration-gadget", + "oe-fragments": ["test-fragment-1"] + }, + { + "name": "gizmo", + "description": "Gizmo configuration", + "oe-template": "test-configuration-gizmo", + "oe-fragments": ["test-fragment-2"], + "setup-dir-name": "this-is-a-custom-gizmo-build" + }, + { + "name": "gizmo-env-passthrough", + "description": "Gizmo configuration with environment-passthrough", + "bb-layers": ["layerC","layerD/meta-layer"], + "oe-fragments": ["test-fragment-1"], + "bb-env-passthrough-additions": [ + "BUILD_ID", + "BUILD_DATE", + "BUILD_SERVER" + ] + }, + { + "name": "gizmo-no-fragment", + "description": "Gizmo no-fragment template-only configuration", + "oe-template": "test-configuration-gizmo" + }, + { + "name": "gadget-notemplate", + "description": "Gadget notemplate configuration", + "bb-layers": ["layerA","layerB/meta-layer"], + "oe-fragments": ["test-fragment-1"] + }, + { + "name": "gizmo-notemplate-fragments-one-of", + "description": "Gizmo notemplate configuration", + "bb-layers": ["layerC","layerD/meta-layer"], + "oe-fragments": ["test-fragment-2"], + "oe-fragments-one-of": { + "fragment-desc": { + "description": "Test fragments (with description)", + "options" : [ + { "name": "fragment-desc-1", "description": "fragment 1 desc" }, + { "name": "fragment-desc-2", "description": "fragment 2 desc" } + ] + }, + "fragment-nodesc": { + "description": "Test fragments (no description)", + "options" : [ + "fragment-nodesc-1", + "fragment-nodesc-2" + ] + } + } + }, + { + "name": "gizmo-notemplate-with-filerelative-layers", + "description": "Gizmo notemplate configuration using filerelative layers", + "bb-layers": ["layerC","layerD/meta-layer"], + "bb-layers-file-relative": ["layerE/meta-layer"], + "oe-fragments": ["test-fragment-2"] + } + ] + }, + "version": "1.0" +} +""" % (sources) + os.makedirs(os.path.join(self.registrypath, os.path.dirname(name)), exist_ok=True) + with open(os.path.join(self.registrypath, name), 'w') as f: + f.write(config) + self.git('add {}'.format(name), cwd=self.registrypath) + self.git('commit -m "Adding {}"'.format(name), cwd=self.registrypath) + return json.loads(config) + + def add_json_config_to_registry(self, name, rev, branch): + sources = """ + "test-repo": { + "git-remote": { + "remotes": { + "origin": { + "uri": "file://%s" + } + }, + "branch": "%s", + "rev": "%s" + } + } +""" % (self.testrepopath, branch, rev) + return self._add_json_config_to_registry_helper(name, sources) + + def add_local_json_config_to_registry(self, name, path): + sources = """ + "test-repo": { + "local": { + "path": "%s" + } + } +""" % (path) + return self._add_json_config_to_registry_helper(name, sources) + + def add_file_to_testrepo(self, name, content, script=False): + fullname = os.path.join(self.testrepopath, name) + os.makedirs(os.path.join(self.testrepopath, os.path.dirname(name)), exist_ok=True) + with open(fullname, 'w') as f: + f.write(content) + if script: + st = os.stat(fullname) + os.chmod(fullname, st.st_mode | stat.S_IEXEC) + self.git('add {}'.format(name), cwd=self.testrepopath) + self.git('commit -m "Adding {}"'.format(name), cwd=self.testrepopath) + + def config_is_unchanged(self, setuppath): + os.environ['BBPATH'] = os.path.join(setuppath, 'build') + out = self.runbbsetup("status") + self.assertIn("Configuration in {} has not changed".format(setuppath), out[0]) + out = self.runbbsetup("update --update-bb-conf='yes'") + self.assertIn("Configuration in {} has not changed".format(setuppath), out[0]) + del os.environ['BBPATH'] + + def check_setupdir_files(self, setuppath, test_file_content): + with open(os.path.join(setuppath, 'config', "config-upstream.json")) as f: + config_upstream = json.load(f) + with open(os.path.join(setuppath, 'layers', 'test-repo', 'test-file')) as f: + self.assertEqual(f.read(), test_file_content) + bitbake_config = config_upstream["bitbake-config"] + bb_build_path = os.path.join(setuppath, 'build') + bb_conf_path = os.path.join(bb_build_path, 'conf') + self.assertTrue(os.path.exists(os.path.join(bb_build_path, 'init-build-env'))) + + with open(os.path.join(setuppath, 'config', "sources-fixed-revisions.json")) as f: + sources_fixed_revisions = json.load(f) + self.assertTrue('test-repo' in sources_fixed_revisions['sources'].keys()) + revision = self.git('rev-parse HEAD', cwd=self.testrepopath).strip() + self.assertEqual(revision, sources_fixed_revisions['sources']['test-repo']['git-remote']['rev']) + + if "oe-template" in bitbake_config: + with open(os.path.join(bb_conf_path, 'conf-summary.txt')) as f: + self.assertEqual(f.read(), bitbake_config["oe-template"]) + with open(os.path.join(bb_conf_path, 'test-file')) as f: + self.assertEqual(f.read(), test_file_content) + else: + with open(os.path.join(bb_conf_path, 'conf-summary.txt')) as f: + self.assertIn(bitbake_config["description"], f.read()) + with open(os.path.join(bb_conf_path, 'bblayers.conf')) as f: + bblayers = f.read() + for l in bitbake_config["bb-layers"]: + self.assertIn(os.path.join(setuppath, "layers", l), bblayers) + for l in bitbake_config.get("bb-layers-file-relative") or []: + filerelative_layer = os.path.join( + os.path.dirname(config_upstream["path"]), + l, + ) + self.assertIn(filerelative_layer, bblayers) + + if 'oe-fragments' in bitbake_config: + for f in bitbake_config["oe-fragments"]: + self.assertTrue(os.path.exists(os.path.join(bb_conf_path, f))) + + if 'bb-env-passthrough-additions' in bitbake_config: + with open(os.path.join(bb_build_path, 'init-build-env'), 'r') as f: + init_build_env = f.read() + self.assertTrue('BB_ENV_PASSTHROUGH_ADDITIONS' in init_build_env) + self.assertTrue('BUILD_ID' in init_build_env) + self.assertTrue('BUILD_DATE' in init_build_env) + self.assertTrue('BUILD_SERVER' in init_build_env) + # a more throrough test could be to initialize a bitbake build-env, export FOO to the shell environment, set the env-passthrough on it and finally check against 'bitbake-getvar FOO' + + self.config_is_unchanged(setuppath) + + def get_setup_path(self, cf, c): + if c == 'gizmo': + return os.path.join(self.tempdir, 'bitbake-builds', 'this-is-a-custom-gizmo-build') + return os.path.join(self.tempdir, 'bitbake-builds', '{}-{}'.format(cf, "-".join(c.split()))) + + def test_setup(self): + # unset BBPATH to ensure tests run in isolation from the existing bitbake environment + if 'BBPATH' in os.environ: + del os.environ['BBPATH'] + + # check that no arguments works + self.runbbsetup("") + + # check that --help works + self.runbbsetup("--help") + + # change to self.tempdir to work with cwd-based default settings + os.chdir(self.tempdir) + + # check that the default top-dir-prefix is cwd (now self.tempdir) with no global settings + out = self.runbbsetup("settings list") + self.assertIn("default top-dir-prefix {}".format(os.getcwd()), out[0]) + + # set up global location for dl-dir + settings_path = "{}/global-config".format(self.tempdir) + out = self.runbbsetup("settings set --global default dl-dir {}".format(os.path.join(self.tempdir, 'downloads'))) + self.assertIn("From section 'default' the setting 'dl-dir' was changed to", out[0]) + self.assertIn("Settings written to".format(settings_path), out[0]) + + # check that writing settings works and then adjust them to point to + # test registry repo + out = self.runbbsetup("settings set default registry 'git://{};protocol=file;branch=master;rev=master'".format(self.registrypath)) + settings_path = "{}/bitbake-builds/settings.conf".format(self.tempdir) + self.assertIn(settings_path, out[0]) + self.assertIn("From section 'default' the setting 'registry' was changed to", out[0]) + self.assertIn("Settings written to".format(settings_path), out[0]) + + # check that listing settings works + out = self.runbbsetup("settings list") + self.assertIn("default top-dir-prefix {}".format(self.tempdir), out[0]) + self.assertIn("default dl-dir {}".format(os.path.join(self.tempdir, 'downloads')), out[0]) + self.assertIn("default registry {}".format('git://{};protocol=file;branch=master;rev=master'.format(self.registrypath)), out[0]) + + # check that 'list' produces correct output with no configs, one config and two configs + out = self.runbbsetup("list") + self.assertNotIn("test-config-1", out[0]) + self.assertNotIn("test-config-2", out[0]) + + json_1 = self.add_json_config_to_registry('test-config-1.conf.json', 'master', 'master') + out = self.runbbsetup("list") + self.assertIn("test-config-1", out[0]) + self.assertNotIn("test-config-2", out[0]) + + json_2 = self.add_json_config_to_registry('config-2/test-config-2.conf.json', 'master', 'master') + out = self.runbbsetup("list --write-json={}".format(os.path.join(self.tempdir, "test-configs.json"))) + self.assertIn("test-config-1", out[0]) + self.assertIn("test-config-2", out[0]) + with open(os.path.join(self.tempdir, "test-configs.json")) as f: + json_configs = json.load(f) + self.assertIn("test-config-1", json_configs) + self.assertIn("test-config-2", json_configs) + + # check that init/status/update work + # (the latter two should do nothing and say that config hasn't changed) + test_file_content = 'initial\n' + self.add_file_to_testrepo('test-file', test_file_content) + + # test-config-1 is tested as a registry config and over http, test-config-2 as a local file + server = HTTPService(self.registrypath, host="127.0.0.1") + server.start() + variants = ( + 'gadget', + 'gizmo', + 'gizmo-env-passthrough', + 'gizmo-no-fragment', + 'gadget-notemplate', + 'gizmo-notemplate-fragments-one-of fragment-desc-1 fragment-nodesc-1', + ) + variants_local = variants + ('gizmo-notemplate-with-filerelative-layers',) + test_configurations = ({'name':'test-config-1','cmdline': 'test-config-1', + 'buildconfigs': variants}, + {'name':'test-config-2','cmdline': os.path.join(self.registrypath,'config-2/test-config-2.conf.json'), + 'buildconfigs': variants_local}, + {'name':'test-config-1','cmdline':'http://127.0.0.1:{}/test-config-1.conf.json'.format(server.port), + 'buildconfigs': variants} + ) + try: + for v in test_configurations: + for c in v['buildconfigs']: + out = self.runbbsetup("init --non-interactive {} {}".format(v['cmdline'], c)) + setuppath = self.get_setup_path(v['name'], c) + self.check_setupdir_files(setuppath, test_file_content) + finally: + server.stop() + + # install buildtools + out = self.runbbsetup("install-buildtools --setup-dir {}".format(setuppath)) + self.assertIn("Buildtools installed into", out[0]) + self.assertTrue(os.path.exists(os.path.join(setuppath, 'buildtools'))) + + # change a file in the test layer repo, make a new commit and + # test that status/update correctly report the change and update the config + prev_test_file_content = test_file_content + test_file_content = 'modified\n' + self.add_file_to_testrepo('test-file', test_file_content) + for c in variants: + setuppath = self.get_setup_path('test-config-1', c) + os.environ['BBPATH'] = os.path.join(setuppath, 'build') + out = self.runbbsetup("status") + self.assertIn("Layer repository file://{} checked out into {}/layers/test-repo updated revision master from".format(self.testrepopath, setuppath), out[0]) + out = self.runbbsetup("update --update-bb-conf='yes'") + if c in ('gadget', 'gizmo'): + self.assertIn("Leaving the previous configuration in {}/build/conf-backup.".format(setuppath), out[0]) + self.assertIn('-{}+{}'.format(prev_test_file_content, test_file_content), out[0]) + self.check_setupdir_files(setuppath, test_file_content) + + # make a new branch in the test layer repo, change a file on that branch, + # make a new commit, update the top level json config to refer to that branch, + # and test that status/update correctly report the change and update the config + prev_test_file_content = test_file_content + test_file_content = 'modified-in-branch\n' + branch = "another-branch" + self.git('checkout -b {}'.format(branch), cwd=self.testrepopath) + self.add_file_to_testrepo('test-file', test_file_content) + json_1 = self.add_json_config_to_registry('test-config-1.conf.json', branch, branch) + for c in variants: + setuppath = self.get_setup_path('test-config-1', c) + os.environ['BBPATH'] = os.path.join(setuppath, 'build') + out = self.runbbsetup("status") + self.assertIn("Configuration in {} has changed:".format(setuppath), out[0]) + self.assertIn('- "rev": "master"\n+ "rev": "another-branch"', out[0]) + out = self.runbbsetup("update --update-bb-conf='yes'") + if c in ('gadget', 'gizmo'): + self.assertIn("Leaving the previous configuration in {}/build/conf-backup.".format(setuppath), out[0]) + self.assertIn('-{}+{}'.format(prev_test_file_content, test_file_content), out[0]) + self.check_setupdir_files(setuppath, test_file_content) + + # do the same as the previous test, but now without updating the bitbake configuration (--update-bb-conf=no) + # and check that files have not been modified + def _conf_chksum(confdir: str) -> list: + sums = [] + for f in glob.glob(f'{confdir}/*'): + if os.path.isfile(f) and not os.path.islink(f): + with open(f, 'rb') as fd: + sha = hashlib.sha256() + sha.update(fd.read()) + sums.append(os.path.basename(f) + '_' + sha.hexdigest()) + return sums + + prev_test_file_content = test_file_content + test_file_content = 'modified-in-branch-no-bb-conf-update\n' + branch = "another-branch-no-bb-conf-update" + self.git('checkout -b {}'.format(branch), cwd=self.testrepopath) + self.add_file_to_testrepo('test-file', test_file_content) + json_1 = self.add_json_config_to_registry('test-config-1.conf.json', branch, branch) + for c in variants: + setuppath = self.get_setup_path('test-config-1', c) + os.environ['BBPATH'] = os.path.join(setuppath, 'build') + # write something in local.conf and bblayers.conf + for f in ["local.conf", "bblayers.conf"]: + with open(f"{setuppath}/build/conf/{f}", "w") as fd: + fd.write("deadbeef") + sums_before = _conf_chksum(f"{setuppath}/build/conf") + out = self.runbbsetup("update --update-bb-conf='no'") + sums_after = _conf_chksum(f"{setuppath}/build/conf") + self.assertEqual(sums_before, sums_after) + + def _check_local_sources(custom_setup_dir): + custom_setup_path = os.path.join(self.tempdir, 'bitbake-builds', custom_setup_dir) + custom_layer_path = os.path.join(custom_setup_path, 'layers', 'test-repo') + self.assertTrue(os.path.islink(custom_layer_path)) + self.assertEqual(self.testrepopath, os.path.realpath(custom_layer_path)) + self.config_is_unchanged(custom_setup_path) + + def _check_layer_backups(layer_path, expected_backups): + files = os.listdir(layer_path) + backups = len([f for f in files if 'backup' in f]) + self.assertEqual(backups, expected_backups, msg = "Expected {} layer backups, got {}, directory listing: {}".format(expected_backups, backups, files)) + + # Change the configuration to refer to a local source, then to another local source, then back to a git remote + # Run status/update after each change and verify that nothing breaks + # Also check that layer backups happen when expected + c = 'gadget' + setuppath = self.get_setup_path('test-config-1', c) + self.config_is_unchanged(setuppath) + + layers_path = os.path.join(setuppath, 'layers') + layer_path = os.path.join(layers_path, 'test-repo') + _check_layer_backups(layers_path, 0) + + json_1 = self.add_local_json_config_to_registry('test-config-1.conf.json', self.testrepopath) + os.environ['BBPATH'] = os.path.join(setuppath, 'build') + out = self.runbbsetup("update --update-bb-conf='yes'") + _check_local_sources(setuppath) + _check_layer_backups(layers_path, 1) + + prev_path = self.testrepopath + self.testrepopath = prev_path + "-2" + self.git("clone {} {}".format(prev_path, self.testrepopath), cwd=self.tempdir) + json_1 = self.add_local_json_config_to_registry('test-config-1.conf.json', self.testrepopath) + os.environ['BBPATH'] = os.path.join(setuppath, 'build') + out = self.runbbsetup("update --update-bb-conf='yes'") + _check_local_sources(setuppath) + _check_layer_backups(layers_path, 1) + + self.testrepopath = prev_path + json_1 = self.add_json_config_to_registry('test-config-1.conf.json', branch, branch) + os.environ['BBPATH'] = os.path.join(setuppath, 'build') + out = self.runbbsetup("update --update-bb-conf='yes'") + self.check_setupdir_files(setuppath, test_file_content) + _check_layer_backups(layers_path, 1) + + ## edit a file without making a commit + with open(os.path.join(layer_path, 'local-modification'), 'w') as f: + f.write('locally-modified\n') + test_file_content = "modified-again\n" + self.add_file_to_testrepo('test-file', test_file_content) + os.environ['BBPATH'] = os.path.join(setuppath, 'build') + out = self.runbbsetup("update --update-bb-conf='yes'") + _check_layer_backups(layers_path, 1) + + ## edit a file and make a commit such that no rebase conflicts occur + with open(os.path.join(layer_path, 'local-modification'), 'w') as f: + f.write('locally-modified-again\n') + self.git('add .', cwd=layer_path) + self.git('commit -m "Adding a local modification"', cwd=layer_path) + test_file_content = "modified-again-and-again\n" + self.add_file_to_testrepo('test-file', test_file_content) + out = self.runbbsetup("update --update-bb-conf='yes'") + _check_layer_backups(layers_path, 1) + + ## edit a file and make a commit in a way that causes a rebase conflict + with open(os.path.join(layer_path, 'test-file'), 'w') as f: + f.write('locally-modified\n') + self.git('add .', cwd=layer_path) + self.git('commit -m "Adding a local modification"', cwd=layer_path) + test_file_content = "remotely-modified\n" + self.add_file_to_testrepo('test-file', test_file_content) + with self.assertRaisesRegex(bb.process.ExecutionError, "Merge conflict in test-file"): + out = self.runbbsetup("update --update-bb-conf='yes'") + _check_layer_backups(layers_path, 1) + + # check source overrides, local sources provided with symlinks, and custom setup dir name + source_override_content = """ +{ + "sources": { + "test-repo": { + "local": { + "path": "." + } + } + } +}""" + override_filename = 'source-overrides.json' + custom_setup_dir = 'special-setup-dir' + self.add_file_to_testrepo(override_filename, source_override_content) + out = self.runbbsetup("init --non-interactive --source-overrides {} --setup-dir-name {} test-config-1 gadget".format(os.path.join(self.testrepopath, override_filename), custom_setup_dir)) + _check_local_sources(custom_setup_dir) + + # same but use command line options to specify local overrides + custom_setup_dir = 'special-setup-dir-with-cmdline-overrides' + out = self.runbbsetup("init --non-interactive -L test-repo {} --setup-dir-name {} test-config-1 gadget".format(self.testrepopath, custom_setup_dir)) + _check_local_sources(custom_setup_dir) + + def test_vscode(self): + if 'BBPATH' in os.environ: + del os.environ['BBPATH'] + os.chdir(self.tempdir) + + self.runbbsetup("settings set default registry 'git://{};protocol=file;branch=master;rev=master'".format(self.registrypath)) + self.add_file_to_testrepo('test-file', 'initial\n') + self.add_json_config_to_registry('test-config-1.conf.json', 'master', 'master') + + # --init-vscode should create bitbake.code-workspace + self.runbbsetup("init --non-interactive --init-vscode test-config-1 gadget") + setuppath = self.get_setup_path('test-config-1', 'gadget') + workspace_file = os.path.join(setuppath, 'bitbake.code-workspace') + self.assertTrue(os.path.exists(workspace_file), + "bitbake.code-workspace should be created with --init-vscode") + + with open(workspace_file) as f: + workspace = json.load(f) + + # top-level structure + self.assertIn('folders', workspace) + self.assertIn('settings', workspace) + self.assertIn('extensions', workspace) + self.assertIn('yocto-project.yocto-bitbake', + workspace['extensions']['recommendations']) + + # folders: conf dir + test-repo (symlinks like oe-init-build-env-dir are skipped) + folder_names = {f['name'] for f in workspace['folders']} + self.assertIn('conf', folder_names) + self.assertIn('test-repo', folder_names) + + # folder paths must be relative so the workspace is portable + for f in workspace['folders']: + self.assertFalse(os.path.isabs(f['path']), + "Folder path should be relative, got: {}".format(f['path'])) + + # bitbake extension settings + settings = workspace['settings'] + self.assertTrue(settings.get('bitbake.disableConfigModification')) + self.assertEqual(settings['bitbake.pathToBuildFolder'], + os.path.join(setuppath, 'build')) + self.assertEqual(settings['bitbake.pathToEnvScript'], + os.path.join(setuppath, 'build', 'init-build-env')) + + # file associations + self.assertIn('*.conf', settings.get('files.associations', {})) + self.assertIn('*.inc', settings.get('files.associations', {})) + + # python extra paths: test-repo/scripts/ exists and should be listed + extra_paths = settings.get('python.analysis.extraPaths', []) + self.assertTrue(any('scripts' in p for p in extra_paths), + "python.analysis.extraPaths should include the scripts dir") + self.assertEqual(settings.get('python.analysis.extraPaths'), + settings.get('python.autoComplete.extraPaths')) + + # --no-init-vscode should NOT create a workspace file + self.runbbsetup("init --non-interactive --no-init-vscode test-config-1 gadget-notemplate") + notemplate_path = self.get_setup_path('test-config-1', 'gadget-notemplate') + self.assertFalse( + os.path.exists(os.path.join(notemplate_path, 'bitbake.code-workspace')), + "bitbake.code-workspace should not be created with --no-init-vscode") + + # update with --init-vscode after a layer change should preserve + # user-added folders and settings while still rewriting managed ones + workspace['folders'].append({"name": "user-folder", "path": "user/custom"}) + workspace['settings']['my.user.setting'] = 'preserved' + with open(workspace_file, 'w') as f: + json.dump(workspace, f, indent=4) + + self.add_file_to_testrepo('test-file', 'updated\n') + os.environ['BBPATH'] = os.path.join(setuppath, 'build') + self.runbbsetup("update --update-bb-conf='no'") + del os.environ['BBPATH'] + + with open(workspace_file) as f: + updated = json.load(f) + self.assertIn('user/custom', {f['path'] for f in updated['folders']}, + "User-added folder was removed during update") + self.assertIn('my.user.setting', updated['settings'], + "User-added setting was removed during update") + + # update with a corrupt workspace file should log an error and leave it unchanged + self.add_file_to_testrepo('test-file', 'updated-again\n') + with open(workspace_file, 'w') as f: + f.write('{invalid json') + os.environ['BBPATH'] = os.path.join(setuppath, 'build') + self.runbbsetup("update --update-bb-conf='no'") + del os.environ['BBPATH'] + with open(workspace_file) as f: + content = f.read() + self.assertEqual(content, '{invalid json', + "Corrupt workspace file should not be modified") + + def _count_layer_backups(self, layers_path): + return len([f for f in os.listdir(layers_path) if 'backup' in f]) + + def test_update_rebase_conflicts_strategy(self): + """Test the --rebase-conflicts-strategy option for the update command. + + Covers three scenarios not exercised by test_setup: + 1. Uncommitted tracked-file change (LocalModificationsError) + default 'abort' + strategy → clean error message containing 'has uncommitted changes' and a + hint at --rebase-conflicts-strategy=backup; no backup directory is created. + 2. Same uncommitted change + 'backup' strategy → directory is renamed to a + timestamped backup and the layer is re-cloned cleanly. + 3. Committed local change that conflicts with an incoming upstream commit + (RebaseError): + a. Default 'abort' strategy → error containing 'Merge conflict' and the + --rebase-conflicts-strategy=backup hint; no backup directory is created. + b. 'backup' strategy → backup + re-clone instead of a hard failure. + """ + if 'BBPATH' in os.environ: + del os.environ['BBPATH'] + os.chdir(self.tempdir) + + self.runbbsetup("settings set default registry 'git://{};protocol=file;branch=master;rev=master'".format(self.registrypath)) + self.add_file_to_testrepo('test-file', 'initial\n') + self.add_json_config_to_registry('test-config-1.conf.json', 'master', 'master') + self.runbbsetup("init --non-interactive test-config-1 gadget") + + setuppath = self.get_setup_path('test-config-1', 'gadget') + layer_path = os.path.join(setuppath, 'layers', 'test-repo') + layers_path = os.path.join(setuppath, 'layers') + + # Scenario 1: uncommitted tracked change, default 'abort' strategy + # Advance upstream so an update is required. + self.add_file_to_testrepo('test-file', 'upstream-v2\n') + # Modify the same tracked file in the layer without committing. + with open(os.path.join(layer_path, 'test-file'), 'w') as f: + f.write('locally-modified\n') + + os.environ['BBPATH'] = os.path.join(setuppath, 'build') + with self.assertRaises(bb.process.ExecutionError) as ctx: + self.runbbsetup("update --update-bb-conf='no'") + self.assertIn('has uncommitted changes', str(ctx.exception)) + self.assertIn('--rebase-conflicts-strategy=backup', str(ctx.exception)) + # No backup directory must have been created. + self.assertEqual(self._count_layer_backups(layers_path), 0, + "abort strategy must not create any backup") + + # Scenario 2: same uncommitted change, 'backup' strategy + out = self.runbbsetup("update --update-bb-conf='no' --rebase-conflicts-strategy=backup") + # One backup directory must now exist. + self.assertEqual(self._count_layer_backups(layers_path), 1, + "backup strategy must create exactly one backup") + # The re-cloned layer must be clean and at the upstream revision. + with open(os.path.join(layer_path, 'test-file')) as f: + self.assertEqual(f.read(), 'upstream-v2\n', + "re-cloned layer must contain the upstream content") + status = self.git('status --porcelain', cwd=layer_path).strip() + self.assertEqual(status, '', + "re-cloned layer must have no local modifications") + del os.environ['BBPATH'] + + # Scenario 3: committed conflicting change, 'backup' strategy + # Re-initialise a fresh setup so we start from a clean state. + self.runbbsetup("init --non-interactive --setup-dir-name rebase-conflict-setup test-config-1 gadget") + conflict_setup = os.path.join(self.tempdir, 'bitbake-builds', 'rebase-conflict-setup') + conflict_layer = os.path.join(conflict_setup, 'layers', 'test-repo') + conflict_layers = os.path.join(conflict_setup, 'layers') + + # Commit a local change that touches the same file as the next upstream commit. + with open(os.path.join(conflict_layer, 'test-file'), 'w') as f: + f.write('conflicting-local\n') + self.git('add test-file', cwd=conflict_layer) + self.git('commit -m "Local conflicting change"', cwd=conflict_layer) + + # Advance upstream with a conflicting edit. + self.add_file_to_testrepo('test-file', 'conflicting-upstream\n') + + os.environ['BBPATH'] = os.path.join(conflict_setup, 'build') + # Default stop strategy must still fail with a conflict error and include + # the --rebase-conflicts-strategy=backup hint (same handler as LocalModificationsError). + with self.assertRaises(bb.process.ExecutionError) as ctx: + self.runbbsetup("update --update-bb-conf='no'") + self.assertIn('Merge conflict in test-file', str(ctx.exception)) + self.assertIn('--rebase-conflicts-strategy=backup', str(ctx.exception)) + self.assertEqual(self._count_layer_backups(conflict_layers), 0) + + # Backup strategy must succeed: backup the conflicted dir and re-clone. + self.runbbsetup("update --update-bb-conf='no' --rebase-conflicts-strategy=backup") + self.assertEqual(self._count_layer_backups(conflict_layers), 1, + "backup strategy must create exactly one backup after a conflict") + with open(os.path.join(conflict_layer, 'test-file')) as f: + self.assertEqual(f.read(), 'conflicting-upstream\n', + "re-cloned layer must contain the upstream content after conflict backup") + del os.environ['BBPATH'] diff --git a/bitbake/lib/bb/tests/siggen.py b/bitbake/lib/bb/tests/siggen.py index 0dc67e6c..eb07cc92 100644 --- a/bitbake/lib/bb/tests/siggen.py +++ b/bitbake/lib/bb/tests/siggen.py @@ -9,7 +9,9 @@ import unittest import logging import bb +import bb.data import time +from contextlib import contextmanager logger = logging.getLogger('BitBake.TestSiggen') @@ -26,3 +28,48 @@ class SiggenTest(unittest.TestCase): for t in tests: self.assertEqual(bb.siggen.build_pnid(*t), tests[t]) + def test_get_unihashes_rejects_invalid_hashserv_unihash(self): + class TestClient: + def get_unihash_batch(self, query): + list(query) + return ["${@os.system('true')}"] + + class TestSiggen(bb.siggen.SignatureGeneratorUniHashMixIn): + def __init__(self): + self.server = "test-server" + self.method = "test-method" + self.extramethod = {} + self.taskhash = {"test.bb:do_compile": "a" * 64} + self.unihash = {} + self.unitaskhashes = {} + self.tidtopn = {} + self.setscenetasks = set() + + @contextmanager + def client(self): + yield TestClient() + + siggen = TestSiggen() + + with self.assertRaises(bb.BBHandledException): + siggen.get_unihashes(["test.bb:do_compile"]) + + self.assertEqual(siggen.unihash, {}) + self.assertEqual(siggen.unitaskhashes, {}) + + def test_report_unihash_reads_bb_unihash_without_expansion(self): + class TestSiggen(bb.siggen.SignatureGeneratorUniHashMixIn): + def __init__(self): + self.setscenetasks = set() + self.taskhash = {"test.bb:do_compile": "b" * 64} + + d = bb.data.init() + d.setVar("BB_TASKHASH", "a" * 64) + d.setVar("BB_UNIHASH", "${@d.setVar('EXPANDED_UNIHASH', '1') or 'bad'}") + d.setVar("SSTATE_HASHEQUIV_REPORT_TASKDATA", "0") + d.setVar("T", "/tmp") + d.setVar("BB_FILENAME", "test.bb") + + TestSiggen().report_unihash(".", "compile", d) + + self.assertIsNone(d.getVar("EXPANDED_UNIHASH")) diff --git a/bitbake/lib/bb/tests/support/httpserver.py b/bitbake/lib/bb/tests/support/httpserver.py index 78f76600..03327e92 100644 --- a/bitbake/lib/bb/tests/support/httpserver.py +++ b/bitbake/lib/bb/tests/support/httpserver.py @@ -3,7 +3,7 @@ # import http.server -import multiprocessing +from bb import multiprocessing import os import traceback import signal @@ -43,7 +43,7 @@ class HTTPService(object): self.process = multiprocessing.Process(target=self.server.server_start, args=[self.root_dir, self.logger]) # The signal handler from testimage.bbclass can cause deadlocks here - # if the HTTPServer is terminated before it can restore the standard + # if the HTTPServer is terminated before it can restore the standard #signal behaviour orig = signal.getsignal(signal.SIGTERM) signal.signal(signal.SIGTERM, signal.SIG_DFL) diff --git a/bitbake/lib/bb/tests/utils.py b/bitbake/lib/bb/tests/utils.py index c363f62d..52b7bf85 100644 --- a/bitbake/lib/bb/tests/utils.py +++ b/bitbake/lib/bb/tests/utils.py @@ -130,6 +130,14 @@ class Checksum(unittest.TestCase): checksum = bb.utils.sha256_file(f.name) self.assertEqual(checksum, "fcfbae8bf6b721dbb9d2dc6a9334a58f2031a9a9b302999243f99da4d7f12d0f") + def test_goh1(self): + import hashlib + with tempfile.NamedTemporaryFile() as f: + f.write(self.filler) + f.flush() + checksum = bb.utils.goh1_file(f.name) + self.assertEqual(checksum, "81191f04d4abf413e5badd234814e4202d9efa73e6f9437e9ddd6b8165b569ef") + class EditMetadataFile(unittest.TestCase): _origfile = """ # A comment @@ -684,3 +692,14 @@ class EnvironmentTests(unittest.TestCase): self.assertIn("A", os.environ) self.assertEqual(os.environ["A"], "this is A") self.assertNotIn("B", os.environ) + +class FilemodeTests(unittest.TestCase): + def test_filemode_convert(self): + self.assertEqual(0o775, bb.utils.to_filemode("0o775")) + self.assertEqual(0o775, bb.utils.to_filemode(0o775)) + self.assertEqual(0o775, bb.utils.to_filemode("775")) + with self.assertRaises(ValueError): + bb.utils.to_filemode("xyz") + with self.assertRaises(ValueError): + bb.utils.to_filemode("999") + diff --git a/bitbake/lib/bb/tinfoil.py b/bitbake/lib/bb/tinfoil.py index dcd3910c..9c176894 100644 --- a/bitbake/lib/bb/tinfoil.py +++ b/bitbake/lib/bb/tinfoil.py @@ -14,7 +14,8 @@ import time import atexit import re from collections import OrderedDict, defaultdict -from functools import partial +from functools import partial, wraps +from contextlib import contextmanager import bb.cache import bb.cooker @@ -26,6 +27,135 @@ import bb.remotedata from bb.main import setup_bitbake, BitBakeConfigParameters import bb.fetch2 +def wait_for(f): + """ + Wrap a function that makes an asynchronous tinfoil call using + self.run_command() and wait for events to say that the call has been + successful, or an error has occurred. + """ + @wraps(f) + def wrapper(self, *args, handle_events=True, extra_events=None, event_callback=None, **kwargs): + if handle_events: + # A reasonable set of default events matching up with those we handle below + eventmask = [ + 'bb.event.BuildStarted', + 'bb.event.BuildCompleted', + 'logging.LogRecord', + 'bb.event.NoProvider', + 'bb.command.CommandCompleted', + 'bb.command.CommandFailed', + 'bb.build.TaskStarted', + 'bb.build.TaskFailed', + 'bb.build.TaskSucceeded', + 'bb.build.TaskFailedSilent', + 'bb.build.TaskProgress', + 'bb.runqueue.runQueueTaskStarted', + 'bb.runqueue.sceneQueueTaskStarted', + 'bb.event.ProcessStarted', + 'bb.event.ProcessProgress', + 'bb.event.ProcessFinished', + ] + if extra_events: + eventmask.extend(extra_events) + ret = self.set_event_mask(eventmask) + + includelogs = self.config_data.getVar('BBINCLUDELOGS') + loglines = self.config_data.getVar('BBINCLUDELOGS_LINES') + + # Call actual function + ret = f(self, *args, **kwargs) + + if handle_events: + lastevent = time.time() + result = False + # Borrowed from knotty, instead somewhat hackily we use the helper + # as the object to store "shutdown" on + helper = bb.ui.uihelper.BBUIHelper() + helper.shutdown = 0 + parseprogress = None + termfilter = bb.ui.knotty.TerminalFilter(helper, helper, self.logger.handlers, quiet=self.quiet) + try: + while True: + try: + event = self.wait_event(0.25) + if event: + lastevent = time.time() + if event_callback and event_callback(event): + continue + if helper.eventHandler(event): + if isinstance(event, bb.build.TaskFailedSilent): + self.logger.warning("Logfile for failed setscene task is %s" % event.logfile) + elif isinstance(event, bb.build.TaskFailed): + bb.ui.knotty.print_event_log(event, includelogs, loglines, termfilter) + continue + if isinstance(event, bb.event.ProcessStarted): + if self.quiet > 1: + continue + if parseprogress: + parseprogress.finish() + parseprogress = bb.ui.knotty.new_progress(event.processname, event.total) + parseprogress.start(False) + continue + if isinstance(event, bb.event.ProcessProgress): + if self.quiet > 1: + continue + if parseprogress and parseprogress.id == event.processname: + parseprogress.update(event.progress) + continue + if isinstance(event, bb.event.ProcessFinished): + if self.quiet > 1: + continue + if parseprogress and parseprogress.id == event.processname: + parseprogress.finish() + parseprogress = None + continue + if isinstance(event, bb.command.CommandCompleted): + result = True + break + if isinstance(event, (bb.command.CommandFailed, bb.command.CommandExit)): + self.logger.error(str(event)) + result = False + break + if isinstance(event, logging.LogRecord): + if event.taskpid == 0 or event.levelno > logging.INFO: + self.logger.handle(event) + continue + if isinstance(event, bb.event.NoProvider): + self.logger.error(str(event)) + result = False + break + elif helper.shutdown > 1: + break + termfilter.updateFooter() + if time.time() > (lastevent + (3*60)): + if not self.run_command('ping', handle_events=False): + print("\nUnable to ping server and no events, closing down...\n") + return False + except KeyboardInterrupt: + termfilter.clearFooter() + if helper.shutdown == 1: + print("\nSecond Keyboard Interrupt, stopping...\n") + ret = self.run_command("stateForceShutdown") + if ret and ret[2]: + self.logger.error("Unable to cleanly stop: %s" % ret[2]) + elif helper.shutdown == 0: + print("\nKeyboard Interrupt, closing down...\n") + interrupted = True + ret = self.run_command("stateShutdown") + if ret and ret[2]: + self.logger.error("Unable to cleanly shutdown: %s" % ret[2]) + helper.shutdown = helper.shutdown + 1 + termfilter.clearFooter() + finally: + termfilter.finish() + if helper.failed_tasks: + result = False + return result + else: + return ret + + return wrapper + # We need this in order to shut down the connection to the bitbake server, # otherwise the process will never properly exit @@ -188,11 +318,19 @@ class TinfoilCookerAdapter: self._cache[name] = attrvalue return attrvalue + class TinfoilSkiplistByMcAdapter: + def __init__(self, tinfoil): + self.tinfoil = tinfoil + + def __getitem__(self, mc): + return self.tinfoil.get_skipped_recipes(mc) + def __init__(self, tinfoil): self.tinfoil = tinfoil self.multiconfigs = [''] + (tinfoil.config_data.getVar('BBMULTICONFIG') or '').split() self.collections = {} self.recipecaches = {} + self.skiplist_by_mc = self.TinfoilSkiplistByMcAdapter(tinfoil) for mc in self.multiconfigs: self.collections[mc] = self.TinfoilCookerCollectionAdapter(tinfoil, mc) self.recipecaches[mc] = self.TinfoilRecipeCacheAdapter(tinfoil, mc) @@ -201,8 +339,6 @@ class TinfoilCookerAdapter: # Grab these only when they are requested since they aren't always used if name in self._cache: return self._cache[name] - elif name == 'skiplist': - attrvalue = self.tinfoil.get_skipped_recipes() elif name == 'bbfile_config_priorities': ret = self.tinfoil.run_command('getLayerPriorities') bbfile_config_priorities = [] @@ -514,12 +650,12 @@ class Tinfoil: """ return defaultdict(list, self.run_command('getOverlayedRecipes', mc)) - def get_skipped_recipes(self): + def get_skipped_recipes(self, mc=''): """ Find recipes which were skipped (i.e. SkipRecipe was raised during parsing). """ - return OrderedDict(self.run_command('getSkippedRecipes')) + return OrderedDict(self.run_command('getSkippedRecipes', mc)) def get_all_providers(self, mc=''): return defaultdict(list, self.run_command('allProviders', mc)) @@ -533,6 +669,7 @@ class Tinfoil: def get_runtime_providers(self, rdep): return self.run_command('getRuntimeProviders', rdep) + # TODO: teach this method about mc def get_recipe_file(self, pn): """ Get the file name for the specified recipe/target. Raises @@ -541,13 +678,21 @@ class Tinfoil: """ best = self.find_best_provider(pn) if not best or (len(best) > 3 and not best[3]): + # TODO: pass down mc skiplist = self.get_skipped_recipes() taskdata = bb.taskdata.TaskData(None, skiplist=skiplist) skipreasons = taskdata.get_reasons(pn) if skipreasons: raise bb.providers.NoProvider('%s is unavailable:\n %s' % (pn, ' \n'.join(skipreasons))) else: - raise bb.providers.NoProvider('Unable to find any recipe file matching "%s"' % pn) + msg = f'Unable to find any recipe file matching "{pn}"' + import difflib + providers = self.get_all_providers() + close_matches = difflib.get_close_matches(pn, providers, cutoff=0.7) + if close_matches: + close_matches = "\n ".join(close_matches) + msg += f'. Close matches:\n {close_matches}' + raise bb.providers.NoProvider(msg) return best[3] def get_file_appends(self, fn, mc=''): @@ -633,6 +778,29 @@ class Tinfoil: fn = self.get_recipe_file(pn) return self.parse_recipe_file(fn) + @contextmanager + def _data_tracked_if_enabled(self): + """ + A context manager to enable data tracking for a code segment if data + tracking was enabled for this tinfoil instance. + """ + if self.tracking: + # Enable history tracking just for the operation + self.run_command('enableDataTracking') + + # Here goes the operation with the optional data tracking + yield + + if self.tracking: + self.run_command('disableDataTracking') + + def finalizeData(self): + """ + Run anonymous functions and expand keys + """ + with self._data_tracked_if_enabled(): + return self._reconvert_type(self.run_command('finalizeData'), 'DataStoreConnectionHandle') + def parse_recipe_file(self, fn, appends=True, appendlist=None, config_data=None): """ Parse the specified recipe file (with or without bbappends) @@ -645,10 +813,7 @@ class Tinfoil: appendlist: optional list of bbappend files to apply, if you want to filter them """ - if self.tracking: - # Enable history tracking just for the parse operation - self.run_command('enableDataTracking') - try: + with self._data_tracked_if_enabled(): if appends and appendlist == []: appends = False if config_data: @@ -660,9 +825,6 @@ class Tinfoil: return self._reconvert_type(dscon, 'DataStoreConnectionHandle') else: return None - finally: - if self.tracking: - self.run_command('disableDataTracking') def build_file(self, buildfile, task, internal=True): """ @@ -674,6 +836,10 @@ class Tinfoil: """ return self.run_command('buildFile', buildfile, task, internal) + @wait_for + def build_file_sync(self, *args): + self.build_file(*args) + def build_targets(self, targets, task=None, handle_events=True, extra_events=None, event_callback=None): """ Builds the specified targets. This is equivalent to a normal invocation diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py index 8b212b78..4ee45d67 100644 --- a/bitbake/lib/bb/ui/buildinfohelper.py +++ b/bitbake/lib/bb/ui/buildinfohelper.py @@ -559,7 +559,10 @@ class ORMWrapper(object): # we might have an invalid link; no way to detect this. just set it to None filetarget_obj = None - parent_obj = Target_File.objects.get(target = target_obj, path = parent_path, inodetype = Target_File.ITYPE_DIRECTORY) + try: + parent_obj = Target_File.objects.get(target = target_obj, path = parent_path, inodetype = Target_File.ITYPE_DIRECTORY) + except Target_File.DoesNotExist: + parent_obj = None Target_File.objects.create( target = target_obj, diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py index f86999bb..82531ef8 100644 --- a/bitbake/lib/bb/ui/knotty.py +++ b/bitbake/lib/bb/ui/knotty.py @@ -10,6 +10,7 @@ from __future__ import division +import io import os import sys import logging @@ -24,6 +25,12 @@ import atexit from itertools import groupby from bb.ui import uihelper +import bb.build +import bb.command +import bb.cooker +import bb.event +import bb.runqueue +import bb.utils featureSet = [bb.cooker.CookerFeatures.SEND_SANITYEVENTS, bb.cooker.CookerFeatures.BASEDATASTORE_TRACKING] @@ -32,6 +39,7 @@ interactive = sys.stdout.isatty() class BBProgress(progressbar.ProgressBar): def __init__(self, msg, maxval, widgets=None, extrapos=-1, resize_handler=None): + self.id = msg self.msg = msg self.extrapos = extrapos if not widgets: @@ -77,7 +85,9 @@ class NonInteractiveProgress(object): fobj = sys.stdout def __init__(self, msg, maxval): + self.id = msg self.msg = msg + self.currval = 0 self.maxval = maxval self.finished = False @@ -87,6 +97,10 @@ class NonInteractiveProgress(object): return self def update(self, value): + self.currval = value + pass + + def clear(self): pass def finish(self): @@ -103,23 +117,42 @@ def new_progress(msg, maxval): return NonInteractiveProgress(msg, maxval) def pluralise(singular, plural, qty): - if(qty == 1): + if qty == 1: return singular % qty else: return plural % qty +def get_pressure_message(pressure_state, pressure_values): + cpu_pressure, io_pressure, mem_pressure = pressure_state + pressure_strs = [] + if cpu_pressure: + pressure_strs.append("CPU pressure (>%sus)" % pressure_values[1]) + if io_pressure: + pressure_strs.append("I/O pressure (>%sus)" % pressure_values[3]) + if mem_pressure: + pressure_strs.append("MEM pressure (>%sus)" % pressure_values[5]) + + if not pressure_strs: + pressure_strs.append("No pressure") + + return "%s is limiting task startup" % ", ".join(pressure_strs) class InteractConsoleLogFilter(logging.Filter): def __init__(self, tf): self.tf = tf + super().__init__() def filter(self, record): - if record.levelno == bb.msg.BBLogFormatter.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")): + if record.levelno == bb.msg.BBLogFormatter.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ") or "limiting task startup" in record.msg): return False self.tf.clearFooter() return True class TerminalFilter(object): + + # 40 Hz (FPS) -> 0.025 secs + _DEFAULT_PRINT_INTERVAL = 0.025 + rows = 25 columns = 80 @@ -131,7 +164,7 @@ class TerminalFilter(object): def getTerminalColumns(self): def ioctl_GWINSZ(fd): try: - cr = struct.unpack('hh', fcntl.ioctl(fd, self.termios.TIOCGWINSZ, '1234')) + cr = struct.unpack('hhhh', fcntl.ioctl(fd, self.termios.TIOCGWINSZ, b'12345678'))[0:2] except: return None return cr @@ -145,7 +178,7 @@ class TerminalFilter(object): pass if not cr: try: - cr = (os.environ['LINES'], os.environ['COLUMNS']) + cr = (int(os.environ['LINES']), int(os.environ['COLUMNS'])) except: cr = (25, 80) return cr @@ -157,10 +190,12 @@ class TerminalFilter(object): self.stdinbackup = None self.interactive = sys.stdout.isatty() self.footer_present = False - self.lastpids = [] - self.lasttime = None + self.lasttime = time.time() self.quiet = quiet + self._footer_buf = io.StringIO() + self._footer_lines = None + if not self.interactive: return @@ -208,11 +243,11 @@ class TerminalFilter(object): def clearFooter(self): if self.footer_present: - lines = self.footer_present - sys.stdout.buffer.write(self.curses.tparm(self.cuu, lines)) + sys.stdout.buffer.write(self.curses.tparm(self.cuu, self._footer_lines)) sys.stdout.buffer.write(self.curses.tparm(self.ed)) sys.stdout.flush() self.footer_present = False + self._footer_lines = None def elapsed(self, sec): hrs = int(sec / 3600.0) @@ -228,9 +263,10 @@ class TerminalFilter(object): def keepAlive(self, t): if not self.cuu: - print("Bitbake still alive (no events for %ds). Active tasks:" % t) + msgbuf = ["Bitbake still alive (no events for %ds). Active tasks:" % t] for t in self.helper.running_tasks: - print(t) + msgbuf.append(str(t)) + print("\n".join(msgbuf)) sys.stdout.flush() def updateFooter(self): @@ -238,29 +274,44 @@ class TerminalFilter(object): return activetasks = self.helper.running_tasks failedtasks = self.helper.failed_tasks - runningpids = self.helper.running_pids currenttime = time.time() - if not self.lasttime or (currenttime - self.lasttime > 5): + deltatime = currenttime - self.lasttime + + if (deltatime > 5.0): self.helper.needUpdate = True - self.lasttime = currenttime - if self.footer_present and not self.helper.needUpdate: + need_update = self.helper.needUpdate + else: + # Do not update faster than _DEFAULT_PRINT_INTERVAL + # to avoid heavy print() flooding. + need_update = self.helper.needUpdate and (deltatime > self._DEFAULT_PRINT_INTERVAL) + + if self.footer_present and not need_update: + # Footer update is not needed. return + + # Remember the time when the footer was last updated. + self.lasttime = currenttime + self.helper.needUpdate = False - if self.footer_present: - self.clearFooter() if (not self.helper.tasknumber_total or self.helper.tasknumber_current == self.helper.tasknumber_total) and not len(activetasks): + self.clearFooter() return + + # Clear footer buffer. + self._footer_buf.truncate(0) + self._footer_buf.seek(0) + tasks = [] - for t in runningpids: - start_time = activetasks[t].get("starttime", None) + for task in activetasks.values(): + start_time = task.get("starttime", None) if start_time: - msg = "%s - %s (pid %s)" % (activetasks[t]["title"], self.elapsed(currenttime - start_time), activetasks[t]["pid"]) + msg = "%s - %s (pid %s)" % (task["title"], self.elapsed(currenttime - start_time), task["pid"]) else: - msg = "%s (pid %s)" % (activetasks[t]["title"], activetasks[t]["pid"]) - progress = activetasks[t].get("progress", None) + msg = "%s (pid %s)" % (task["title"], task["pid"]) + progress = task.get("progress", None) if progress is not None: - pbar = activetasks[t].get("progressbar", None) - rate = activetasks[t].get("rate", None) + pbar = task.get("progressbar", None) + rate = task.get("rate", None) if not pbar or pbar.bouncing != (progress < 0): if progress < 0: pbar = BBProgress("0: %s" % msg, 100, widgets=[' ', progressbar.BouncingSlider(), ''], extrapos=3, resize_handler=self.sigwinch_handle) @@ -268,7 +319,8 @@ class TerminalFilter(object): else: pbar = BBProgress("0: %s" % msg, 100, widgets=[' ', progressbar.Percentage(), ' ', progressbar.Bar(), ''], extrapos=5, resize_handler=self.sigwinch_handle) pbar.bouncing = False - activetasks[t]["progressbar"] = pbar + pbar.fd = self._footer_buf + task["progressbar"] = pbar tasks.append((pbar, msg, progress, rate, start_time)) else: tasks.append(msg) @@ -278,7 +330,7 @@ class TerminalFilter(object): "Waiting for %s running tasks to finish", len(activetasks)) if not self.quiet: content += ':' - print(content) + print(content, file=self._footer_buf) else: scene_tasks = "%s of %s" % (self.helper.setscene_current, self.helper.setscene_total) cur_tasks = "%s of %s" % (self.helper.tasknumber_current, self.helper.tasknumber_total) @@ -287,7 +339,12 @@ class TerminalFilter(object): if not self.quiet: msg = "Setscene tasks: %s" % scene_tasks content += msg + "\n" - print(msg) + print(msg, file=self._footer_buf) + + if any(self.helper.pressure_state): + msg = get_pressure_message(self.helper.pressure_state, self.helper.pressure_values) + content += msg + "\n" + print(msg, file=self._footer_buf) if self.quiet: msg = "Running tasks (%s, %s)" % (scene_tasks, cur_tasks) @@ -299,11 +356,12 @@ class TerminalFilter(object): if not self.main_progress or self.main_progress.maxval != maxtask: widgets = [' ', progressbar.Percentage(), ' ', progressbar.Bar()] self.main_progress = BBProgress("Running tasks", maxtask, widgets=widgets, resize_handler=self.sigwinch_handle) + self.main_progress.fd = self._footer_buf self.main_progress.start(False) self.main_progress.setmessage(msg) progress = max(0, self.helper.tasknumber_current - 1) content += self.main_progress.update(progress) - print('') + print('', file=self._footer_buf) lines = self.getlines(content) if not self.quiet: for tasknum, task in enumerate(tasks[:(self.rows - 1 - lines)]): @@ -319,15 +377,19 @@ class TerminalFilter(object): content = pbar.update(progress) else: content = pbar.update(1) - print('') + print('', file=self._footer_buf) else: content = "%s: %s" % (tasknum, task) - print(content) + print(content, file=self._footer_buf) lines = lines + self.getlines(content) - self.footer_present = lines - self.lastpids = runningpids[:] self.lastcount = self.helper.tasknumber_current + # Clear footer and Print buffer. + self.clearFooter() + print(self._footer_buf.getvalue(), end='') + self._footer_lines = lines + self.footer_present = True + def getlines(self, content): lines = 0 for line in content.split("\n"): @@ -335,18 +397,18 @@ class TerminalFilter(object): return lines def finish(self): + self._footer_buf.close() if self.stdinbackup: fd = sys.stdin.fileno() self.termios.tcsetattr(fd, self.termios.TCSADRAIN, self.stdinbackup) def print_event_log(event, includelogs, loglines, termfilter): - # FIXME refactor this out further logfile = event.logfile if logfile and os.path.exists(logfile): termfilter.clearFooter() bb.error("Logfile of failure stored in: %s" % logfile) if includelogs and not event.errprinted: - print("Log data follows:") + msgbuf = ["Log data follows:"] f = open(logfile, "r") lines = [] while True: @@ -359,11 +421,11 @@ def print_event_log(event, includelogs, loglines, termfilter): if len(lines) > int(loglines): lines.pop(0) else: - print('| %s' % l) + msgbuf.append('| %s' % l) f.close() if lines: - for line in lines: - print(line) + msgbuf.extend(lines) + print("\n".join(msgbuf)) def _log_settings_from_server(server, observe_only): # Get values of variables which control our output @@ -397,7 +459,8 @@ _evt_list = [ "bb.runqueue.runQueueExitWait", "bb.event.LogExecTTY", "logging.Lo "bb.event.MultipleProviders", "bb.event.NoProvider", "bb.runqueue.sceneQueueTaskStarted", "bb.runqueue.runQueueTaskStarted", "bb.runqueue.runQueueTaskFailed", "bb.runqueue.sceneQueueTaskFailed", "bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent", - "bb.build.TaskProgress", "bb.event.ProcessStarted", "bb.event.ProcessProgress", "bb.event.ProcessFinished"] + "bb.build.TaskProgress", "bb.event.ProcessStarted", "bb.event.ProcessProgress", "bb.event.ProcessFinished", + "bb.runqueue.PSIEvent"] def drain_events_errorhandling(eventHandler): # We don't have logging setup, we do need to show any events we see before exiting @@ -551,17 +614,31 @@ def main(server, eventHandler, params, tf = TerminalFilter): "BitBake.RunQueue.HashEquiv": { "level": "VERBOSE", "handlers": ["BitBake.verbconsolelog"], + }, + "BitBake.RunQueue.PSI": { + "level": "VERBOSE", + "handlers": ["BitBake.verbconsolelog"], } } }) - bb.utils.mkdirhier(os.path.dirname(consolelogfile)) - loglink = os.path.join(os.path.dirname(consolelogfile), 'console-latest.log') + consolelogdirname = os.path.dirname(consolelogfile) + # `bb.utils.mkdirhier` has this check, but it reports failure using bb.fatal, which logs + # to the very logger we are trying to set up. + if '${' in str(consolelogdirname): + print( + "FATAL: Directory name {} contains unexpanded bitbake variable. This may cause build failures and WORKDIR pollution.".format( + consolelogdirname)) + if '${MACHINE}' in consolelogdirname: + print("HINT: It looks like you forgot to set MACHINE in local.conf.") + + bb.utils.mkdirhier(consolelogdirname) + loglink = os.path.join(consolelogdirname, 'console-latest.log') bb.utils.remove(loglink) try: - os.symlink(os.path.basename(consolelogfile), loglink) + os.symlink(os.path.basename(consolelogfile), loglink) except OSError: - pass + pass # Add the logging domains specified by the user on the command line for (domainarg, iterator) in groupby(params.debug_domains): @@ -577,6 +654,8 @@ def main(server, eventHandler, params, tf = TerminalFilter): else: log_exec_tty = False + should_print_hyperlinks = sys.stdout.isatty() and os.environ.get('NO_COLOR', '') == '' + helper = uihelper.BBUIHelper() # Look for the specially designated handlers which need to be passed to the @@ -640,7 +719,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): return_value = 0 errors = 0 warnings = 0 - taskfailures = [] + taskfailures = {} printintervaldelta = 10 * 60 # 10 minutes printinterval = printintervaldelta @@ -717,7 +796,12 @@ def main(server, eventHandler, params, tf = TerminalFilter): event.msg = taskinfo['title'] + ': ' + event.msg if hasattr(event, 'fn') and event.levelno not in [bb.msg.BBLogFormatter.WARNONCE, bb.msg.BBLogFormatter.ERRORONCE]: event.msg = event.fn + ': ' + event.msg + # Need to remove any progress bar, then add it back after we print this message + if parseprogress: + parseprogress.clear() logging.getLogger(event.name).handle(event) + if parseprogress: + parseprogress.update(parseprogress.currval) continue if isinstance(event, bb.build.TaskFailedSilent): @@ -726,6 +810,8 @@ def main(server, eventHandler, params, tf = TerminalFilter): if isinstance(event, bb.build.TaskFailed): return_value = 1 print_event_log(event, includelogs, loglines, termfilter) + k = "{}:{}".format(event._fn, event._task) + taskfailures[k] = event.logfile if isinstance(event, bb.build.TaskBase): logger.info(event._message) continue @@ -821,7 +907,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): if isinstance(event, bb.runqueue.runQueueTaskFailed): return_value = 1 - taskfailures.append(event.taskstring) + taskfailures.setdefault(event.taskstring) logger.error(str(event)) continue @@ -836,23 +922,28 @@ def main(server, eventHandler, params, tf = TerminalFilter): if params.options.quiet > 1: continue termfilter.clearFooter() + if parseprogress: + parseprogress.finish() parseprogress = new_progress(event.processname, event.total) parseprogress.start(False) continue if isinstance(event, bb.event.ProcessProgress): if params.options.quiet > 1: continue - if parseprogress: + if parseprogress and parseprogress.id == event.processname: parseprogress.update(event.progress) - else: - bb.warn("Got ProcessProgress event for someting that never started?") continue if isinstance(event, bb.event.ProcessFinished): if params.options.quiet > 1: continue - if parseprogress: + if parseprogress and parseprogress.id == event.processname: parseprogress.finish() - parseprogress = None + parseprogress = None + continue + if isinstance(event, bb.runqueue.PSIEvent): + if params.options.quiet > 1: + continue + logger.info(get_pressure_message(event.pressure_state, event.pressure_values)) continue # ignore @@ -942,11 +1033,21 @@ def main(server, eventHandler, params, tf = TerminalFilter): try: termfilter.clearFooter() summary = "" + def format_hyperlink(url, link_text): + if should_print_hyperlinks: + start = f'\033]8;;{url}\033\\' + end = '\033]8;;\033\\' + return f'{start}{link_text}{end}' + return link_text + if taskfailures: summary += pluralise("\nSummary: %s task failed:", "\nSummary: %s tasks failed:", len(taskfailures)) - for failure in taskfailures: + for (failure, log_file) in taskfailures.items(): summary += "\n %s" % failure + if log_file: + hyperlink = format_hyperlink(f"file://{log_file}", log_file) + summary += "\n log: {}".format(hyperlink) if warnings: summary += pluralise("\nSummary: There was %s WARNING message.", "\nSummary: There were %s WARNING messages.", warnings) diff --git a/bitbake/lib/bb/ui/taskexp_ncurses.py b/bitbake/lib/bb/ui/taskexp_ncurses.py index ea94a498..551ad142 100755 --- a/bitbake/lib/bb/ui/taskexp_ncurses.py +++ b/bitbake/lib/bb/ui/taskexp_ncurses.py @@ -146,6 +146,9 @@ def debug_frame(nbox_ojb): unit_test = os.environ.get('TASK_EXP_UNIT_TEST') unit_test_cmnds=[ '# Default selected task in primary box', + 'tst_selected=.do_create_recipe_spdx', + '# Move to next entry, more predictable', + 'tst_entry=', # optional injected error 'tst_selected=.do_recipe_qa', '# Default selected task in deps', 'tst_entry=', diff --git a/bitbake/lib/bb/ui/teamcity.py b/bitbake/lib/bb/ui/teamcity.py index 7eeaab8d..3177f4aa 100644 --- a/bitbake/lib/bb/ui/teamcity.py +++ b/bitbake/lib/bb/ui/teamcity.py @@ -200,6 +200,7 @@ def main(server, eventHandler, params): logger.error("XMLRPC Fault getting commandline: {0}".format(x)) return 1 + active_process = None active_process_total = None is_tasks_running = False @@ -300,16 +301,23 @@ def main(server, eventHandler, params): if isinstance(event, bb.event.ProcessStarted): if event.processname in ["Initialising tasks", "Checking sstate mirror object availability"]: + if active_process: + ui.progress(active_process, 100) + ui.block_end() + active_process = event.processname active_process_total = event.total ui.block_start(event.processname) if isinstance(event, bb.event.ProcessFinished): if event.processname in ["Initialising tasks", "Checking sstate mirror object availability"]: - ui.progress(event.processname, 100) - ui.block_end() + if active_process and active_process == event.processname: + ui.progress(event.processname, 100) + ui.block_end() + active_process = None if isinstance(event, bb.event.ProcessProgress): if event.processname in ["Initialising tasks", "Checking sstate mirror object availability"] and active_process_total != 0: - ui.progress(event.processname, event.progress * 100 / active_process_total) + if active_process and active_process == event.processname: + ui.progress(event.processname, event.progress * 100 / active_process_total) if isinstance(event, bb.event.CacheLoadStarted): ui.block_start("Loading cache") if isinstance(event, bb.event.CacheLoadProgress): diff --git a/bitbake/lib/bb/ui/uihelper.py b/bitbake/lib/bb/ui/uihelper.py index 82913e0d..3e077f04 100644 --- a/bitbake/lib/bb/ui/uihelper.py +++ b/bitbake/lib/bb/ui/uihelper.py @@ -6,6 +6,7 @@ # import bb.build +import bb.runqueue import time class BBUIHelper: @@ -13,17 +14,17 @@ class BBUIHelper: self.needUpdate = False self.running_tasks = {} # Running PIDs preserves the order tasks were executed in - self.running_pids = [] self.failed_tasks = [] self.pidmap = {} self.tasknumber_current = 0 self.tasknumber_total = 0 + self.pressure_state = (False, False, False) + self.pressure_values = None def eventHandler(self, event): # PIDs are a bad idea as they can be reused before we process all UI events. # We maintain a 'fuzzy' match for TaskProgress since there is no other way to match def removetid(pid, tid): - self.running_pids.remove(tid) del self.running_tasks[tid] if self.pidmap[pid] == tid: del self.pidmap[pid] @@ -31,11 +32,10 @@ class BBUIHelper: if isinstance(event, bb.build.TaskStarted): tid = event._fn + ":" + event._task - if event._mc != "default": + if event._mc != "": self.running_tasks[tid] = { 'title' : "mc:%s:%s %s" % (event._mc, event._package, event._task), 'starttime' : time.time(), 'pid' : event.pid } else: self.running_tasks[tid] = { 'title' : "%s %s" % (event._package, event._task), 'starttime' : time.time(), 'pid' : event.pid } - self.running_pids.append(tid) self.pidmap[event.pid] = tid self.needUpdate = True elif isinstance(event, bb.build.TaskSucceeded): @@ -60,6 +60,10 @@ class BBUIHelper: self.running_tasks[self.pidmap[event.pid]]['progress'] = event.progress self.running_tasks[self.pidmap[event.pid]]['rate'] = event.rate self.needUpdate = True + elif isinstance(event, bb.runqueue.PSIEvent): + self.pressure_state = event.pressure_state + self.pressure_values = event.pressure_values + self.needUpdate = True else: return False return True diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py index ebee65d3..a91bebf5 100644 --- a/bitbake/lib/bb/utils.py +++ b/bitbake/lib/bb/utils.py @@ -11,11 +11,8 @@ import re, fcntl, os, string, stat, shutil, time import sys import errno import logging -import bb -import bb.msg import locale -import multiprocessing -import fcntl +from bb import multiprocessing import importlib import importlib.machinery import importlib.util @@ -24,7 +21,6 @@ import subprocess import glob import fnmatch import traceback -import errno import signal import collections import copy @@ -36,6 +32,9 @@ import tempfile from subprocess import getstatusoutput from contextlib import contextmanager from ctypes import cdll +import bb +import bb.msg +import bb.filter logger = logging.getLogger("BitBake.Util") python_extensions = importlib.machinery.all_suffixes() @@ -84,7 +83,16 @@ def explode_version(s): return r def split_version(s): - """Split a version string into its constituent parts (PE, PV, PR)""" + """Split a version string into its constituent parts (PE, PV, PR). + + Arguments: + + - ``s``: version string. The format of the input string should be:: + + ${PE}:${PV}-${PR} + + Returns a tuple ``(pe, pv, pr)``. + """ s = s.strip(" <>=") e = 0 if s.count(':'): @@ -136,16 +144,30 @@ def vercmp(ta, tb): return r def vercmp_string(a, b): - """ Split version strings and compare them """ + """ Split version strings using ``bb.utils.split_version()`` and compare + them with ``bb.utils.vercmp().`` + + Arguments: + + - ``a``: left version string operand. + - ``b``: right version string operand. + + Returns what ``bb.utils.vercmp()`` returns.""" ta = split_version(a) tb = split_version(b) return vercmp(ta, tb) def vercmp_string_op(a, b, op): """ - Compare two versions and check if the specified comparison operator matches the result of the comparison. - This function is fairly liberal about what operators it will accept since there are a variety of styles - depending on the context. + Takes the return value ``bb.utils.vercmp()`` and returns the operation + defined by ``op`` between the return value and 0. + + Arguments: + + - ``a``: left version string operand. + - ``b``: right version string operand. + - ``op``: operator string. Can be one of ``=``, ``==``, ``<=``, ``>=``, + ``>``, ``>>``, ``<``, ``<<`` or ``!=``. """ res = vercmp_string(a, b) if op in ('=', '=='): @@ -163,11 +185,19 @@ def vercmp_string_op(a, b, op): else: raise VersionStringException('Unsupported comparison operator "%s"' % op) +@bb.filter.filter_proc(name="bb.utils.explode_deps") def explode_deps(s): """ - Take an RDEPENDS style string of format: - "DEPEND1 (optional version) DEPEND2 (optional version) ..." - and return a list of dependencies. + Takes an RDEPENDS style string of format:: + + DEPEND1 (optional version) DEPEND2 (optional version) ... + + Arguments: + + - ``s``: input RDEPENDS style string + + Returns a list of dependencies. + Version information is ignored. """ r = [] @@ -189,9 +219,17 @@ def explode_deps(s): def explode_dep_versions2(s, *, sort=True): """ - Take an RDEPENDS style string of format: - "DEPEND1 (optional version) DEPEND2 (optional version) ..." - and return a dictionary of dependencies and versions. + Takes an RDEPENDS style string of format:: + + DEPEND1 (optional version) DEPEND2 (optional version) ... + + Arguments: + + - ``s``: input RDEPENDS style string + - ``*``: *Unused*. + - ``sort``: whether to sort the output or not. + + Returns a dictionary of dependencies and versions. """ r = collections.OrderedDict() l = s.replace(",", "").split() @@ -256,10 +294,17 @@ def explode_dep_versions2(s, *, sort=True): def explode_dep_versions(s): """ - Take an RDEPENDS style string of format: - "DEPEND1 (optional version) DEPEND2 (optional version) ..." - skip null value and items appeared in dependency string multiple times - and return a dictionary of dependencies and versions. + Take an RDEPENDS style string of format:: + + DEPEND1 (optional version) DEPEND2 (optional version) ... + + Skips null values and items appeared in dependency string multiple times. + + Arguments: + + - ``s``: input RDEPENDS style string + + Returns a dictionary of dependencies and versions. """ r = explode_dep_versions2(s) for d in r: @@ -273,7 +318,17 @@ def explode_dep_versions(s): def join_deps(deps, commasep=True): """ - Take the result from explode_dep_versions and generate a dependency string + Take a result from ``bb.utils.explode_dep_versions()`` and generate a + dependency string. + + Arguments: + + - ``deps``: dictionary of dependencies and versions. + - ``commasep``: makes the return value separated by commas if ``True``, + separated by spaces otherwise. + + Returns a comma-separated (space-separated if ``comma-sep`` is ``False``) + string of dependencies and versions. """ result = [] for dep in deps: @@ -435,7 +490,11 @@ def better_eval(source, locals, extraglobals = None): @contextmanager def fileslocked(files, *args, **kwargs): - """Context manager for locking and unlocking file locks.""" + """Context manager for locking and unlocking file locks. Uses + ``bb.utils.lockfile()`` and ``bb.utils.unlockfile()`` to lock and unlock + files. + + No return value.""" locks = [] if files: for lockfile in files: @@ -446,19 +505,29 @@ def fileslocked(files, *args, **kwargs): try: yield finally: + locks.reverse() for lock in locks: bb.utils.unlockfile(lock) def lockfile(name, shared=False, retry=True, block=False): """ - Use the specified file as a lock file, return when the lock has - been acquired. Returns a variable to pass to unlockfile(). - Parameters: - retry: True to re-try locking if it fails, False otherwise - block: True to block until the lock succeeds, False otherwise + Use the specified file (with filename ``name``) as a lock file, return when + the lock has been acquired. Returns a variable to pass to unlockfile(). + + Arguments: + + - ``shared``: sets the lock as a shared lock instead of an + exclusive lock. + - ``retry``: ``True`` to re-try locking if it fails, ``False`` + otherwise. + - ``block``: ``True`` to block until the lock succeeds, + ``False`` otherwise. + The retry and block parameters are kind of equivalent unless you consider the possibility of sending a signal to the process to break out - at which point you want block=True rather than retry=True. + + Returns the locked file descriptor in case of success, ``None`` otherwise. """ basename = os.path.basename(name) if len(basename) > 255: @@ -517,7 +586,13 @@ def lockfile(name, shared=False, retry=True, block=False): def unlockfile(lf): """ - Unlock a file locked using lockfile() + Unlock a file locked using ``bb.utils.lockfile()``. + + Arguments: + + - ``lf``: the locked file descriptor. + + No return value. """ try: # If we had a shared lock, we need to promote to exclusive before @@ -545,7 +620,11 @@ def _hasher(method, filename): def md5_file(filename): """ - Return the hex string representation of the MD5 checksum of filename. + Arguments: + + - ``filename``: path to the input file. + + Returns the hexadecimal string representation of the MD5 checksum of filename. """ import hashlib try: @@ -557,36 +636,86 @@ def md5_file(filename): def sha256_file(filename): """ - Return the hex string representation of the 256-bit SHA checksum of + Returns the hexadecimal representation of the 256-bit SHA checksum of filename. + + Arguments: + + - ``filename``: path to the file. """ import hashlib return _hasher(hashlib.sha256(), filename) def sha1_file(filename): """ - Return the hex string representation of the SHA1 checksum of the filename + Returns the hexadecimal representation of the SHA1 checksum of the filename + + Arguments: + + - ``filename``: path to the file. """ import hashlib return _hasher(hashlib.sha1(), filename) def sha384_file(filename): """ - Return the hex string representation of the SHA384 checksum of the filename + Returns the hexadecimal representation of the SHA384 checksum of the filename + + Arguments: + + - ``filename``: path to the file. """ import hashlib return _hasher(hashlib.sha384(), filename) def sha512_file(filename): """ - Return the hex string representation of the SHA512 checksum of the filename + Returns the hexadecimal representation of the SHA512 checksum of the filename + + Arguments: + + - ``filename``: path to the file. """ import hashlib return _hasher(hashlib.sha512(), filename) +def goh1_file(filename): + """ + Returns the hexadecimal string representation of the Go mod h1 checksum of the + filename. The Go mod h1 checksum uses the Go dirhash package. The package + defines hashes over directory trees and is used by go mod for mod files and + zip archives. + + Arguments: + + - ``filename``: path to the file. + """ + import hashlib + import zipfile + + lines = [] + is_zipfile = False + if zipfile.is_zipfile(filename): + try: + with zipfile.ZipFile(filename) as archive: + for fn in sorted(archive.namelist()): + method = hashlib.sha256() + method.update(archive.read(fn)) + hash = method.hexdigest() + lines.append("%s %s\n" % (hash, fn)) + is_zipfile = True + except zipfile.BadZipFile: + is_zipfile = False + if not is_zipfile: + hash = _hasher(hashlib.sha256(), filename) + lines.append("%s go.mod\n" % hash) + method = hashlib.sha256() + method.update("".join(lines).encode('utf-8')) + return method.hexdigest() + def preserved_envvars_exported(): - """Variables which are taken from the environment and placed in and exported - from the metadata""" + """Returns the list of variables which are taken from the environment and + placed in and exported from the metadata.""" return [ 'BB_TASKHASH', 'HOME', @@ -600,7 +729,8 @@ def preserved_envvars_exported(): ] def preserved_envvars(): - """Variables which are taken from the environment and placed in the metadata""" + """Returns the list of variables which are taken from the environment and + placed in the metadata.""" v = [ 'BBPATH', 'BB_PRESERVE_ENV', @@ -609,7 +739,9 @@ def preserved_envvars(): return v + preserved_envvars_exported() def check_system_locale(): - """Make sure the required system locale are available and configured""" + """Make sure the required system locale are available and configured. + + No return value.""" default_locale = locale.getlocale(locale.LC_CTYPE) try: @@ -627,6 +759,12 @@ def filter_environment(good_vars): """ Create a pristine environment for bitbake. This will remove variables that are not known and may influence the build in a negative way. + + Arguments: + + - ``good_vars``: list of variable to exclude from the filtering. + + No return value. """ removed_vars = {} @@ -671,6 +809,8 @@ def clean_environment(): """ Clean up any spurious environment variables. This will remove any variables the user hasn't chosen to preserve. + + No return value. """ if 'BB_PRESERVE_ENV' not in os.environ: good_vars = approved_variables() @@ -681,6 +821,8 @@ def clean_environment(): def empty_environment(): """ Remove all variables from the environment. + + No return value. """ for s in list(os.environ.keys()): os.unsetenv(s) @@ -689,6 +831,12 @@ def empty_environment(): def build_environment(d): """ Build an environment from all exported variables. + + Arguments: + + - ``d``: the data store. + + No return value. """ import bb.data for var in bb.data.keys(d): @@ -713,7 +861,17 @@ def _check_unsafe_delete_path(path): return False def remove(path, recurse=False, ionice=False): - """Equivalent to rm -f or rm -rf""" + """Equivalent to rm -f or rm -rf. + + Arguments: + + - ``path``: path to file/directory to remove. + - ``recurse``: deletes recursively if ``True``. + - ``ionice``: prepends ``ionice -c 3`` to the ``rm`` command. See ``man + ionice``. + + No return value. + """ if not path: return if recurse: @@ -734,7 +892,17 @@ def remove(path, recurse=False, ionice=False): raise def prunedir(topdir, ionice=False): - """ Delete everything reachable from the directory named in 'topdir'. """ + """ + Delete everything reachable from the directory named in ``topdir``. + + Arguments: + + - ``topdir``: directory path. + - ``ionice``: prepends ``ionice -c 3`` to the ``rm`` command. See ``man + ionice``. + + No return value. + """ # CAUTION: This is dangerous! if _check_unsafe_delete_path(topdir): raise Exception('bb.utils.prunedir: called with dangerous path "%s", refusing to delete!' % topdir) @@ -746,8 +914,15 @@ def prunedir(topdir, ionice=False): # def prune_suffix(var, suffixes, d): """ - See if var ends with any of the suffixes listed and - remove it if found + Check if ``var`` ends with any of the suffixes listed in ``suffixes`` and + remove it if found. + + Arguments: + + - ``var``: string to check for suffixes. + - ``suffixes``: list of strings representing suffixes to check for. + + Returns the string ``var`` without the suffix. """ for suffix in suffixes: if suffix and var.endswith(suffix): @@ -756,21 +931,53 @@ def prune_suffix(var, suffixes, d): def mkdirhier(directory): """Create a directory like 'mkdir -p', but does not complain if - directory already exists like os.makedirs + directory already exists like ``os.makedirs()``. + + Arguments: + + - ``directory``: path to the directory. + + No return value. """ if '${' in str(directory): bb.fatal("Directory name {} contains unexpanded bitbake variable. This may cause build failures and WORKDIR polution.".format(directory)) try: os.makedirs(directory) except OSError as e: - if e.errno != errno.EEXIST or not os.path.isdir(directory): + if e.errno != errno.EEXIST: raise e + if os.path.isdir(directory): + return + # We can end up here if there is a race between two mkdirs on an NFS mount, + # which happens more often with sstate that you'd think. The server returns + # EEXIST but the local attribute cache is out of date. It can be refreshed with + # an opendir call, so try that (via listdir) and check the directory again + # before we really fail. + os.listdir(os.path.dirname(directory)) + if os.path.isdir(directory): + return + bb.warn("mkdir: %s is not a directory?") + raise e def movefile(src, dest, newmtime = None, sstat = None): - """Moves a file from src to dest, preserving all permissions and + """Moves a file from ``src`` to ``dest``, preserving all permissions and attributes; mtime will be preserved even when moving across - filesystems. Returns true on success and false on failure. Move is + filesystems. Returns ``True`` on success and ``False`` on failure. Move is atomic. + + Arguments: + + - ``src`` -- Source file. + - ``dest`` -- Destination file. + - ``newmtime`` -- new mtime to be passed as float seconds since the epoch. + - ``sstat`` -- os.stat_result to use for the destination file. + + Returns an ``os.stat_result`` of the destination file if the + source file is a symbolic link or the ``sstat`` argument represents a + symbolic link - in which case the destination file will also be created as + a symbolic link. + + Otherwise, returns ``newmtime`` on success and ``False`` on failure. """ #print "movefile(" + src + "," + dest + "," + str(newmtime) + "," + str(sstat) + ")" @@ -861,9 +1068,24 @@ def movefile(src, dest, newmtime = None, sstat = None): def copyfile(src, dest, newmtime = None, sstat = None): """ - Copies a file from src to dest, preserving all permissions and + Copies a file from ``src`` to ``dest``, preserving all permissions and attributes; mtime will be preserved even when moving across - filesystems. Returns true on success and false on failure. + filesystems. + + Arguments: + + - ``src``: Source file. + - ``dest``: Destination file. + - ``newmtime``: new mtime to be passed as float seconds since the epoch. + - ``sstat``: os.stat_result to use for the destination file. + + Returns an ``os.stat_result`` of the destination file if the + source file is a symbolic link or the ``sstat`` argument represents a + symbolic link - in which case the destination file will also be created as + a symbolic link. + + Otherwise, returns ``newmtime`` on success and ``False`` on failure. + """ #print "copyfile(" + src + "," + dest + "," + str(newmtime) + "," + str(sstat) + ")" try: @@ -941,10 +1163,16 @@ def copyfile(src, dest, newmtime = None, sstat = None): def break_hardlinks(src, sstat = None): """ - Ensures src is the only hardlink to this file. Other hardlinks, + Ensures ``src`` is the only hardlink to this file. Other hardlinks, if any, are not affected (other than in their st_nlink value, of - course). Returns true on success and false on failure. + course). + + Arguments: + - ``src``: source file path. + - ``sstat``: os.stat_result to use when checking if the file is a link. + + Returns ``True`` on success and ``False`` on failure. """ try: if not sstat: @@ -958,11 +1186,24 @@ def break_hardlinks(src, sstat = None): def which(path, item, direction = 0, history = False, executable=False): """ - Locate `item` in the list of paths `path` (colon separated string like $PATH). - If `direction` is non-zero then the list is reversed. - If `history` is True then the list of candidates also returned as result,history. - If `executable` is True then the candidate has to be an executable file, - otherwise the candidate simply has to exist. + Locate ``item`` in the list of paths ``path`` (colon separated string like + ``$PATH``). + + Arguments: + + - ``path``: list of colon-separated paths. + - ``item``: string to search for. + - ``direction``: if non-zero then the list is reversed. + - ``history``: if ``True`` then the list of candidates also returned as + ``result,history`` where ``history`` is the list of previous path + checked. + - ``executable``: if ``True`` then the candidate defined by ``path`` has + to be an executable file, otherwise if ``False`` the candidate simply + has to exist. + + Returns the item if found in the list of path, otherwise an empty string. + If ``history`` is ``True``, return the list of previous path checked in a + tuple with the found (or not found) item as ``(item, history)``. """ if executable: @@ -989,10 +1230,29 @@ def which(path, item, direction = 0, history = False, executable=False): return "", hist return "" +def to_filemode(input): + """ + Take a bitbake variable contents defining a file mode and return + the proper python representation of the number + + Arguments: + + - ``input``: a string or number to convert, e.g. a bitbake variable + string, assumed to be an octal representation + + Returns the python file mode as a number + """ + # umask might come in as a number or text string.. + if type(input) is int: + return input + return int(input, 8) + @contextmanager def umask(new_mask): """ Context manager to set the umask to a specific mask, and restore it afterwards. + + No return value. """ current_mask = os.umask(new_mask) try: @@ -1003,7 +1263,17 @@ def umask(new_mask): def to_boolean(string, default=None): """ Check input string and return boolean value True/False/None - depending upon the checks + depending upon the checks. + + Arguments: + + - ``string``: input string. + - ``default``: default return value if the input ``string`` is ``None``, + ``0``, ``False`` or an empty string. + + Returns ``True`` if the string is one of "y", "yes", "1", "true", ``False`` + if the string is one of "n", "no", "0", or "false". Return ``default`` if + the input ``string`` is ``None``, ``0``, ``False`` or an empty string. """ if not string: return default @@ -1024,18 +1294,17 @@ def contains(variable, checkvalues, truevalue, falsevalue, d): Arguments: - variable -- the variable name. This will be fetched and expanded (using - d.getVar(variable)) and then split into a set(). - - checkvalues -- if this is a string it is split on whitespace into a set(), - otherwise coerced directly into a set(). - - truevalue -- the value to return if checkvalues is a subset of variable. + - ``variable``: the variable name. This will be fetched and expanded (using + d.getVar(variable)) and then split into a set(). + - ``checkvalues``: if this is a string it is split on whitespace into a set(), + otherwise coerced directly into a set(). + - ``truevalue``: the value to return if checkvalues is a subset of variable. + - ``falsevalue``: the value to return if variable is empty or if checkvalues is + not a subset of variable. + - ``d``: the data store. - falsevalue -- the value to return if variable is empty or if checkvalues is - not a subset of variable. - - d -- the data store. + Returns ``True`` if the variable contains the values specified, ``False`` + otherwise. """ val = d.getVar(variable) @@ -1055,18 +1324,17 @@ def contains_any(variable, checkvalues, truevalue, falsevalue, d): Arguments: - variable -- the variable name. This will be fetched and expanded (using - d.getVar(variable)) and then split into a set(). - - checkvalues -- if this is a string it is split on whitespace into a set(), - otherwise coerced directly into a set(). - - truevalue -- the value to return if checkvalues is a subset of variable. - - falsevalue -- the value to return if variable is empty or if checkvalues is - not a subset of variable. + - ``variable``: the variable name. This will be fetched and expanded (using + d.getVar(variable)) and then split into a set(). + - ``checkvalues``: if this is a string it is split on whitespace into a set(), + otherwise coerced directly into a set(). + - ``truevalue``: the value to return if checkvalues is a subset of variable. + - ``falsevalue``: the value to return if variable is empty or if checkvalues is + not a subset of variable. + - ``d``: the data store. - d -- the data store. + Returns ``True`` if the variable contains any of the values specified, + ``False`` otherwise. """ val = d.getVar(variable) if not val: @@ -1080,21 +1348,18 @@ def contains_any(variable, checkvalues, truevalue, falsevalue, d): return truevalue return falsevalue -def filter(variable, checkvalues, d): - """Return all words in the variable that are present in the checkvalues. +def filter_string(val, checkvalues): + """Return all words in the string that are present in the ``checkvalues``. Arguments: - variable -- the variable name. This will be fetched and expanded (using - d.getVar(variable)) and then split into a set(). - - checkvalues -- if this is a string it is split on whitespace into a set(), - otherwise coerced directly into a set(). + - ``val``: the string data to filter after being split into a set(). + - ``checkvalues``: if this is a string it is split on whitespace into a set(), + otherwise coerced directly into a set(). + - ``d``: the data store. - d -- the data store. + Returns a list of string. """ - - val = d.getVar(variable) if not val: return '' val = set(val.split()) @@ -1104,11 +1369,46 @@ def filter(variable, checkvalues, d): checkvalues = set(checkvalues) return ' '.join(sorted(checkvalues & val)) +def filter(variable, checkvalues, d): + """Return all words in the variable that are present in the ``checkvalues``. + + Arguments: + + - ``variable``: the variable name. This will be fetched and expanded (using + d.getVar(variable)) and then split into a set(). + - ``checkvalues``: if this is a string it is split on whitespace into a set(), + otherwise coerced directly into a set(). + - ``d``: the data store. + + Returns a list of string. + """ + + val = d.getVar(variable) + return filter_string(val, checkvalues) def get_referenced_vars(start_expr, d): """ - :return: names of vars referenced in start_expr (recursively), in quasi-BFS order (variables within the same level - are ordered arbitrarily) + Get the names of the variables referenced in a given expression. + + Arguments: + + - ``start_expr``: the expression where to look for variables references. + + For example:: + + ${VAR_A} string ${VAR_B} + + Or:: + + ${@d.getVar('VAR')} + + If a variables makes references to other variables, the latter are also + returned recursively. + + - ``d``: the data store. + + Returns the names of vars referenced in ``start_expr`` (recursively), in + quasi-BFS order (variables within the same level are ordered arbitrarily). """ seen = set() @@ -1150,45 +1450,71 @@ def cpu_count(): def nonblockingfd(fd): fcntl.fcntl(fd, fcntl.F_SETFL, fcntl.fcntl(fd, fcntl.F_GETFL) | os.O_NONBLOCK) -def process_profilelog(fn, pout = None): +def profile_function(profile, function, output_fn, process=True): + """Common function to profile a code block and optionally process the + output using or processing function. + + Arguments: + + - ``profile``: a boolean saying whether to enable profiling or not + - ``function``: the function call to profile/run + - ``outputfn``: where to write the profiling data + - ``process``: whether to process the profiling data and write a report + + Returns the wrapped function return value + """ + if profile: + try: + import cProfile as profile + except: + import profile + prof = profile.Profile() + ret = profile.Profile.runcall(prof, function) + prof.dump_stats(output_fn) + if process: + process_profilelog(output_fn) + serverlog("Raw profiling information saved to %s and processed statistics to %s.report*" % (output_fn, output_fn)) + return ret + else: + return function() + +def process_profilelog(fn, fn_out = None): # Either call with a list of filenames and set pout or a filename and optionally pout. - if not pout: - pout = fn + '.processed' + import pstats + + if not fn_out: + fn_out = fn + '.report' - with open(pout, 'w') as pout: - import pstats + def pstatopen(): if isinstance(fn, list): - p = pstats.Stats(*fn, stream=pout) - else: - p = pstats.Stats(fn, stream=pout) + return pstats.Stats(*fn, stream=pout) + return pstats.Stats(fn, stream=pout) + + with open(fn_out + '.time', 'w') as pout: + p = pstatopen() p.sort_stats('time') p.print_stats() + + with open(fn_out + '.time-callers', 'w') as pout: + p = pstatopen() + p.sort_stats('time') p.print_callers() + + with open(fn_out + '.cumulative', 'w') as pout: + p = pstatopen() p.sort_stats('cumulative') p.print_stats() - pout.flush() - -# -# Was present to work around multiprocessing pool bugs in python < 2.7.3 -# -def multiprocessingpool(*args, **kwargs): - - import multiprocessing.pool - #import multiprocessing.util - #multiprocessing.util.log_to_stderr(10) - # Deal with a multiprocessing bug where signals to the processes would be delayed until the work - # completes. Putting in a timeout means the signals (like SIGINT/SIGTERM) get processed. - def wrapper(func): - def wrap(self, timeout=None): - return func(self, timeout=timeout if timeout is not None else 1e100) - return wrap - multiprocessing.pool.IMapIterator.next = wrapper(multiprocessing.pool.IMapIterator.next) + with open(fn_out + '.cumulative-callers', 'w') as pout: + p = pstatopen() + p.sort_stats('cumulative') + p.print_callers() - return multiprocessing.Pool(*args, **kwargs) def exec_flat_python_func(func, *args, **kwargs): - """Execute a flat python function (defined with def funcname(args):...)""" + """Execute a flat python function (defined with ``def funcname(args): ...``) + + Returns the return value of the function.""" # Prepare a small piece of python code which calls the requested function # To do this we need to prepare two things - a set of variables we can use to pass # the values of arguments into the calling function, and the list of arguments for @@ -1214,48 +1540,57 @@ def edit_metadata(meta_lines, variables, varfunc, match_overrides=False): """Edit lines from a recipe or config file and modify one or more specified variable values set in the file using a specified callback function. Lines are expected to have trailing newlines. - Parameters: - meta_lines: lines from the file; can be a list or an iterable - (e.g. file pointer) - variables: a list of variable names to look for. Functions - may also be specified, but must be specified with '()' at - the end of the name. Note that the function doesn't have - any intrinsic understanding of :append, :prepend, :remove, - or overrides, so these are considered as part of the name. - These values go into a regular expression, so regular - expression syntax is allowed. - varfunc: callback function called for every variable matching - one of the entries in the variables parameter. The function - should take four arguments: - varname: name of variable matched - origvalue: current value in file - op: the operator (e.g. '+=') - newlines: list of lines up to this point. You can use - this to prepend lines before this variable setting - if you wish. - and should return a four-element tuple: - newvalue: new value to substitute in, or None to drop - the variable setting entirely. (If the removal - results in two consecutive blank lines, one of the - blank lines will also be dropped). - newop: the operator to use - if you specify None here, - the original operation will be used. - indent: number of spaces to indent multi-line entries, - or -1 to indent up to the level of the assignment - and opening quote, or a string to use as the indent. - minbreak: True to allow the first element of a - multi-line value to continue on the same line as - the assignment, False to indent before the first - element. - To clarify, if you wish not to change the value, then you - would return like this: return origvalue, None, 0, True - match_overrides: True to match items with _overrides on the end, - False otherwise + + Arguments: + + - ``meta_lines``: lines from the file; can be a list or an iterable + (e.g. file pointer) + - ``variables``: a list of variable names to look for. Functions + may also be specified, but must be specified with ``()`` at + the end of the name. Note that the function doesn't have + any intrinsic understanding of ``:append``, ``:prepend``, ``:remove``, + or overrides, so these are considered as part of the name. + These values go into a regular expression, so regular + expression syntax is allowed. + - ``varfunc``: callback function called for every variable matching + one of the entries in the variables parameter. + + The function should take four arguments: + + - ``varname``: name of variable matched + - ``origvalue``: current value in file + - ``op``: the operator (e.g. ``+=``) + - ``newlines``: list of lines up to this point. You can use + this to prepend lines before this variable setting + if you wish. + + And should return a four-element tuple: + + - ``newvalue``: new value to substitute in, or ``None`` to drop + the variable setting entirely. (If the removal + results in two consecutive blank lines, one of the + blank lines will also be dropped). + - ``newop``: the operator to use - if you specify ``None`` here, + the original operation will be used. + - ``indent``: number of spaces to indent multi-line entries, + or ``-1`` to indent up to the level of the assignment + and opening quote, or a string to use as the indent. + - ``minbreak``: ``True`` to allow the first element of a + multi-line value to continue on the same line as + the assignment, ``False`` to indent before the first + element. + + To clarify, if you wish not to change the value, then you + would return like this:: + + return origvalue, None, 0, True + - ``match_overrides``: True to match items with _overrides on the end, + False otherwise + Returns a tuple: - updated: - True if changes were made, False otherwise. - newlines: - Lines after processing + + - ``updated``: ``True`` if changes were made, ``False`` otherwise. + - ``newlines``: Lines after processing. """ var_res = {} @@ -1399,12 +1734,13 @@ def edit_metadata(meta_lines, variables, varfunc, match_overrides=False): def edit_metadata_file(meta_file, variables, varfunc): - """Edit a recipe or config file and modify one or more specified + """Edit a recipe or configuration file and modify one or more specified variable values set in the file using a specified callback function. The file is only written to if the value(s) actually change. - This is basically the file version of edit_metadata(), see that + This is basically the file version of ``bb.utils.edit_metadata()``, see that function's description for parameter/usage information. - Returns True if the file was written to, False otherwise. + + Returns ``True`` if the file was written to, ``False`` otherwise. """ with open(meta_file, 'r') as f: (updated, newlines) = edit_metadata(f, variables, varfunc) @@ -1415,23 +1751,25 @@ def edit_metadata_file(meta_file, variables, varfunc): def edit_bblayers_conf(bblayers_conf, add, remove, edit_cb=None): - """Edit bblayers.conf, adding and/or removing layers - Parameters: - bblayers_conf: path to bblayers.conf file to edit - add: layer path (or list of layer paths) to add; None or empty - list to add nothing - remove: layer path (or list of layer paths) to remove; None or - empty list to remove nothing - edit_cb: optional callback function that will be called after - processing adds/removes once per existing entry. + """Edit ``bblayers.conf``, adding and/or removing layers. + + Arguments: + + - ``bblayers_conf``: path to ``bblayers.conf`` file to edit + - ``add``: layer path (or list of layer paths) to add; ``None`` or empty + list to add nothing + - ``remove``: layer path (or list of layer paths) to remove; ``None`` or + empty list to remove nothing + - ``edit_cb``: optional callback function that will be called + after processing adds/removes once per existing entry. + Returns a tuple: - notadded: list of layers specified to be added but weren't - (because they were already in the list) - notremoved: list of layers that were specified to be removed - but weren't (because they weren't in the list) - """ - import fnmatch + - ``notadded``: list of layers specified to be added but weren't + (because they were already in the list) + - ``notremoved``: list of layers that were specified to be removed + but weren't (because they weren't in the list) + """ def remove_trailing_sep(pth): if pth and pth[-1] == os.sep: @@ -1550,7 +1888,22 @@ def get_collection_res(d): def get_file_layer(filename, d, collection_res={}): - """Determine the collection (as defined by a layer's layer.conf file) containing the specified file""" + """Determine the collection (or layer name, as defined by a layer's + ``layer.conf`` file) containing the specified file. + + Arguments: + + - ``filename``: the filename to look for. + - ``d``: the data store. + - ``collection_res``: dictionary with the layer names as keys and file + patterns to match as defined with the BBFILE_COLLECTIONS and + BBFILE_PATTERN variables respectively. The return value of + ``bb.utils.get_collection_res()`` is the default if this variable is + not specified. + + Returns the layer name containing the file. If multiple layers contain the + file, the last matching layer name from collection_res is returned. + """ if not collection_res: collection_res = get_collection_res(d) @@ -1588,7 +1941,13 @@ class PrCtlError(Exception): def signal_on_parent_exit(signame): """ - Trigger signame to be sent when the parent process dies + Trigger ``signame`` to be sent when the parent process dies. + + Arguments: + + - ``signame``: name of the signal. See ``man signal``. + + No return value. """ signum = getattr(signal, signame) # http://linux.die.net/man/2/prctl @@ -1623,7 +1982,7 @@ def ioprio_set(who, cls, value): bb.warn("Unable to set IO Prio for arch %s" % _unamearch) def set_process_name(name): - from ctypes import cdll, byref, create_string_buffer + from ctypes import byref, create_string_buffer # This is nice to have for debugging, not essential try: libc = cdll.LoadLibrary('libc.so.6') @@ -1675,6 +2034,13 @@ def disable_network(uid=None, gid=None): Disable networking in the current process if the kernel supports it, else just return after logging to debug. To do this we need to create a new user namespace, then map back to the original uid/gid. + + Arguments: + + - ``uid``: original user id. + - ``gid``: original user group id. + + No return value. """ libc = ctypes.CDLL('libc.so.6') @@ -1744,9 +2110,14 @@ class LogCatcher(logging.Handler): def is_semver(version): """ - Is the version string following the semver semantic? + Arguments: - https://semver.org/spec/v2.0.0.html + - ``version``: the version string. + + Returns ``True`` if the version string follow semantic versioning, ``False`` + otherwise. + + See https://semver.org/spec/v2.0.0.html. """ regex = re.compile( r""" @@ -1784,6 +2155,8 @@ def rename(src, dst): def environment(**envvars): """ Context manager to selectively update the environment with the specified mapping. + + No return value. """ backup = dict(os.environ) try: @@ -1800,6 +2173,13 @@ def is_local_uid(uid=''): """ Check whether uid is a local one or not. Can't use pwd module since it gets all UIDs, not local ones only. + + Arguments: + + - ``uid``: user id. If not specified the user id is determined from + ``os.getuid()``. + + Returns ``True`` is the user id is local, ``False`` otherwise. """ if not uid: uid = os.getuid() @@ -1814,7 +2194,7 @@ def is_local_uid(uid=''): def mkstemp(suffix=None, prefix=None, dir=None, text=False): """ - Generates a unique filename, independent of time. + Generates a unique temporary file, independent of time. mkstemp() in glibc (at least) generates unique file names based on the current system time. When combined with highly parallel builds, and @@ -1823,6 +2203,18 @@ def mkstemp(suffix=None, prefix=None, dir=None, text=False): This function adds additional entropy to the file name so that a collision is independent of time and thus extremely unlikely. + + Arguments: + + - ``suffix``: filename suffix. + - ``prefix``: filename prefix. + - ``dir``: directory where the file will be created. + - ``text``: if ``True``, the file is opened in text mode. + + Returns a tuple containing: + + - the file descriptor for the created file + - the name of the file. """ entropy = "".join(random.choices("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890", k=20)) if prefix: @@ -1833,12 +2225,20 @@ def mkstemp(suffix=None, prefix=None, dir=None, text=False): def path_is_descendant(descendant, ancestor): """ - Returns True if the path `descendant` is a descendant of `ancestor` - (including being equivalent to `ancestor` itself). Otherwise returns False. + Returns ``True`` if the path ``descendant`` is a descendant of ``ancestor`` + (including being equivalent to ``ancestor`` itself). Otherwise returns + ``False``. + Correctly accounts for symlinks, bind mounts, etc. by using - os.path.samestat() to compare paths + ``os.path.samestat()`` to compare paths. - May raise any exception that os.stat() raises + May raise any exception that ``os.stat()`` raises. + + Arguments: + + - ``descendant``: path to check for being an ancestor. + - ``ancestor``: path to the ancestor ``descendant`` will be checked + against. """ ancestor_stat = os.stat(ancestor) @@ -1854,15 +2254,56 @@ def path_is_descendant(descendant, ancestor): return False +# Recomputing the sets in signal.py is expensive (bitbake -pP idle) +# so try and use _signal directly to avoid it +valid_signals = signal.valid_signals() +try: + import _signal + sigmask = _signal.pthread_sigmask +except ImportError: + sigmask = signal.pthread_sigmask + # If we don't have a timeout of some kind and a process/thread exits badly (for example # OOM killed) and held a lock, we'd just hang in the lock futex forever. It is better # we exit at some point than hang. 5 minutes with no progress means we're probably deadlocked. +# This function can still deadlock python since it can't signal the other threads to exit +# (signals are handled in the main thread) and even os._exit() will wait on non-daemon threads +# to exit. @contextmanager def lock_timeout(lock): - held = lock.acquire(timeout=5*60) try: + s = sigmask(signal.SIG_BLOCK, valid_signals) + held = lock.acquire(timeout=5*60) if not held: + bb.server.process.serverlog("Couldn't get the lock for 5 mins, timed out, exiting.\n%s" % traceback.format_stack()) os._exit(1) yield held finally: lock.release() + sigmask(signal.SIG_SETMASK, s) + +# A version of lock_timeout without the check that the lock was locked and a shorter timeout +@contextmanager +def lock_timeout_nocheck(lock): + l = False + try: + s = sigmask(signal.SIG_BLOCK, valid_signals) + l = lock.acquire(timeout=10) + yield l + finally: + if l: + lock.release() + sigmask(signal.SIG_SETMASK, s) + +def is_path_on_nfs(path): + """ + Returns True if ``path`` argument is on a NFS mount. + """ + # strip not existing path + if os.path.isabs(path): + while not os.path.exists(path): + path = os.path.dirname(path) + + import bb.process + fstype = bb.process.run("stat -f -c %T {}".format(path))[0].strip() + return fstype == "nfs" diff --git a/bitbake/lib/bblayers/action.py b/bitbake/lib/bblayers/action.py index a8f26993..a14f1994 100644 --- a/bitbake/lib/bblayers/action.py +++ b/bitbake/lib/bblayers/action.py @@ -50,8 +50,8 @@ class ActionPlugin(LayerPlugin): try: notadded, _ = bb.utils.edit_bblayers_conf(bblayers_conf, layerdirs, None) - self.tinfoil.modified_files() if not (args.force or notadded): + self.tinfoil.modified_files() try: self.tinfoil.run_command('parseConfiguration') except (bb.tinfoil.TinfoilUIException, bb.BBHandledException): @@ -83,6 +83,8 @@ class ActionPlugin(LayerPlugin): layerdir = os.path.abspath(item) layerdirs.append(layerdir) (_, notremoved) = bb.utils.edit_bblayers_conf(bblayers_conf, None, layerdirs) + if args.force > 1: + return 0 self.tinfoil.modified_files() if notremoved: for item in notremoved: diff --git a/bitbake/lib/bblayers/layerindex.py b/bitbake/lib/bblayers/layerindex.py index ba91fac6..2ba1103a 100644 --- a/bitbake/lib/bblayers/layerindex.py +++ b/bitbake/lib/bblayers/layerindex.py @@ -55,13 +55,15 @@ class LayerIndexPlugin(ActionPlugin): switching branches if necessary and possible. """ base_cmd = ['git', '--git-dir=%s/.git' % repodir, '--work-tree=%s' % repodir] - cmd = base_cmd + ['branch'] + # Get current branch name + cmd = base_cmd + ['name-rev', '--name-only', 'HEAD'] completed_proc = subprocess.run(cmd, text=True, capture_output=True) if completed_proc.returncode: - logger.error("Unable to validate repo %s (%s)" % (repodir, stderr)) + logger.error("Unable to validate repo %s (%s)" % (repodir, completed_proc.stderr)) return None, None, None else: - if branch != completed_proc.stdout[2:-1]: + current_branch = completed_proc.stdout.strip() + if branch != current_branch: cmd = base_cmd + ['status', '--short'] completed_proc = subprocess.run(cmd, text=True, capture_output=True) if completed_proc.stdout.count('\n') != 0: @@ -114,7 +116,7 @@ class LayerIndexPlugin(ActionPlugin): # Load the cooker DB cookerIndex = layerindexlib.LayerIndex(self.tinfoil.config_data) - cookerIndex.load_layerindex('cooker://', load='layerDependencies') + cookerIndex.load_layerindex('cooker://', load=['layerDependencies']) # Fast path, check if we already have what has been requested! (dependencies, invalidnames) = cookerIndex.find_dependencies(names=args.layername, ignores=ignore_layers) @@ -135,7 +137,7 @@ class LayerIndexPlugin(ActionPlugin): for remoteurl in _construct_url(apiurl, branches): logger.plain("Loading %s..." % remoteurl) - remoteIndex.load_layerindex(remoteurl) + remoteIndex.load_layerindex(remoteurl, load=['layerDependencies']) if remoteIndex.is_empty(): logger.error("Remote layer index %s is empty for branches %s" % (apiurl, branches)) diff --git a/bitbake/lib/bblayers/query.py b/bitbake/lib/bblayers/query.py index bfc18a75..7b8d1afd 100644 --- a/bitbake/lib/bblayers/query.py +++ b/bitbake/lib/bblayers/query.py @@ -52,7 +52,7 @@ layer, with the preferred version first. Note that skipped recipes that are overlayed will also be listed, with a " (skipped)" suffix. """ - items_listed = self.list_recipes('Overlayed recipes', None, True, args.same_version, args.filenames, False, True, None, False, None, args.mc) + items_listed = self.list_recipes('Overlayed recipes', None, True, args.same_version, args.filenames, False, True, None, False, None, args.mc, False) # Check for overlayed .bbclass files classes = collections.defaultdict(list) @@ -119,9 +119,9 @@ skipped recipes will also be listed, with a " (skipped)" suffix. title = 'Matching recipes:' else: title = 'Available recipes:' - self.list_recipes(title, args.pnspec, False, False, args.filenames, args.recipes_only, args.multiple, args.layer, args.bare, inheritlist, args.mc) + self.list_recipes(title, args.pnspec, False, False, args.filenames, args.recipes_only, args.multiple, args.layer, args.bare, inheritlist, args.mc, args.show_variants) - def list_recipes(self, title, pnspec, show_overlayed_only, show_same_ver_only, show_filenames, show_recipes_only, show_multi_provider_only, selected_layer, bare, inherits, mc): + def list_recipes(self, title, pnspec, show_overlayed_only, show_same_ver_only, show_filenames, show_recipes_only, show_multi_provider_only, selected_layer, bare, inherits, mc, show_variants=False): if inherits: bbpath = str(self.tinfoil.config_data.getVar('BBPATH')) for classname in inherits: @@ -142,10 +142,10 @@ skipped recipes will also be listed, with a " (skipped)" suffix. # Ensure we list skipped recipes # We are largely guessing about PN, PV and the preferred version here, # but we have no choice since skipped recipes are not fully parsed - skiplist = list(self.tinfoil.cooker.skiplist.keys()) - mcspec = 'mc:%s:' % mc + skiplist = list(self.tinfoil.cooker.skiplist_by_mc[mc].keys()) + if mc: - skiplist = [s[len(mcspec):] for s in skiplist if s.startswith(mcspec)] + skiplist = [s.removeprefix(f'mc:{mc}:') for s in skiplist] for fn in skiplist: recipe_parts = os.path.splitext(os.path.basename(fn))[0].split('_') @@ -162,7 +162,7 @@ skipped recipes will also be listed, with a " (skipped)" suffix. def print_item(f, pn, ver, layer, ispref): if not selected_layer or layer == selected_layer: if not bare and f in skiplist: - skipped = ' (skipped: %s)' % self.tinfoil.cooker.skiplist[f].skipreason + skipped = ' (skipped: %s)' % self.tinfoil.cooker.skiplist_by_mc[mc][f].skipreason else: skipped = '' if show_filenames: @@ -203,8 +203,10 @@ skipped recipes will also be listed, with a " (skipped)" suffix. # We only display once per recipe, we should prefer non extended versions of the # recipe if present (so e.g. in OpenEmbedded, openssl rather than nativesdk-openssl # which would otherwise sort first). + # However, if show_variants is True, we show all BBCLASSEXTEND variants. if realfn[1] and realfn[0] in self.tinfoil.cooker.recipecaches[mc].pkg_fn: - continue + if not show_variants: + continue if inherits: matchcount = 0 @@ -229,7 +231,11 @@ skipped recipes will also be listed, with a " (skipped)" suffix. for prov in allproviders[p]: provfile = bb.cache.virtualfn2realfn(prov[1])[0] provlayer = self.get_file_layer(provfile) - provs.append((provfile, provlayer, prov[0])) + # When show_variants is True, use the virtual filename to distinguish BBCLASSEXTEND variants + if show_variants: + provs.append((prov[1], provlayer, prov[0])) + else: + provs.append((provfile, provlayer, prov[0])) if provlayer != preflayer: multilayer = True if prov[0] != pref[0]: @@ -238,10 +244,12 @@ skipped recipes will also be listed, with a " (skipped)" suffix. if not items_listed: logger.plain('=== %s ===' % title) items_listed = True + # When show_variants is True, use the virtual filename for preference check + pref_cmp_file = pref[1] if show_variants else preffile print_item(preffile, p, self.version_str(pref[0][0], pref[0][1]), preflayer, True) for (provfile, provlayer, provver) in provs: - if provfile != preffile: - print_item(provfile, p, self.version_str(provver[0], provver[1]), provlayer, False) + if provfile != pref_cmp_file: + print_item(bb.cache.virtualfn2realfn(provfile)[0] if show_variants else provfile, p, self.version_str(provver[0], provver[1]), provlayer, False) # Ensure we don't show two entries for BBCLASSEXTENDed recipes preffiles.append(preffile) @@ -301,7 +309,7 @@ Lists recipes with the bbappends that apply to them as subitems. if self.show_appends_for_pn(pn, cooker_data, args.mc): appends = True - if not args.pnspec and self.show_appends_for_skipped(): + if not args.pnspec and self.show_appends_for_skipped(args.mc): appends = True if not appends: @@ -317,9 +325,9 @@ Lists recipes with the bbappends that apply to them as subitems. return self.show_appends_output(filenames, best_filename) - def show_appends_for_skipped(self): + def show_appends_for_skipped(self, mc): filenames = [os.path.basename(f) - for f in self.tinfoil.cooker.skiplist.keys()] + for f in self.tinfoil.cooker.skiplist_by_mc[mc].keys()] return self.show_appends_output(filenames, None, " (skipped)") def show_appends_output(self, filenames, best_filename, name_suffix = ''): @@ -530,6 +538,7 @@ NOTE: .bbappend files can impact the dependencies. parser_show_recipes.add_argument('-i', '--inherits', help='only list recipes that inherit the named class(es) - separate multiple classes using , (without spaces)', metavar='CLASS', default='') parser_show_recipes.add_argument('-l', '--layer', help='only list recipes from the selected layer', default='') parser_show_recipes.add_argument('-b', '--bare', help='output just names without the "(skipped)" marker', action='store_true') + parser_show_recipes.add_argument('--show-variants', help='show all BBCLASSEXTEND recipe variants (e.g., devupstream, nativesdk)', action='store_true') parser_show_recipes.add_argument('--mc', help='use specified multiconfig', default='') parser_show_recipes.add_argument('pnspec', nargs='*', help='optional recipe name specification (wildcards allowed, enclose in quotes to avoid shell expansion)') diff --git a/bitbake/lib/bs4/AUTHORS b/bitbake/lib/bs4/AUTHORS new file mode 100644 index 00000000..1f14fe07 --- /dev/null +++ b/bitbake/lib/bs4/AUTHORS @@ -0,0 +1,49 @@ +Behold, mortal, the origins of Beautiful Soup... +================================================ + +Leonard Richardson is the primary maintainer. + +Aaron DeVore and Isaac Muse have made significant contributions to the +code base. + +Mark Pilgrim provided the encoding detection code that forms the base +of UnicodeDammit. + +Thomas Kluyver and Ezio Melotti finished the work of getting Beautiful +Soup 4 working under Python 3. + +Simon Willison wrote soupselect, which was used to make Beautiful Soup +support CSS selectors. Isaac Muse wrote SoupSieve, which made it +possible to _remove_ the CSS selector code from Beautiful Soup. + +Sam Ruby helped with a lot of edge cases. + +Jonathan Ellis was awarded the prestigious Beau Potage D'Or for his +work in solving the nestable tags conundrum. + +An incomplete list of people have contributed patches to Beautiful +Soup: + + Istvan Albert, Andrew Lin, Anthony Baxter, Oliver Beattie, Andrew +Boyko, Tony Chang, Francisco Canas, "Delong", Zephyr Fang, Fuzzy, +Roman Gaufman, Yoni Gilad, Richie Hindle, Toshihiro Kamiya, Peteris +Krumins, Kent Johnson, Marek Kapolka, Andreas Kostyrka, Roel Kramer, +Ben Last, Robert Leftwich, Stefaan Lippens, "liquider", Staffan +Malmgren, Ksenia Marasanova, JP Moins, Adam Monsen, John Nagle, "Jon", +Ed Oskiewicz, Martijn Peters, Greg Phillips, Giles Radford, Stefano +Revera, Arthur Rudolph, Marko Samastur, James Salter, Jouni Seppänen, +Alexander Schmolck, Tim Shirley, Geoffrey Sneddon, Ville Skyttä, +"Vikas", Jens Svalgaard, Andy Theyers, Eric Weiser, Glyn Webster, John +Wiseman, Paul Wright, Danny Yoo + +An incomplete list of people who made suggestions or found bugs or +found ways to break Beautiful Soup: + + Hanno Böck, Matteo Bertini, Chris Curvey, Simon Cusack, Bruce Eckel, + Matt Ernst, Michael Foord, Tom Harris, Bill de hOra, Donald Howes, + Matt Patterson, Scott Roberts, Steve Strassmann, Mike Williams, + warchild at redho dot com, Sami Kuisma, Carlos Rocha, Bob Hutchison, + Joren Mc, Michal Migurski, John Kleven, Tim Heaney, Tripp Lilley, Ed + Summers, Dennis Sutch, Chris Smith, Aaron Swartz, Stuart + Turner, Greg Edwards, Kevin J Kalupson, Nikos Kouremenos, Artur de + Sousa Rocha, Yichun Wei, Per Vognsen diff --git a/bitbake/lib/bs4/AUTHORS.txt b/bitbake/lib/bs4/AUTHORS.txt deleted file mode 100644 index 2ac8fcc8..00000000 --- a/bitbake/lib/bs4/AUTHORS.txt +++ /dev/null @@ -1,43 +0,0 @@ -Behold, mortal, the origins of Beautiful Soup... -================================================ - -Leonard Richardson is the primary programmer. - -Aaron DeVore is awesome. - -Mark Pilgrim provided the encoding detection code that forms the base -of UnicodeDammit. - -Thomas Kluyver and Ezio Melotti finished the work of getting Beautiful -Soup 4 working under Python 3. - -Simon Willison wrote soupselect, which was used to make Beautiful Soup -support CSS selectors. - -Sam Ruby helped with a lot of edge cases. - -Jonathan Ellis was awarded the prestigous Beau Potage D'Or for his -work in solving the nestable tags conundrum. - -An incomplete list of people have contributed patches to Beautiful -Soup: - - Istvan Albert, Andrew Lin, Anthony Baxter, Andrew Boyko, Tony Chang, - Zephyr Fang, Fuzzy, Roman Gaufman, Yoni Gilad, Richie Hindle, Peteris - Krumins, Kent Johnson, Ben Last, Robert Leftwich, Staffan Malmgren, - Ksenia Marasanova, JP Moins, Adam Monsen, John Nagle, "Jon", Ed - Oskiewicz, Greg Phillips, Giles Radford, Arthur Rudolph, Marko - Samastur, Jouni Seppänen, Alexander Schmolck, Andy Theyers, Glyn - Webster, Paul Wright, Danny Yoo - -An incomplete list of people who made suggestions or found bugs or -found ways to break Beautiful Soup: - - Hanno Böck, Matteo Bertini, Chris Curvey, Simon Cusack, Bruce Eckel, - Matt Ernst, Michael Foord, Tom Harris, Bill de hOra, Donald Howes, - Matt Patterson, Scott Roberts, Steve Strassmann, Mike Williams, - warchild at redho dot com, Sami Kuisma, Carlos Rocha, Bob Hutchison, - Joren Mc, Michal Migurski, John Kleven, Tim Heaney, Tripp Lilley, Ed - Summers, Dennis Sutch, Chris Smith, Aaron Sweep^W Swartz, Stuart - Turner, Greg Edwards, Kevin J Kalupson, Nikos Kouremenos, Artur de - Sousa Rocha, Yichun Wei, Per Vognsen diff --git a/bitbake/lib/bs4/NEWS.txt b/bitbake/lib/bs4/CHANGELOG similarity index 56% rename from bitbake/lib/bs4/NEWS.txt rename to bitbake/lib/bs4/CHANGELOG index 88a60a24..2701446a 100644 --- a/bitbake/lib/bs4/NEWS.txt +++ b/bitbake/lib/bs4/CHANGELOG @@ -1,3 +1,776 @@ += 4.12.3 (20240117) + +* The Beautiful Soup documentation now has a Spanish translation, thanks + to Carlos Romero. Delong Wang's Chinese translation has been updated + to cover Beautiful Soup 4.12.0. + +* Fixed a regression such that if you set .hidden on a tag, the tag + becomes invisible but its contents are still visible. User manipulation + of .hidden is not a documented or supported feature, so don't do this, + but it wasn't too difficult to keep the old behavior working. + +* Fixed a case found by Mengyuhan where html.parser giving up on + markup would result in an AssertionError instead of a + ParserRejectedMarkup exception. + +* Added the correct stacklevel to instances of the XMLParsedAsHTMLWarning. + [bug=2034451] + +* Corrected the syntax of the license definition in pyproject.toml. Patch + by Louis Maddox. [bug=2032848] + +* Corrected a typo in a test that was causing test failures when run against + libxml2 2.12.1. [bug=2045481] + += 4.12.2 (20230407) + +* Fixed an unhandled exception in BeautifulSoup.decode_contents + and methods that call it. [bug=2015545] + += 4.12.1 (20230405) + +NOTE: the following things are likely to be dropped in the next +feature release of Beautiful Soup: + + Official support for Python 3.6. + Inclusion of unit tests and test data in the wheel file. + Two scripts: demonstrate_parser_differences.py and test-all-versions. + +Changes: + +* This version of Beautiful Soup replaces setup.py and setup.cfg + with pyproject.toml. Beautiful Soup now uses tox as its test backend + and hatch to do builds. + +* The main functional improvement in this version is a nonrecursive technique + for regenerating a tree. This technique is used to avoid situations where, + in previous versions, doing something to a very deeply nested tree + would overflow the Python interpreter stack: + + 1. Outputting a tree as a string, e.g. with + BeautifulSoup.encode() [bug=1471755] + + 2. Making copies of trees (copy.copy() and + copy.deepcopy() from the Python standard library). [bug=1709837] + + 3. Pickling a BeautifulSoup object. (Note that pickling a Tag + object can still cause an overflow.) + +* Making a copy of a BeautifulSoup object no longer parses the + document again, which should improve performance significantly. + +* When a BeautifulSoup object is unpickled, Beautiful Soup now + tries to associate an appropriate TreeBuilder object with it. + +* Tag.prettify() will now consistently end prettified markup with + a newline. + +* Added unit tests for fuzz test cases created by third + parties. Some of these tests are skipped since they point + to problems outside of Beautiful Soup, but this change + puts them all in one convenient place. + +* PageElement now implements the known_xml attribute. (This was technically + a bug, but it shouldn't be an issue in normal use.) [bug=2007895] + +* The demonstrate_parser_differences.py script was still written in + Python 2. I've converted it to Python 3, but since no one has + mentioned this over the years, it's a sign that no one uses this + script and it's not serving its purpose. + += 4.12.0 (20230320) + +* Introduced the .css property, which centralizes all access to + the Soup Sieve API. This allows Beautiful Soup to give direct + access to as much of Soup Sieve that makes sense, without cluttering + the BeautifulSoup and Tag classes with a lot of new methods. + + This does mean one addition to the BeautifulSoup and Tag classes + (the .css property itself), so this might be a breaking change if you + happen to use Beautiful Soup to parse XML that includes a tag called + . In particular, code like this will stop working in 4.12.0: + + soup.css['id'] + + Code like this will work just as before: + + soup.find_one('css')['id'] + + The Soup Sieve methods supported through the .css property are + select(), select_one(), iselect(), closest(), match(), filter(), + escape(), and compile(). The BeautifulSoup and Tag classes still + support the select() and select_one() methods; they have not been + deprecated, but they have been demoted to convenience methods. + + [bug=2003677] + +* When the html.parser parser decides it can't parse a document, Beautiful + Soup now consistently propagates this fact by raising a + ParserRejectedMarkup error. [bug=2007343] + +* Removed some error checking code from diagnose(), which is redundant with + similar (but more Pythonic) code in the BeautifulSoup constructor. + [bug=2007344] + +* Added intersphinx references to the documentation so that other + projects have a target to point to when they reference Beautiful + Soup classes. [bug=1453370] + += 4.11.2 (20230131) + +* Fixed test failures caused by nondeterministic behavior of + UnicodeDammit's character detection, depending on the platform setup. + [bug=1973072] + +* Fixed another crash when overriding multi_valued_attributes and using the + html5lib parser. [bug=1948488] + +* The HTMLFormatter and XMLFormatter constructors no longer return a + value. [bug=1992693] + +* Tag.interesting_string_types is now propagated when a tag is + copied. [bug=1990400] + +* Warnings now do their best to provide an appropriate stacklevel, + improving the usefulness of the message. [bug=1978744] + +* Passing a Tag's .contents into PageElement.extend() now works the + same way as passing the Tag itself. + +* Soup Sieve tests will be skipped if the library is not installed. + += 4.11.1 (20220408) + +This release was done to ensure that the unit tests are packaged along +with the released source. There are no functionality changes in this +release, but there are a few other packaging changes: + +* The Japanese and Korean translations of the documentation are included. +* The changelog is now packaged as CHANGELOG, and the license file is + packaged as LICENSE. NEWS.txt and COPYING.txt are still present, + but may be removed in the future. +* TODO.txt is no longer packaged, since a TODO is not relevant for released + code. + += 4.11.0 (20220407) + +* Ported unit tests to use pytest. + +* Added special string classes, RubyParenthesisString and RubyTextString, + to make it possible to treat ruby text specially in get_text() calls. + [bug=1941980] + +* It's now possible to customize the way output is indented by + providing a value for the 'indent' argument to the Formatter + constructor. The 'indent' argument works very similarly to the + argument of the same name in the Python standard library's + json.dump() function. [bug=1955497] + +* If the charset-normalizer Python module + (https://pypi.org/project/charset-normalizer/) is installed, Beautiful + Soup will use it to detect the character sets of incoming documents. + This is also the module used by newer versions of the Requests library. + For the sake of backwards compatibility, chardet and cchardet both take + precedence if installed. [bug=1955346] + +* Added a workaround for an lxml bug + (https://bugs.launchpad.net/lxml/+bug/1948551) that causes + problems when parsing a Unicode string beginning with BYTE ORDER MARK. + [bug=1947768] + +* Issue a warning when an HTML parser is used to parse a document that + looks like XML but not XHTML. [bug=1939121] + +* Do a better job of keeping track of namespaces as an XML document is + parsed, so that CSS selectors that use namespaces will do the right + thing more often. [bug=1946243] + +* Some time ago, the misleadingly named "text" argument to find-type + methods was renamed to the more accurate "string." But this supposed + "renaming" didn't make it into important places like the method + signatures or the docstrings. That's corrected in this + version. "text" still works, but will give a DeprecationWarning. + [bug=1947038] + +* Fixed a crash when pickling a BeautifulSoup object that has no + tree builder. [bug=1934003] + +* Fixed a crash when overriding multi_valued_attributes and using the + html5lib parser. [bug=1948488] + +* Standardized the wording of the MarkupResemblesLocatorWarning + warnings to omit untrusted input and make the warnings less + judgmental about what you ought to be doing. [bug=1955450] + +* Removed support for the iconv_codec library, which doesn't seem + to exist anymore and was never put up on PyPI. (The closest + replacement on PyPI, iconv_codecs, is GPL-licensed, so we can't use + it--it's also quite old.) + += 4.10.0 (20210907) + +* This is the first release of Beautiful Soup to only support Python + 3. I dropped Python 2 support to maintain support for newer versions + (58 and up) of setuptools. See: + https://github.com/pypa/setuptools/issues/2769 [bug=1942919] + +* The behavior of methods like .get_text() and .strings now differs + depending on the type of tag. The change is visible with HTML tags + like - -Hello, world! - - -''' - soup = self.soup(html) - self.assertEqual("text/javascript", soup.find('script')['type']) - - def test_comment(self): - # Comments are represented as Comment objects. - markup = "

foobaz

" - self.assertSoupEquals(markup) - - soup = self.soup(markup) - comment = soup.find(text="foobar") - self.assertEqual(comment.__class__, Comment) - - # The comment is properly integrated into the tree. - foo = soup.find(text="foo") - self.assertEqual(comment, foo.next_element) - baz = soup.find(text="baz") - self.assertEqual(comment, baz.previous_element) - - def test_preserved_whitespace_in_pre_and_textarea(self): - """Whitespace must be preserved in
 and ")
-
-    def test_nested_inline_elements(self):
-        """Inline elements can be nested indefinitely."""
-        b_tag = "Inside a B tag"
-        self.assertSoupEquals(b_tag)
-
-        nested_b_tag = "

A nested tag

" - self.assertSoupEquals(nested_b_tag) - - double_nested_b_tag = "

A doubly nested tag

" - self.assertSoupEquals(nested_b_tag) - - def test_nested_block_level_elements(self): - """Block elements can be nested.""" - soup = self.soup('

Foo

') - blockquote = soup.blockquote - self.assertEqual(blockquote.p.b.string, 'Foo') - self.assertEqual(blockquote.b.string, 'Foo') - - def test_correctly_nested_tables(self): - """One table can go inside another one.""" - markup = ('' - '' - "') - - self.assertSoupEquals( - markup, - '
Here's another table:" - '' - '' - '
foo
Here\'s another table:' - '
foo
' - '
') - - self.assertSoupEquals( - "" - "" - "
Foo
Bar
Baz
") - - def test_deeply_nested_multivalued_attribute(self): - # html5lib can set the attributes of the same tag many times - # as it rearranges the tree. This has caused problems with - # multivalued attributes. - markup = '
' - soup = self.soup(markup) - self.assertEqual(["css"], soup.div.div['class']) - - def test_multivalued_attribute_on_html(self): - # html5lib uses a different API to set the attributes ot the - # tag. This has caused problems with multivalued - # attributes. - markup = '' - soup = self.soup(markup) - self.assertEqual(["a", "b"], soup.html['class']) - - def test_angle_brackets_in_attribute_values_are_escaped(self): - self.assertSoupEquals('', '') - - def test_entities_in_attributes_converted_to_unicode(self): - expect = '

' - self.assertSoupEquals('

', expect) - self.assertSoupEquals('

', expect) - self.assertSoupEquals('

', expect) - self.assertSoupEquals('

', expect) - - def test_entities_in_text_converted_to_unicode(self): - expect = '

pi\N{LATIN SMALL LETTER N WITH TILDE}ata

' - self.assertSoupEquals("

piñata

", expect) - self.assertSoupEquals("

piñata

", expect) - self.assertSoupEquals("

piñata

", expect) - self.assertSoupEquals("

piñata

", expect) - - def test_quot_entity_converted_to_quotation_mark(self): - self.assertSoupEquals("

I said "good day!"

", - '

I said "good day!"

') - - def test_out_of_range_entity(self): - expect = "\N{REPLACEMENT CHARACTER}" - self.assertSoupEquals("�", expect) - self.assertSoupEquals("�", expect) - self.assertSoupEquals("�", expect) - - def test_multipart_strings(self): - "Mostly to prevent a recurrence of a bug in the html5lib treebuilder." - soup = self.soup("

\nfoo

") - self.assertEqual("p", soup.h2.string.next_element.name) - self.assertEqual("p", soup.p.name) - self.assertConnectedness(soup) - - def test_head_tag_between_head_and_body(self): - "Prevent recurrence of a bug in the html5lib treebuilder." - content = """ - - foo - -""" - soup = self.soup(content) - self.assertNotEqual(None, soup.html.body) - self.assertConnectedness(soup) - - def test_multiple_copies_of_a_tag(self): - "Prevent recurrence of a bug in the html5lib treebuilder." - content = """ - - - - - -""" - soup = self.soup(content) - self.assertConnectedness(soup.article) - - def test_basic_namespaces(self): - """Parsers don't need to *understand* namespaces, but at the - very least they should not choke on namespaces or lose - data.""" - - markup = b'4' - soup = self.soup(markup) - self.assertEqual(markup, soup.encode()) - html = soup.html - self.assertEqual('http://www.w3.org/1999/xhtml', soup.html['xmlns']) - self.assertEqual( - 'http://www.w3.org/1998/Math/MathML', soup.html['xmlns:mathml']) - self.assertEqual( - 'http://www.w3.org/2000/svg', soup.html['xmlns:svg']) - - def test_multivalued_attribute_value_becomes_list(self): - markup = b'' - soup = self.soup(markup) - self.assertEqual(['foo', 'bar'], soup.a['class']) - - # - # Generally speaking, tests below this point are more tests of - # Beautiful Soup than tests of the tree builders. But parsers are - # weird, so we run these tests separately for every tree builder - # to detect any differences between them. - # - - def test_can_parse_unicode_document(self): - # A seemingly innocuous document... but it's in Unicode! And - # it contains characters that can't be represented in the - # encoding found in the declaration! The horror! - markup = 'Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!' - soup = self.soup(markup) - self.assertEqual('Sacr\xe9 bleu!', soup.body.string) - - def test_soupstrainer(self): - """Parsers should be able to work with SoupStrainers.""" - strainer = SoupStrainer("b") - soup = self.soup("A bold statement", - parse_only=strainer) - self.assertEqual(soup.decode(), "bold") - - def test_single_quote_attribute_values_become_double_quotes(self): - self.assertSoupEquals("", - '') - - def test_attribute_values_with_nested_quotes_are_left_alone(self): - text = """a""" - self.assertSoupEquals(text) - - def test_attribute_values_with_double_nested_quotes_get_quoted(self): - text = """a""" - soup = self.soup(text) - soup.foo['attr'] = 'Brawls happen at "Bob\'s Bar"' - self.assertSoupEquals( - soup.foo.decode(), - """a""") - - def test_ampersand_in_attribute_value_gets_escaped(self): - self.assertSoupEquals('', - '') - - self.assertSoupEquals( - 'foo', - 'foo') - - def test_escaped_ampersand_in_attribute_value_is_left_alone(self): - self.assertSoupEquals('') - - def test_entities_in_strings_converted_during_parsing(self): - # Both XML and HTML entities are converted to Unicode characters - # during parsing. - text = "

<<sacré bleu!>>

" - expected = "

<<sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>>

" - self.assertSoupEquals(text, expected) - - def test_smart_quotes_converted_on_the_way_in(self): - # Microsoft smart quotes are converted to Unicode characters during - # parsing. - quote = b"

\x91Foo\x92

" - soup = self.soup(quote) - self.assertEqual( - soup.p.string, - "\N{LEFT SINGLE QUOTATION MARK}Foo\N{RIGHT SINGLE QUOTATION MARK}") - - def test_non_breaking_spaces_converted_on_the_way_in(self): - soup = self.soup("  ") - self.assertEqual(soup.a.string, "\N{NO-BREAK SPACE}" * 2) - - def test_entities_converted_on_the_way_out(self): - text = "

<<sacré bleu!>>

" - expected = "

<<sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>>

".encode("utf-8") - soup = self.soup(text) - self.assertEqual(soup.p.encode("utf-8"), expected) - - def test_real_iso_latin_document(self): - # Smoke test of interrelated functionality, using an - # easy-to-understand document. - - # Here it is in Unicode. Note that it claims to be in ISO-Latin-1. - unicode_html = '

Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!

' - - # That's because we're going to encode it into ISO-Latin-1, and use - # that to test. - iso_latin_html = unicode_html.encode("iso-8859-1") - - # Parse the ISO-Latin-1 HTML. - soup = self.soup(iso_latin_html) - # Encode it to UTF-8. - result = soup.encode("utf-8") - - # What do we expect the result to look like? Well, it would - # look like unicode_html, except that the META tag would say - # UTF-8 instead of ISO-Latin-1. - expected = unicode_html.replace("ISO-Latin-1", "utf-8") - - # And, of course, it would be in UTF-8, not Unicode. - expected = expected.encode("utf-8") - - # Ta-da! - self.assertEqual(result, expected) - - def test_real_shift_jis_document(self): - # Smoke test to make sure the parser can handle a document in - # Shift-JIS encoding, without choking. - shift_jis_html = ( - b'
'
-            b'\x82\xb1\x82\xea\x82\xcdShift-JIS\x82\xc5\x83R\x81[\x83f'
-            b'\x83B\x83\x93\x83O\x82\xb3\x82\xea\x82\xbd\x93\xfa\x96{\x8c'
-            b'\xea\x82\xcc\x83t\x83@\x83C\x83\x8b\x82\xc5\x82\xb7\x81B'
-            b'
') - unicode_html = shift_jis_html.decode("shift-jis") - soup = self.soup(unicode_html) - - # Make sure the parse tree is correctly encoded to various - # encodings. - self.assertEqual(soup.encode("utf-8"), unicode_html.encode("utf-8")) - self.assertEqual(soup.encode("euc_jp"), unicode_html.encode("euc_jp")) - - def test_real_hebrew_document(self): - # A real-world test to make sure we can convert ISO-8859-9 (a - # Hebrew encoding) to UTF-8. - hebrew_document = b'Hebrew (ISO 8859-8) in Visual Directionality

Hebrew (ISO 8859-8) in Visual Directionality

\xed\xe5\xec\xf9' - soup = self.soup( - hebrew_document, from_encoding="iso8859-8") - self.assertEqual(soup.original_encoding, 'iso8859-8') - self.assertEqual( - soup.encode('utf-8'), - hebrew_document.decode("iso8859-8").encode("utf-8")) - - def test_meta_tag_reflects_current_encoding(self): - # Here's the tag saying that a document is - # encoded in Shift-JIS. - meta_tag = ('') - - # Here's a document incorporating that meta tag. - shift_jis_html = ( - '\n%s\n' - '' - 'Shift-JIS markup goes here.') % meta_tag - soup = self.soup(shift_jis_html) - - # Parse the document, and the charset is seemingly unaffected. - parsed_meta = soup.find('meta', {'http-equiv': 'Content-type'}) - content = parsed_meta['content'] - self.assertEqual('text/html; charset=x-sjis', content) - - # But that value is actually a ContentMetaAttributeValue object. - self.assertTrue(isinstance(content, ContentMetaAttributeValue)) - - # And it will take on a value that reflects its current - # encoding. - self.assertEqual('text/html; charset=utf8', content.encode("utf8")) - - # For the rest of the story, see TestSubstitutions in - # test_tree.py. - - def test_html5_style_meta_tag_reflects_current_encoding(self): - # Here's the tag saying that a document is - # encoded in Shift-JIS. - meta_tag = ('') - - # Here's a document incorporating that meta tag. - shift_jis_html = ( - '\n%s\n' - '' - 'Shift-JIS markup goes here.') % meta_tag - soup = self.soup(shift_jis_html) - - # Parse the document, and the charset is seemingly unaffected. - parsed_meta = soup.find('meta', id="encoding") - charset = parsed_meta['charset'] - self.assertEqual('x-sjis', charset) - - # But that value is actually a CharsetMetaAttributeValue object. - self.assertTrue(isinstance(charset, CharsetMetaAttributeValue)) - - # And it will take on a value that reflects its current - # encoding. - self.assertEqual('utf8', charset.encode("utf8")) - - def test_tag_with_no_attributes_can_have_attributes_added(self): - data = self.soup("text") - data.a['foo'] = 'bar' - self.assertEqual('text', data.a.decode()) - -class XMLTreeBuilderSmokeTest(SoupTest): - - def test_pickle_and_unpickle_identity(self): - # Pickling a tree, then unpickling it, yields a tree identical - # to the original. - tree = self.soup("foo") - dumped = pickle.dumps(tree, 2) - loaded = pickle.loads(dumped) - self.assertEqual(loaded.__class__, BeautifulSoup) - self.assertEqual(loaded.decode(), tree.decode()) - - def test_docstring_generated(self): - soup = self.soup("") - self.assertEqual( - soup.encode(), b'\n') - - def test_xml_declaration(self): - markup = b"""\n""" - soup = self.soup(markup) - self.assertEqual(markup, soup.encode("utf8")) - - def test_real_xhtml_document(self): - """A real XHTML document should come out *exactly* the same as it went in.""" - markup = b""" - - -Hello. -Goodbye. -""" - soup = self.soup(markup) - self.assertEqual( - soup.encode("utf-8"), markup) - - def test_formatter_processes_script_tag_for_xml_documents(self): - doc = """ - -""" - soup = BeautifulSoup(doc, "lxml-xml") - # lxml would have stripped this while parsing, but we can add - # it later. - soup.script.string = 'console.log("< < hey > > ");' - encoded = soup.encode() - self.assertTrue(b"< < hey > >" in encoded) - - def test_can_parse_unicode_document(self): - markup = 'Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!' - soup = self.soup(markup) - self.assertEqual('Sacr\xe9 bleu!', soup.root.string) - - def test_popping_namespaced_tag(self): - markup = 'b2012-07-02T20:33:42Zcd' - soup = self.soup(markup) - self.assertEqual( - str(soup.rss), markup) - - def test_docstring_includes_correct_encoding(self): - soup = self.soup("") - self.assertEqual( - soup.encode("latin1"), - b'\n') - - def test_large_xml_document(self): - """A large XML document should come out the same as it went in.""" - markup = (b'\n' - + b'0' * (2**12) - + b'') - soup = self.soup(markup) - self.assertEqual(soup.encode("utf-8"), markup) - - - def test_tags_are_empty_element_if_and_only_if_they_are_empty(self): - self.assertSoupEquals("

", "

") - self.assertSoupEquals("

foo

") - - def test_namespaces_are_preserved(self): - markup = 'This tag is in the a namespaceThis tag is in the b namespace' - soup = self.soup(markup) - root = soup.root - self.assertEqual("http://example.com/", root['xmlns:a']) - self.assertEqual("http://example.net/", root['xmlns:b']) - - def test_closing_namespaced_tag(self): - markup = '

20010504

' - soup = self.soup(markup) - self.assertEqual(str(soup.p), markup) - - def test_namespaced_attributes(self): - markup = '' - soup = self.soup(markup) - self.assertEqual(str(soup.foo), markup) - - def test_namespaced_attributes_xml_namespace(self): - markup = 'bar' - soup = self.soup(markup) - self.assertEqual(str(soup.foo), markup) - -class HTML5TreeBuilderSmokeTest(HTMLTreeBuilderSmokeTest): - """Smoke test for a tree builder that supports HTML5.""" - - def test_real_xhtml_document(self): - # Since XHTML is not HTML5, HTML5 parsers are not tested to handle - # XHTML documents in any particular way. - pass - - def test_html_tags_have_namespace(self): - markup = "" - soup = self.soup(markup) - self.assertEqual("http://www.w3.org/1999/xhtml", soup.a.namespace) - - def test_svg_tags_have_namespace(self): - markup = '' - soup = self.soup(markup) - namespace = "http://www.w3.org/2000/svg" - self.assertEqual(namespace, soup.svg.namespace) - self.assertEqual(namespace, soup.circle.namespace) - - - def test_mathml_tags_have_namespace(self): - markup = '5' - soup = self.soup(markup) - namespace = 'http://www.w3.org/1998/Math/MathML' - self.assertEqual(namespace, soup.math.namespace) - self.assertEqual(namespace, soup.msqrt.namespace) - - def test_xml_declaration_becomes_comment(self): - markup = '' - soup = self.soup(markup) - self.assertTrue(isinstance(soup.contents[0], Comment)) - self.assertEqual(soup.contents[0], '?xml version="1.0" encoding="utf-8"?') - self.assertEqual("html", soup.contents[0].next_element.name) - -def skipIf(condition, reason): - def nothing(test, *args, **kwargs): - return None - - def decorator(test_item): - if condition: - return nothing - else: - return test_item - - return decorator diff --git a/bitbake/lib/bs4/tests/__init__.py b/bitbake/lib/bs4/tests/__init__.py deleted file mode 100644 index 142c8cc3..00000000 --- a/bitbake/lib/bs4/tests/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"The beautifulsoup tests." diff --git a/bitbake/lib/bs4/tests/test_builder_registry.py b/bitbake/lib/bs4/tests/test_builder_registry.py deleted file mode 100644 index 90cad829..00000000 --- a/bitbake/lib/bs4/tests/test_builder_registry.py +++ /dev/null @@ -1,147 +0,0 @@ -"""Tests of the builder registry.""" - -import unittest -import warnings - -from bs4 import BeautifulSoup -from bs4.builder import ( - builder_registry as registry, - HTMLParserTreeBuilder, - TreeBuilderRegistry, -) - -try: - from bs4.builder import HTML5TreeBuilder - HTML5LIB_PRESENT = True -except ImportError: - HTML5LIB_PRESENT = False - -try: - from bs4.builder import ( - LXMLTreeBuilderForXML, - LXMLTreeBuilder, - ) - LXML_PRESENT = True -except ImportError: - LXML_PRESENT = False - - -class BuiltInRegistryTest(unittest.TestCase): - """Test the built-in registry with the default builders registered.""" - - def test_combination(self): - if LXML_PRESENT: - self.assertEqual(registry.lookup('fast', 'html'), - LXMLTreeBuilder) - - if LXML_PRESENT: - self.assertEqual(registry.lookup('permissive', 'xml'), - LXMLTreeBuilderForXML) - self.assertEqual(registry.lookup('strict', 'html'), - HTMLParserTreeBuilder) - if HTML5LIB_PRESENT: - self.assertEqual(registry.lookup('html5lib', 'html'), - HTML5TreeBuilder) - - def test_lookup_by_markup_type(self): - if LXML_PRESENT: - self.assertEqual(registry.lookup('html'), LXMLTreeBuilder) - self.assertEqual(registry.lookup('xml'), LXMLTreeBuilderForXML) - else: - self.assertEqual(registry.lookup('xml'), None) - if HTML5LIB_PRESENT: - self.assertEqual(registry.lookup('html'), HTML5TreeBuilder) - else: - self.assertEqual(registry.lookup('html'), HTMLParserTreeBuilder) - - def test_named_library(self): - if LXML_PRESENT: - self.assertEqual(registry.lookup('lxml', 'xml'), - LXMLTreeBuilderForXML) - self.assertEqual(registry.lookup('lxml', 'html'), - LXMLTreeBuilder) - if HTML5LIB_PRESENT: - self.assertEqual(registry.lookup('html5lib'), - HTML5TreeBuilder) - - self.assertEqual(registry.lookup('html.parser'), - HTMLParserTreeBuilder) - - def test_beautifulsoup_constructor_does_lookup(self): - - with warnings.catch_warnings(record=True) as w: - # This will create a warning about not explicitly - # specifying a parser, but we'll ignore it. - - # You can pass in a string. - BeautifulSoup("", features="html") - # Or a list of strings. - BeautifulSoup("", features=["html", "fast"]) - - # You'll get an exception if BS can't find an appropriate - # builder. - self.assertRaises(ValueError, BeautifulSoup, - "", features="no-such-feature") - -class RegistryTest(unittest.TestCase): - """Test the TreeBuilderRegistry class in general.""" - - def setUp(self): - self.registry = TreeBuilderRegistry() - - def builder_for_features(self, *feature_list): - cls = type('Builder_' + '_'.join(feature_list), - (object,), {'features' : feature_list}) - - self.registry.register(cls) - return cls - - def test_register_with_no_features(self): - builder = self.builder_for_features() - - # Since the builder advertises no features, you can't find it - # by looking up features. - self.assertEqual(self.registry.lookup('foo'), None) - - # But you can find it by doing a lookup with no features, if - # this happens to be the only registered builder. - self.assertEqual(self.registry.lookup(), builder) - - def test_register_with_features_makes_lookup_succeed(self): - builder = self.builder_for_features('foo', 'bar') - self.assertEqual(self.registry.lookup('foo'), builder) - self.assertEqual(self.registry.lookup('bar'), builder) - - def test_lookup_fails_when_no_builder_implements_feature(self): - builder = self.builder_for_features('foo', 'bar') - self.assertEqual(self.registry.lookup('baz'), None) - - def test_lookup_gets_most_recent_registration_when_no_feature_specified(self): - builder1 = self.builder_for_features('foo') - builder2 = self.builder_for_features('bar') - self.assertEqual(self.registry.lookup(), builder2) - - def test_lookup_fails_when_no_tree_builders_registered(self): - self.assertEqual(self.registry.lookup(), None) - - def test_lookup_gets_most_recent_builder_supporting_all_features(self): - has_one = self.builder_for_features('foo') - has_the_other = self.builder_for_features('bar') - has_both_early = self.builder_for_features('foo', 'bar', 'baz') - has_both_late = self.builder_for_features('foo', 'bar', 'quux') - lacks_one = self.builder_for_features('bar') - has_the_other = self.builder_for_features('foo') - - # There are two builders featuring 'foo' and 'bar', but - # the one that also features 'quux' was registered later. - self.assertEqual(self.registry.lookup('foo', 'bar'), - has_both_late) - - # There is only one builder featuring 'foo', 'bar', and 'baz'. - self.assertEqual(self.registry.lookup('foo', 'bar', 'baz'), - has_both_early) - - def test_lookup_fails_when_cannot_reconcile_requested_features(self): - builder1 = self.builder_for_features('foo', 'bar') - builder2 = self.builder_for_features('foo', 'baz') - self.assertEqual(self.registry.lookup('bar', 'baz'), None) diff --git a/bitbake/lib/bs4/tests/test_docs.py b/bitbake/lib/bs4/tests/test_docs.py deleted file mode 100644 index d1d76a33..00000000 --- a/bitbake/lib/bs4/tests/test_docs.py +++ /dev/null @@ -1,32 +0,0 @@ -"Test harness for doctests." - -# pylint: disable-msg=E0611,W0142 - -__metaclass__ = type -__all__ = [ - 'additional_tests', - ] - -import doctest -#from pkg_resources import ( -# resource_filename, resource_exists, resource_listdir, cleanup_resources) - -DOCTEST_FLAGS = ( - doctest.ELLIPSIS | - doctest.NORMALIZE_WHITESPACE | - doctest.REPORT_NDIFF) - -# def additional_tests(): -# "Run the doc tests (README.txt and docs/*, if any exist)" -# doctest_files = [ -# os.path.abspath(resource_filename('bs4', 'README.txt'))] -# if resource_exists('bs4', 'docs'): -# for name in resource_listdir('bs4', 'docs'): -# if name.endswith('.txt'): -# doctest_files.append( -# os.path.abspath( -# resource_filename('bs4', 'docs/%s' % name))) -# kwargs = dict(module_relative=False, optionflags=DOCTEST_FLAGS) -# atexit.register(cleanup_resources) -# return unittest.TestSuite(( -# doctest.DocFileSuite(*doctest_files, **kwargs))) diff --git a/bitbake/lib/bs4/tests/test_html5lib.py b/bitbake/lib/bs4/tests/test_html5lib.py deleted file mode 100644 index a7494ca5..00000000 --- a/bitbake/lib/bs4/tests/test_html5lib.py +++ /dev/null @@ -1,98 +0,0 @@ -"""Tests to ensure that the html5lib tree builder generates good trees.""" - -import warnings - -try: - from bs4.builder import HTML5TreeBuilder - HTML5LIB_PRESENT = True -except ImportError as e: - HTML5LIB_PRESENT = False -from bs4.element import SoupStrainer -from bs4.testing import ( - HTML5TreeBuilderSmokeTest, - SoupTest, - skipIf, -) - -@skipIf( - not HTML5LIB_PRESENT, - "html5lib seems not to be present, not testing its tree builder.") -class HTML5LibBuilderSmokeTest(SoupTest, HTML5TreeBuilderSmokeTest): - """See ``HTML5TreeBuilderSmokeTest``.""" - - @property - def default_builder(self): - return HTML5TreeBuilder() - - def test_soupstrainer(self): - # The html5lib tree builder does not support SoupStrainers. - strainer = SoupStrainer("b") - markup = "

A bold statement.

" - with warnings.catch_warnings(record=True) as w: - soup = self.soup(markup, parse_only=strainer) - self.assertEqual( - soup.decode(), self.document_for(markup)) - - self.assertTrue( - "the html5lib tree builder doesn't support parse_only" in - str(w[0].message)) - - def test_correctly_nested_tables(self): - """html5lib inserts tags where other parsers don't.""" - markup = ('' - '' - "') - - self.assertSoupEquals( - markup, - '
Here's another table:" - '' - '' - '
foo
Here\'s another table:' - '
foo
' - '
') - - self.assertSoupEquals( - "" - "" - "
Foo
Bar
Baz
") - - def test_xml_declaration_followed_by_doctype(self): - markup = ''' - - - - - -

foo

- -''' - soup = self.soup(markup) - # Verify that we can reach the

tag; this means the tree is connected. - self.assertEqual(b"

foo

", soup.p.encode()) - - def test_reparented_markup(self): - markup = '

foo

\n

bar

' - soup = self.soup(markup) - self.assertEqual("

foo

\n

bar

", soup.body.decode()) - self.assertEqual(2, len(soup.find_all('p'))) - - - def test_reparented_markup_ends_with_whitespace(self): - markup = '

foo

\n

bar

\n' - soup = self.soup(markup) - self.assertEqual("

foo

\n

bar

\n", soup.body.decode()) - self.assertEqual(2, len(soup.find_all('p'))) - - def test_processing_instruction(self): - """Processing instructions become comments.""" - markup = b"""""" - soup = self.soup(markup) - assert str(soup).startswith("") - - def test_cloned_multivalue_node(self): - markup = b"""

""" - soup = self.soup(markup) - a1, a2 = soup.find_all('a') - self.assertEqual(a1, a2) - assert a1 is not a2 diff --git a/bitbake/lib/bs4/tests/test_htmlparser.py b/bitbake/lib/bs4/tests/test_htmlparser.py deleted file mode 100644 index 30a25e67..00000000 --- a/bitbake/lib/bs4/tests/test_htmlparser.py +++ /dev/null @@ -1,31 +0,0 @@ -"""Tests to ensure that the html.parser tree builder generates good -trees.""" - -import pickle -from bs4.testing import SoupTest, HTMLTreeBuilderSmokeTest -from bs4.builder import HTMLParserTreeBuilder - -class HTMLParserTreeBuilderSmokeTest(SoupTest, HTMLTreeBuilderSmokeTest): - - @property - def default_builder(self): - return HTMLParserTreeBuilder() - - def test_namespaced_system_doctype(self): - # html.parser can't handle namespaced doctypes, so skip this one. - pass - - def test_namespaced_public_doctype(self): - # html.parser can't handle namespaced doctypes, so skip this one. - pass - - def test_builder_is_pickled(self): - """Unlike most tree builders, HTMLParserTreeBuilder and will - be restored after pickling. - """ - tree = self.soup("foo") - dumped = pickle.dumps(tree, 2) - loaded = pickle.loads(dumped) - self.assertTrue(isinstance(loaded.builder, type(tree.builder))) - - diff --git a/bitbake/lib/bs4/tests/test_lxml.py b/bitbake/lib/bs4/tests/test_lxml.py deleted file mode 100644 index 6b6cdd07..00000000 --- a/bitbake/lib/bs4/tests/test_lxml.py +++ /dev/null @@ -1,70 +0,0 @@ -"""Tests to ensure that the lxml tree builder generates good trees.""" - -import warnings - -try: - import lxml.etree - LXML_PRESENT = True - LXML_VERSION = lxml.etree.LXML_VERSION -except ImportError as e: - LXML_PRESENT = False - LXML_VERSION = (0,) - -if LXML_PRESENT: - from bs4.builder import LXMLTreeBuilder, LXMLTreeBuilderForXML - -from bs4 import BeautifulStoneSoup -from bs4.testing import skipIf -from bs4.testing import ( - HTMLTreeBuilderSmokeTest, - XMLTreeBuilderSmokeTest, - SoupTest, - skipIf, -) - -@skipIf( - not LXML_PRESENT, - "lxml seems not to be present, not testing its tree builder.") -class LXMLTreeBuilderSmokeTest(SoupTest, HTMLTreeBuilderSmokeTest): - """See ``HTMLTreeBuilderSmokeTest``.""" - - @property - def default_builder(self): - return LXMLTreeBuilder() - - def test_out_of_range_entity(self): - self.assertSoupEquals( - "

foo�bar

", "

foobar

") - self.assertSoupEquals( - "

foo�bar

", "

foobar

") - self.assertSoupEquals( - "

foo�bar

", "

foobar

") - - # In lxml < 2.3.5, an empty doctype causes a segfault. Skip this - # test if an old version of lxml is installed. - - @skipIf( - not LXML_PRESENT or LXML_VERSION < (2,3,5,0), - "Skipping doctype test for old version of lxml to avoid segfault.") - def test_empty_doctype(self): - soup = self.soup("") - doctype = soup.contents[0] - self.assertEqual("", doctype.strip()) - - def test_beautifulstonesoup_is_xml_parser(self): - # Make sure that the deprecated BSS class uses an xml builder - # if one is installed. - with warnings.catch_warnings(record=True) as w: - soup = BeautifulStoneSoup("") - self.assertEqual("", str(soup.b)) - self.assertTrue("BeautifulStoneSoup class is deprecated" in str(w[0].message)) - -@skipIf( - not LXML_PRESENT, - "lxml seems not to be present, not testing its XML tree builder.") -class LXMLXMLTreeBuilderSmokeTest(SoupTest, XMLTreeBuilderSmokeTest): - """See ``HTMLTreeBuilderSmokeTest``.""" - - @property - def default_builder(self): - return LXMLTreeBuilderForXML() diff --git a/bitbake/lib/bs4/tests/test_soup.py b/bitbake/lib/bs4/tests/test_soup.py deleted file mode 100644 index 6ad3cb37..00000000 --- a/bitbake/lib/bs4/tests/test_soup.py +++ /dev/null @@ -1,479 +0,0 @@ -# -*- coding: utf-8 -*- -"""Tests of Beautiful Soup as a whole.""" - -import logging -import unittest -import sys -import tempfile - -from bs4 import BeautifulSoup -from bs4.element import ( - CharsetMetaAttributeValue, - ContentMetaAttributeValue, - SoupStrainer, - NamespacedAttribute, - ) -import bs4.dammit -from bs4.dammit import ( - EntitySubstitution, - UnicodeDammit, - EncodingDetector, -) -from bs4.testing import ( - SoupTest, - skipIf, -) -import warnings - -try: - from bs4.builder import LXMLTreeBuilder, LXMLTreeBuilderForXML - LXML_PRESENT = True -except ImportError as e: - LXML_PRESENT = False - -PYTHON_2_PRE_2_7 = (sys.version_info < (2,7)) -PYTHON_3_PRE_3_2 = (sys.version_info[0] == 3 and sys.version_info < (3,2)) - -class TestConstructor(SoupTest): - - def test_short_unicode_input(self): - data = "

éé

" - soup = self.soup(data) - self.assertEqual("éé", soup.h1.string) - - def test_embedded_null(self): - data = "

foo\0bar

" - soup = self.soup(data) - self.assertEqual("foo\0bar", soup.h1.string) - - def test_exclude_encodings(self): - utf8_data = "Räksmörgås".encode("utf-8") - soup = self.soup(utf8_data, exclude_encodings=["utf-8"]) - self.assertEqual("windows-1252", soup.original_encoding) - - -class TestWarnings(SoupTest): - - def _no_parser_specified(self, s, is_there=True): - v = s.startswith(BeautifulSoup.NO_PARSER_SPECIFIED_WARNING[:80]) - self.assertTrue(v) - - def test_warning_if_no_parser_specified(self): - with warnings.catch_warnings(record=True) as w: - soup = self.soup("") - msg = str(w[0].message) - self._assert_no_parser_specified(msg) - - def test_warning_if_parser_specified_too_vague(self): - with warnings.catch_warnings(record=True) as w: - soup = self.soup("", "html") - msg = str(w[0].message) - self._assert_no_parser_specified(msg) - - def test_no_warning_if_explicit_parser_specified(self): - with warnings.catch_warnings(record=True) as w: - soup = self.soup("", "html.parser") - self.assertEqual([], w) - - def test_parseOnlyThese_renamed_to_parse_only(self): - with warnings.catch_warnings(record=True) as w: - soup = self.soup("", parseOnlyThese=SoupStrainer("b")) - msg = str(w[0].message) - self.assertTrue("parseOnlyThese" in msg) - self.assertTrue("parse_only" in msg) - self.assertEqual(b"", soup.encode()) - - def test_fromEncoding_renamed_to_from_encoding(self): - with warnings.catch_warnings(record=True) as w: - utf8 = b"\xc3\xa9" - soup = self.soup(utf8, fromEncoding="utf8") - msg = str(w[0].message) - self.assertTrue("fromEncoding" in msg) - self.assertTrue("from_encoding" in msg) - self.assertEqual("utf8", soup.original_encoding) - - def test_unrecognized_keyword_argument(self): - self.assertRaises( - TypeError, self.soup, "", no_such_argument=True) - -class TestWarnings(SoupTest): - - def test_disk_file_warning(self): - filehandle = tempfile.NamedTemporaryFile() - filename = filehandle.name - try: - with warnings.catch_warnings(record=True) as w: - soup = self.soup(filename) - msg = str(w[0].message) - self.assertTrue("looks like a filename" in msg) - finally: - filehandle.close() - - # The file no longer exists, so Beautiful Soup will no longer issue the warning. - with warnings.catch_warnings(record=True) as w: - soup = self.soup(filename) - self.assertEqual(0, len(w)) - - def test_url_warning(self): - with warnings.catch_warnings(record=True) as w: - soup = self.soup("http://www.crummy.com/") - msg = str(w[0].message) - self.assertTrue("looks like a URL" in msg) - - with warnings.catch_warnings(record=True) as w: - soup = self.soup("http://www.crummy.com/ is great") - self.assertEqual(0, len(w)) - -class TestSelectiveParsing(SoupTest): - - def test_parse_with_soupstrainer(self): - markup = "NoYesNoYes Yes" - strainer = SoupStrainer("b") - soup = self.soup(markup, parse_only=strainer) - self.assertEqual(soup.encode(), b"YesYes Yes") - - -class TestEntitySubstitution(unittest.TestCase): - """Standalone tests of the EntitySubstitution class.""" - def setUp(self): - self.sub = EntitySubstitution - - def test_simple_html_substitution(self): - # Unicode characters corresponding to named HTML entites - # are substituted, and no others. - s = "foo\u2200\N{SNOWMAN}\u00f5bar" - self.assertEqual(self.sub.substitute_html(s), - "foo∀\N{SNOWMAN}õbar") - - def test_smart_quote_substitution(self): - # MS smart quotes are a common source of frustration, so we - # give them a special test. - quotes = b"\x91\x92foo\x93\x94" - dammit = UnicodeDammit(quotes) - self.assertEqual(self.sub.substitute_html(dammit.markup), - "‘’foo“”") - - def test_xml_converstion_includes_no_quotes_if_make_quoted_attribute_is_false(self): - s = 'Welcome to "my bar"' - self.assertEqual(self.sub.substitute_xml(s, False), s) - - def test_xml_attribute_quoting_normally_uses_double_quotes(self): - self.assertEqual(self.sub.substitute_xml("Welcome", True), - '"Welcome"') - self.assertEqual(self.sub.substitute_xml("Bob's Bar", True), - '"Bob\'s Bar"') - - def test_xml_attribute_quoting_uses_single_quotes_when_value_contains_double_quotes(self): - s = 'Welcome to "my bar"' - self.assertEqual(self.sub.substitute_xml(s, True), - "'Welcome to \"my bar\"'") - - def test_xml_attribute_quoting_escapes_single_quotes_when_value_contains_both_single_and_double_quotes(self): - s = 'Welcome to "Bob\'s Bar"' - self.assertEqual( - self.sub.substitute_xml(s, True), - '"Welcome to "Bob\'s Bar""') - - def test_xml_quotes_arent_escaped_when_value_is_not_being_quoted(self): - quoted = 'Welcome to "Bob\'s Bar"' - self.assertEqual(self.sub.substitute_xml(quoted), quoted) - - def test_xml_quoting_handles_angle_brackets(self): - self.assertEqual( - self.sub.substitute_xml("foo"), - "foo<bar>") - - def test_xml_quoting_handles_ampersands(self): - self.assertEqual(self.sub.substitute_xml("AT&T"), "AT&T") - - def test_xml_quoting_including_ampersands_when_they_are_part_of_an_entity(self): - self.assertEqual( - self.sub.substitute_xml("ÁT&T"), - "&Aacute;T&T") - - def test_xml_quoting_ignoring_ampersands_when_they_are_part_of_an_entity(self): - self.assertEqual( - self.sub.substitute_xml_containing_entities("ÁT&T"), - "ÁT&T") - - def test_quotes_not_html_substituted(self): - """There's no need to do this except inside attribute values.""" - text = 'Bob\'s "bar"' - self.assertEqual(self.sub.substitute_html(text), text) - - -class TestEncodingConversion(SoupTest): - # Test Beautiful Soup's ability to decode and encode from various - # encodings. - - def setUp(self): - super(TestEncodingConversion, self).setUp() - self.unicode_data = 'Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!' - self.utf8_data = self.unicode_data.encode("utf-8") - # Just so you know what it looks like. - self.assertEqual( - self.utf8_data, - b'Sacr\xc3\xa9 bleu!') - - def test_ascii_in_unicode_out(self): - # ASCII input is converted to Unicode. The original_encoding - # attribute is set to 'utf-8', a superset of ASCII. - chardet = bs4.dammit.chardet_dammit - logging.disable(logging.WARNING) - try: - def noop(str): - return None - # Disable chardet, which will realize that the ASCII is ASCII. - bs4.dammit.chardet_dammit = noop - ascii = b"a" - soup_from_ascii = self.soup(ascii) - unicode_output = soup_from_ascii.decode() - self.assertTrue(isinstance(unicode_output, str)) - self.assertEqual(unicode_output, self.document_for(ascii.decode())) - self.assertEqual(soup_from_ascii.original_encoding.lower(), "utf-8") - finally: - logging.disable(logging.NOTSET) - bs4.dammit.chardet_dammit = chardet - - def test_unicode_in_unicode_out(self): - # Unicode input is left alone. The original_encoding attribute - # is not set. - soup_from_unicode = self.soup(self.unicode_data) - self.assertEqual(soup_from_unicode.decode(), self.unicode_data) - self.assertEqual(soup_from_unicode.foo.string, 'Sacr\xe9 bleu!') - self.assertEqual(soup_from_unicode.original_encoding, None) - - def test_utf8_in_unicode_out(self): - # UTF-8 input is converted to Unicode. The original_encoding - # attribute is set. - soup_from_utf8 = self.soup(self.utf8_data) - self.assertEqual(soup_from_utf8.decode(), self.unicode_data) - self.assertEqual(soup_from_utf8.foo.string, 'Sacr\xe9 bleu!') - - def test_utf8_out(self): - # The internal data structures can be encoded as UTF-8. - soup_from_unicode = self.soup(self.unicode_data) - self.assertEqual(soup_from_unicode.encode('utf-8'), self.utf8_data) - - @skipIf( - PYTHON_2_PRE_2_7 or PYTHON_3_PRE_3_2, - "Bad HTMLParser detected; skipping test of non-ASCII characters in attribute name.") - def test_attribute_name_containing_unicode_characters(self): - markup = '
' - self.assertEqual(self.soup(markup).div.encode("utf8"), markup.encode("utf8")) - -class TestUnicodeDammit(unittest.TestCase): - """Standalone tests of UnicodeDammit.""" - - def test_unicode_input(self): - markup = "I'm already Unicode! \N{SNOWMAN}" - dammit = UnicodeDammit(markup) - self.assertEqual(dammit.unicode_markup, markup) - - def test_smart_quotes_to_unicode(self): - markup = b"\x91\x92\x93\x94" - dammit = UnicodeDammit(markup) - self.assertEqual( - dammit.unicode_markup, "\u2018\u2019\u201c\u201d") - - def test_smart_quotes_to_xml_entities(self): - markup = b"\x91\x92\x93\x94" - dammit = UnicodeDammit(markup, smart_quotes_to="xml") - self.assertEqual( - dammit.unicode_markup, "‘’“”") - - def test_smart_quotes_to_html_entities(self): - markup = b"\x91\x92\x93\x94" - dammit = UnicodeDammit(markup, smart_quotes_to="html") - self.assertEqual( - dammit.unicode_markup, "‘’“”") - - def test_smart_quotes_to_ascii(self): - markup = b"\x91\x92\x93\x94" - dammit = UnicodeDammit(markup, smart_quotes_to="ascii") - self.assertEqual( - dammit.unicode_markup, """''""""") - - def test_detect_utf8(self): - utf8 = b"Sacr\xc3\xa9 bleu! \xe2\x98\x83" - dammit = UnicodeDammit(utf8) - self.assertEqual(dammit.original_encoding.lower(), 'utf-8') - self.assertEqual(dammit.unicode_markup, 'Sacr\xe9 bleu! \N{SNOWMAN}') - - - def test_convert_hebrew(self): - hebrew = b"\xed\xe5\xec\xf9" - dammit = UnicodeDammit(hebrew, ["iso-8859-8"]) - self.assertEqual(dammit.original_encoding.lower(), 'iso-8859-8') - self.assertEqual(dammit.unicode_markup, '\u05dd\u05d5\u05dc\u05e9') - - def test_dont_see_smart_quotes_where_there_are_none(self): - utf_8 = b"\343\202\261\343\203\274\343\202\277\343\202\244 Watch" - dammit = UnicodeDammit(utf_8) - self.assertEqual(dammit.original_encoding.lower(), 'utf-8') - self.assertEqual(dammit.unicode_markup.encode("utf-8"), utf_8) - - def test_ignore_inappropriate_codecs(self): - utf8_data = "Räksmörgås".encode("utf-8") - dammit = UnicodeDammit(utf8_data, ["iso-8859-8"]) - self.assertEqual(dammit.original_encoding.lower(), 'utf-8') - - def test_ignore_invalid_codecs(self): - utf8_data = "Räksmörgås".encode("utf-8") - for bad_encoding in ['.utf8', '...', 'utF---16.!']: - dammit = UnicodeDammit(utf8_data, [bad_encoding]) - self.assertEqual(dammit.original_encoding.lower(), 'utf-8') - - def test_exclude_encodings(self): - # This is UTF-8. - utf8_data = "Räksmörgås".encode("utf-8") - - # But if we exclude UTF-8 from consideration, the guess is - # Windows-1252. - dammit = UnicodeDammit(utf8_data, exclude_encodings=["utf-8"]) - self.assertEqual(dammit.original_encoding.lower(), 'windows-1252') - - # And if we exclude that, there is no valid guess at all. - dammit = UnicodeDammit( - utf8_data, exclude_encodings=["utf-8", "windows-1252"]) - self.assertEqual(dammit.original_encoding, None) - - def test_encoding_detector_replaces_junk_in_encoding_name_with_replacement_character(self): - detected = EncodingDetector( - b'') - encodings = list(detected.encodings) - assert 'utf-\N{REPLACEMENT CHARACTER}' in encodings - - def test_detect_html5_style_meta_tag(self): - - for data in ( - b'', - b"", - b"", - b""): - dammit = UnicodeDammit(data, is_html=True) - self.assertEqual( - "euc-jp", dammit.original_encoding) - - def test_last_ditch_entity_replacement(self): - # This is a UTF-8 document that contains bytestrings - # completely incompatible with UTF-8 (ie. encoded with some other - # encoding). - # - # Since there is no consistent encoding for the document, - # Unicode, Dammit will eventually encode the document as UTF-8 - # and encode the incompatible characters as REPLACEMENT - # CHARACTER. - # - # If chardet is installed, it will detect that the document - # can be converted into ISO-8859-1 without errors. This happens - # to be the wrong encoding, but it is a consistent encoding, so the - # code we're testing here won't run. - # - # So we temporarily disable chardet if it's present. - doc = b"""\357\273\277 -\330\250\330\252\330\261 -\310\322\321\220\312\321\355\344""" - chardet = bs4.dammit.chardet_dammit - logging.disable(logging.WARNING) - try: - def noop(str): - return None - bs4.dammit.chardet_dammit = noop - dammit = UnicodeDammit(doc) - self.assertEqual(True, dammit.contains_replacement_characters) - self.assertTrue("\ufffd" in dammit.unicode_markup) - - soup = BeautifulSoup(doc, "html.parser") - self.assertTrue(soup.contains_replacement_characters) - finally: - logging.disable(logging.NOTSET) - bs4.dammit.chardet_dammit = chardet - - def test_byte_order_mark_removed(self): - # A document written in UTF-16LE will have its byte order marker stripped. - data = b'\xff\xfe<\x00a\x00>\x00\xe1\x00\xe9\x00<\x00/\x00a\x00>\x00' - dammit = UnicodeDammit(data) - self.assertEqual("áé", dammit.unicode_markup) - self.assertEqual("utf-16le", dammit.original_encoding) - - def test_detwingle(self): - # Here's a UTF8 document. - utf8 = ("\N{SNOWMAN}" * 3).encode("utf8") - - # Here's a Windows-1252 document. - windows_1252 = ( - "\N{LEFT DOUBLE QUOTATION MARK}Hi, I like Windows!" - "\N{RIGHT DOUBLE QUOTATION MARK}").encode("windows_1252") - - # Through some unholy alchemy, they've been stuck together. - doc = utf8 + windows_1252 + utf8 - - # The document can't be turned into UTF-8: - self.assertRaises(UnicodeDecodeError, doc.decode, "utf8") - - # Unicode, Dammit thinks the whole document is Windows-1252, - # and decodes it into "☃☃☃“Hi, I like Windows!”☃☃☃" - - # But if we run it through fix_embedded_windows_1252, it's fixed: - - fixed = UnicodeDammit.detwingle(doc) - self.assertEqual( - "☃☃☃“Hi, I like Windows!”☃☃☃", fixed.decode("utf8")) - - def test_detwingle_ignores_multibyte_characters(self): - # Each of these characters has a UTF-8 representation ending - # in \x93. \x93 is a smart quote if interpreted as - # Windows-1252. But our code knows to skip over multibyte - # UTF-8 characters, so they'll survive the process unscathed. - for tricky_unicode_char in ( - "\N{LATIN SMALL LIGATURE OE}", # 2-byte char '\xc5\x93' - "\N{LATIN SUBSCRIPT SMALL LETTER X}", # 3-byte char '\xe2\x82\x93' - "\xf0\x90\x90\x93", # This is a CJK character, not sure which one. - ): - input = tricky_unicode_char.encode("utf8") - self.assertTrue(input.endswith(b'\x93')) - output = UnicodeDammit.detwingle(input) - self.assertEqual(output, input) - -class TestNamedspacedAttribute(SoupTest): - - def test_name_may_be_none(self): - a = NamespacedAttribute("xmlns", None) - self.assertEqual(a, "xmlns") - - def test_attribute_is_equivalent_to_colon_separated_string(self): - a = NamespacedAttribute("a", "b") - self.assertEqual("a:b", a) - - def test_attributes_are_equivalent_if_prefix_and_name_identical(self): - a = NamespacedAttribute("a", "b", "c") - b = NamespacedAttribute("a", "b", "c") - self.assertEqual(a, b) - - # The actual namespace is not considered. - c = NamespacedAttribute("a", "b", None) - self.assertEqual(a, c) - - # But name and prefix are important. - d = NamespacedAttribute("a", "z", "c") - self.assertNotEqual(a, d) - - e = NamespacedAttribute("z", "b", "c") - self.assertNotEqual(a, e) - - -class TestAttributeValueWithCharsetSubstitution(unittest.TestCase): - - def test_content_meta_attribute_value(self): - value = CharsetMetaAttributeValue("euc-jp") - self.assertEqual("euc-jp", value) - self.assertEqual("euc-jp", value.original_value) - self.assertEqual("utf8", value.encode("utf8")) - - - def test_content_meta_attribute_value(self): - value = ContentMetaAttributeValue("text/html; charset=euc-jp") - self.assertEqual("text/html; charset=euc-jp", value) - self.assertEqual("text/html; charset=euc-jp", value.original_value) - self.assertEqual("text/html; charset=utf8", value.encode("utf8")) diff --git a/bitbake/lib/bs4/tests/test_tree.py b/bitbake/lib/bs4/tests/test_tree.py deleted file mode 100644 index cf0f1abe..00000000 --- a/bitbake/lib/bs4/tests/test_tree.py +++ /dev/null @@ -1,2004 +0,0 @@ -# -*- coding: utf-8 -*- -"""Tests for Beautiful Soup's tree traversal methods. - -The tree traversal methods are the main advantage of using Beautiful -Soup over just using a parser. - -Different parsers will build different Beautiful Soup trees given the -same markup, but all Beautiful Soup trees can be traversed with the -methods tested here. -""" - -import copy -import pickle -import re -import warnings -from bs4 import BeautifulSoup -from bs4.builder import builder_registry -from bs4.element import ( - PY3K, - CData, - Comment, - Declaration, - Doctype, - NavigableString, - SoupStrainer, - Tag, -) -from bs4.testing import SoupTest - -XML_BUILDER_PRESENT = (builder_registry.lookup("xml") is not None) -LXML_PRESENT = (builder_registry.lookup("lxml") is not None) - -class TreeTest(SoupTest): - - def assertSelects(self, tags, should_match): - """Make sure that the given tags have the correct text. - - This is used in tests that define a bunch of tags, each - containing a single string, and then select certain strings by - some mechanism. - """ - self.assertEqual([tag.string for tag in tags], should_match) - - def assertSelectsIDs(self, tags, should_match): - """Make sure that the given tags have the correct IDs. - - This is used in tests that define a bunch of tags, each - containing a single string, and then select certain strings by - some mechanism. - """ - self.assertEqual([tag['id'] for tag in tags], should_match) - - -class TestFind(TreeTest): - """Basic tests of the find() method. - - find() just calls find_all() with limit=1, so it's not tested all - that thouroughly here. - """ - - def test_find_tag(self): - soup = self.soup("1234") - self.assertEqual(soup.find("b").string, "2") - - def test_unicode_text_find(self): - soup = self.soup('

Räksmörgås

') - self.assertEqual(soup.find(string='Räksmörgås'), 'Räksmörgås') - - def test_unicode_attribute_find(self): - soup = self.soup('

here it is

') - str(soup) - self.assertEqual("here it is", soup.find(id='Räksmörgås').text) - - - def test_find_everything(self): - """Test an optimization that finds all tags.""" - soup = self.soup("foobar") - self.assertEqual(2, len(soup.find_all())) - - def test_find_everything_with_name(self): - """Test an optimization that finds all tags with a given name.""" - soup = self.soup("foobarbaz") - self.assertEqual(2, len(soup.find_all('a'))) - -class TestFindAll(TreeTest): - """Basic tests of the find_all() method.""" - - def test_find_all_text_nodes(self): - """You can search the tree for text nodes.""" - soup = self.soup("Foobar\xbb") - # Exact match. - self.assertEqual(soup.find_all(string="bar"), ["bar"]) - self.assertEqual(soup.find_all(text="bar"), ["bar"]) - # Match any of a number of strings. - self.assertEqual( - soup.find_all(text=["Foo", "bar"]), ["Foo", "bar"]) - # Match a regular expression. - self.assertEqual(soup.find_all(text=re.compile('.*')), - ["Foo", "bar", '\xbb']) - # Match anything. - self.assertEqual(soup.find_all(text=True), - ["Foo", "bar", '\xbb']) - - def test_find_all_limit(self): - """You can limit the number of items returned by find_all.""" - soup = self.soup("12345") - self.assertSelects(soup.find_all('a', limit=3), ["1", "2", "3"]) - self.assertSelects(soup.find_all('a', limit=1), ["1"]) - self.assertSelects( - soup.find_all('a', limit=10), ["1", "2", "3", "4", "5"]) - - # A limit of 0 means no limit. - self.assertSelects( - soup.find_all('a', limit=0), ["1", "2", "3", "4", "5"]) - - def test_calling_a_tag_is_calling_findall(self): - soup = self.soup("123") - self.assertSelects(soup('a', limit=1), ["1"]) - self.assertSelects(soup.b(id="foo"), ["3"]) - - def test_find_all_with_self_referential_data_structure_does_not_cause_infinite_recursion(self): - soup = self.soup("") - # Create a self-referential list. - l = [] - l.append(l) - - # Without special code in _normalize_search_value, this would cause infinite - # recursion. - self.assertEqual([], soup.find_all(l)) - - def test_find_all_resultset(self): - """All find_all calls return a ResultSet""" - soup = self.soup("") - result = soup.find_all("a") - self.assertTrue(hasattr(result, "source")) - - result = soup.find_all(True) - self.assertTrue(hasattr(result, "source")) - - result = soup.find_all(text="foo") - self.assertTrue(hasattr(result, "source")) - - -class TestFindAllBasicNamespaces(TreeTest): - - def test_find_by_namespaced_name(self): - soup = self.soup('4') - self.assertEqual("4", soup.find("mathml:msqrt").string) - self.assertEqual("a", soup.find(attrs= { "svg:fill" : "red" }).name) - - -class TestFindAllByName(TreeTest): - """Test ways of finding tags by tag name.""" - - def setUp(self): - super(TreeTest, self).setUp() - self.tree = self.soup("""First tag. - Second tag. - Third Nested tag. tag.""") - - def test_find_all_by_tag_name(self): - # Find all the tags. - self.assertSelects( - self.tree.find_all('a'), ['First tag.', 'Nested tag.']) - - def test_find_all_by_name_and_text(self): - self.assertSelects( - self.tree.find_all('a', text='First tag.'), ['First tag.']) - - self.assertSelects( - self.tree.find_all('a', text=True), ['First tag.', 'Nested tag.']) - - self.assertSelects( - self.tree.find_all('a', text=re.compile("tag")), - ['First tag.', 'Nested tag.']) - - - def test_find_all_on_non_root_element(self): - # You can call find_all on any node, not just the root. - self.assertSelects(self.tree.c.find_all('a'), ['Nested tag.']) - - def test_calling_element_invokes_find_all(self): - self.assertSelects(self.tree('a'), ['First tag.', 'Nested tag.']) - - def test_find_all_by_tag_strainer(self): - self.assertSelects( - self.tree.find_all(SoupStrainer('a')), - ['First tag.', 'Nested tag.']) - - def test_find_all_by_tag_names(self): - self.assertSelects( - self.tree.find_all(['a', 'b']), - ['First tag.', 'Second tag.', 'Nested tag.']) - - def test_find_all_by_tag_dict(self): - self.assertSelects( - self.tree.find_all({'a' : True, 'b' : True}), - ['First tag.', 'Second tag.', 'Nested tag.']) - - def test_find_all_by_tag_re(self): - self.assertSelects( - self.tree.find_all(re.compile('^[ab]$')), - ['First tag.', 'Second tag.', 'Nested tag.']) - - def test_find_all_with_tags_matching_method(self): - # You can define an oracle method that determines whether - # a tag matches the search. - def id_matches_name(tag): - return tag.name == tag.get('id') - - tree = self.soup("""Match 1. - Does not match. - Match 2.""") - - self.assertSelects( - tree.find_all(id_matches_name), ["Match 1.", "Match 2."]) - - -class TestFindAllByAttribute(TreeTest): - - def test_find_all_by_attribute_name(self): - # You can pass in keyword arguments to find_all to search by - # attribute. - tree = self.soup(""" - Matching a. - - Non-matching Matching b.a. - """) - self.assertSelects(tree.find_all(id='first'), - ["Matching a.", "Matching b."]) - - def test_find_all_by_utf8_attribute_value(self): - peace = "םולש".encode("utf8") - data = ''.encode("utf8") - soup = self.soup(data) - self.assertEqual([soup.a], soup.find_all(title=peace)) - self.assertEqual([soup.a], soup.find_all(title=peace.decode("utf8"))) - self.assertEqual([soup.a], soup.find_all(title=[peace, "something else"])) - - def test_find_all_by_attribute_dict(self): - # You can pass in a dictionary as the argument 'attrs'. This - # lets you search for attributes like 'name' (a fixed argument - # to find_all) and 'class' (a reserved word in Python.) - tree = self.soup(""" - Name match. - Class match. - Non-match. - A tag called 'name1'. - """) - - # This doesn't do what you want. - self.assertSelects(tree.find_all(name='name1'), - ["A tag called 'name1'."]) - # This does what you want. - self.assertSelects(tree.find_all(attrs={'name' : 'name1'}), - ["Name match."]) - - self.assertSelects(tree.find_all(attrs={'class' : 'class2'}), - ["Class match."]) - - def test_find_all_by_class(self): - tree = self.soup(""" - Class 1. - Class 2. - Class 1. - Class 3 and 4. - """) - - # Passing in the class_ keyword argument will search against - # the 'class' attribute. - self.assertSelects(tree.find_all('a', class_='1'), ['Class 1.']) - self.assertSelects(tree.find_all('c', class_='3'), ['Class 3 and 4.']) - self.assertSelects(tree.find_all('c', class_='4'), ['Class 3 and 4.']) - - # Passing in a string to 'attrs' will also search the CSS class. - self.assertSelects(tree.find_all('a', '1'), ['Class 1.']) - self.assertSelects(tree.find_all(attrs='1'), ['Class 1.', 'Class 1.']) - self.assertSelects(tree.find_all('c', '3'), ['Class 3 and 4.']) - self.assertSelects(tree.find_all('c', '4'), ['Class 3 and 4.']) - - def test_find_by_class_when_multiple_classes_present(self): - tree = self.soup("Found it") - - f = tree.find_all("gar", class_=re.compile("o")) - self.assertSelects(f, ["Found it"]) - - f = tree.find_all("gar", class_=re.compile("a")) - self.assertSelects(f, ["Found it"]) - - # Since the class is not the string "foo bar", but the two - # strings "foo" and "bar", this will not find anything. - f = tree.find_all("gar", class_=re.compile("o b")) - self.assertSelects(f, []) - - def test_find_all_with_non_dictionary_for_attrs_finds_by_class(self): - soup = self.soup("Found it") - - self.assertSelects(soup.find_all("a", re.compile("ba")), ["Found it"]) - - def big_attribute_value(value): - return len(value) > 3 - - self.assertSelects(soup.find_all("a", big_attribute_value), []) - - def small_attribute_value(value): - return len(value) <= 3 - - self.assertSelects( - soup.find_all("a", small_attribute_value), ["Found it"]) - - def test_find_all_with_string_for_attrs_finds_multiple_classes(self): - soup = self.soup('') - a, a2 = soup.find_all("a") - self.assertEqual([a, a2], soup.find_all("a", "foo")) - self.assertEqual([a], soup.find_all("a", "bar")) - - # If you specify the class as a string that contains a - # space, only that specific value will be found. - self.assertEqual([a], soup.find_all("a", class_="foo bar")) - self.assertEqual([a], soup.find_all("a", "foo bar")) - self.assertEqual([], soup.find_all("a", "bar foo")) - - def test_find_all_by_attribute_soupstrainer(self): - tree = self.soup(""" - Match. - Non-match.""") - - strainer = SoupStrainer(attrs={'id' : 'first'}) - self.assertSelects(tree.find_all(strainer), ['Match.']) - - def test_find_all_with_missing_atribute(self): - # You can pass in None as the value of an attribute to find_all. - # This will match tags that do not have that attribute set. - tree = self.soup("""ID present. - No ID present. - ID is empty.""") - self.assertSelects(tree.find_all('a', id=None), ["No ID present."]) - - def test_find_all_with_defined_attribute(self): - # You can pass in None as the value of an attribute to find_all. - # This will match tags that have that attribute set to any value. - tree = self.soup("""ID present. - No ID present. - ID is empty.""") - self.assertSelects( - tree.find_all(id=True), ["ID present.", "ID is empty."]) - - def test_find_all_with_numeric_attribute(self): - # If you search for a number, it's treated as a string. - tree = self.soup("""Unquoted attribute. - Quoted attribute.""") - - expected = ["Unquoted attribute.", "Quoted attribute."] - self.assertSelects(tree.find_all(id=1), expected) - self.assertSelects(tree.find_all(id="1"), expected) - - def test_find_all_with_list_attribute_values(self): - # You can pass a list of attribute values instead of just one, - # and you'll get tags that match any of the values. - tree = self.soup("""1 - 2 - 3 - No ID.""") - self.assertSelects(tree.find_all(id=["1", "3", "4"]), - ["1", "3"]) - - def test_find_all_with_regular_expression_attribute_value(self): - # You can pass a regular expression as an attribute value, and - # you'll get tags whose values for that attribute match the - # regular expression. - tree = self.soup("""One a. - Two as. - Mixed as and bs. - One b. - No ID.""") - - self.assertSelects(tree.find_all(id=re.compile("^a+$")), - ["One a.", "Two as."]) - - def test_find_by_name_and_containing_string(self): - soup = self.soup("foobarfoo") - a = soup.a - - self.assertEqual([a], soup.find_all("a", text="foo")) - self.assertEqual([], soup.find_all("a", text="bar")) - self.assertEqual([], soup.find_all("a", text="bar")) - - def test_find_by_name_and_containing_string_when_string_is_buried(self): - soup = self.soup("foofoo") - self.assertEqual(soup.find_all("a"), soup.find_all("a", text="foo")) - - def test_find_by_attribute_and_containing_string(self): - soup = self.soup('foofoo') - a = soup.a - - self.assertEqual([a], soup.find_all(id=2, text="foo")) - self.assertEqual([], soup.find_all(id=1, text="bar")) - - - - -class TestIndex(TreeTest): - """Test Tag.index""" - def test_index(self): - tree = self.soup("""
- Identical - Not identical - Identical - - Identical with child - Also not identical - Identical with child -
""") - div = tree.div - for i, element in enumerate(div.contents): - self.assertEqual(i, div.index(element)) - self.assertRaises(ValueError, tree.index, 1) - - -class TestParentOperations(TreeTest): - """Test navigation and searching through an element's parents.""" - - def setUp(self): - super(TestParentOperations, self).setUp() - self.tree = self.soup('''
    -
      -
        -
          - Start here -
        -
      ''') - self.start = self.tree.b - - - def test_parent(self): - self.assertEqual(self.start.parent['id'], 'bottom') - self.assertEqual(self.start.parent.parent['id'], 'middle') - self.assertEqual(self.start.parent.parent.parent['id'], 'top') - - def test_parent_of_top_tag_is_soup_object(self): - top_tag = self.tree.contents[0] - self.assertEqual(top_tag.parent, self.tree) - - def test_soup_object_has_no_parent(self): - self.assertEqual(None, self.tree.parent) - - def test_find_parents(self): - self.assertSelectsIDs( - self.start.find_parents('ul'), ['bottom', 'middle', 'top']) - self.assertSelectsIDs( - self.start.find_parents('ul', id="middle"), ['middle']) - - def test_find_parent(self): - self.assertEqual(self.start.find_parent('ul')['id'], 'bottom') - self.assertEqual(self.start.find_parent('ul', id='top')['id'], 'top') - - def test_parent_of_text_element(self): - text = self.tree.find(text="Start here") - self.assertEqual(text.parent.name, 'b') - - def test_text_element_find_parent(self): - text = self.tree.find(text="Start here") - self.assertEqual(text.find_parent('ul')['id'], 'bottom') - - def test_parent_generator(self): - parents = [parent['id'] for parent in self.start.parents - if parent is not None and 'id' in parent.attrs] - self.assertEqual(parents, ['bottom', 'middle', 'top']) - - -class ProximityTest(TreeTest): - - def setUp(self): - super(TreeTest, self).setUp() - self.tree = self.soup( - 'OneTwoThree') - - -class TestNextOperations(ProximityTest): - - def setUp(self): - super(TestNextOperations, self).setUp() - self.start = self.tree.b - - def test_next(self): - self.assertEqual(self.start.next_element, "One") - self.assertEqual(self.start.next_element.next_element['id'], "2") - - def test_next_of_last_item_is_none(self): - last = self.tree.find(text="Three") - self.assertEqual(last.next_element, None) - - def test_next_of_root_is_none(self): - # The document root is outside the next/previous chain. - self.assertEqual(self.tree.next_element, None) - - def test_find_all_next(self): - self.assertSelects(self.start.find_all_next('b'), ["Two", "Three"]) - self.start.find_all_next(id=3) - self.assertSelects(self.start.find_all_next(id=3), ["Three"]) - - def test_find_next(self): - self.assertEqual(self.start.find_next('b')['id'], '2') - self.assertEqual(self.start.find_next(text="Three"), "Three") - - def test_find_next_for_text_element(self): - text = self.tree.find(text="One") - self.assertEqual(text.find_next("b").string, "Two") - self.assertSelects(text.find_all_next("b"), ["Two", "Three"]) - - def test_next_generator(self): - start = self.tree.find(text="Two") - successors = [node for node in start.next_elements] - # There are two successors: the final tag and its text contents. - tag, contents = successors - self.assertEqual(tag['id'], '3') - self.assertEqual(contents, "Three") - -class TestPreviousOperations(ProximityTest): - - def setUp(self): - super(TestPreviousOperations, self).setUp() - self.end = self.tree.find(text="Three") - - def test_previous(self): - self.assertEqual(self.end.previous_element['id'], "3") - self.assertEqual(self.end.previous_element.previous_element, "Two") - - def test_previous_of_first_item_is_none(self): - first = self.tree.find('html') - self.assertEqual(first.previous_element, None) - - def test_previous_of_root_is_none(self): - # The document root is outside the next/previous chain. - # XXX This is broken! - #self.assertEqual(self.tree.previous_element, None) - pass - - def test_find_all_previous(self): - # The tag containing the "Three" node is the predecessor - # of the "Three" node itself, which is why "Three" shows up - # here. - self.assertSelects( - self.end.find_all_previous('b'), ["Three", "Two", "One"]) - self.assertSelects(self.end.find_all_previous(id=1), ["One"]) - - def test_find_previous(self): - self.assertEqual(self.end.find_previous('b')['id'], '3') - self.assertEqual(self.end.find_previous(text="One"), "One") - - def test_find_previous_for_text_element(self): - text = self.tree.find(text="Three") - self.assertEqual(text.find_previous("b").string, "Three") - self.assertSelects( - text.find_all_previous("b"), ["Three", "Two", "One"]) - - def test_previous_generator(self): - start = self.tree.find(text="One") - predecessors = [node for node in start.previous_elements] - - # There are four predecessors: the tag containing "One" - # the tag, the tag, and the tag. - b, body, head, html = predecessors - self.assertEqual(b['id'], '1') - self.assertEqual(body.name, "body") - self.assertEqual(head.name, "head") - self.assertEqual(html.name, "html") - - -class SiblingTest(TreeTest): - - def setUp(self): - super(SiblingTest, self).setUp() - markup = ''' - - - - - - - - - - - ''' - # All that whitespace looks good but makes the tests more - # difficult. Get rid of it. - markup = re.compile(r"\n\s*").sub("", markup) - self.tree = self.soup(markup) - - -class TestNextSibling(SiblingTest): - - def setUp(self): - super(TestNextSibling, self).setUp() - self.start = self.tree.find(id="1") - - def test_next_sibling_of_root_is_none(self): - self.assertEqual(self.tree.next_sibling, None) - - def test_next_sibling(self): - self.assertEqual(self.start.next_sibling['id'], '2') - self.assertEqual(self.start.next_sibling.next_sibling['id'], '3') - - # Note the difference between next_sibling and next_element. - self.assertEqual(self.start.next_element['id'], '1.1') - - def test_next_sibling_may_not_exist(self): - self.assertEqual(self.tree.html.next_sibling, None) - - nested_span = self.tree.find(id="1.1") - self.assertEqual(nested_span.next_sibling, None) - - last_span = self.tree.find(id="4") - self.assertEqual(last_span.next_sibling, None) - - def test_find_next_sibling(self): - self.assertEqual(self.start.find_next_sibling('span')['id'], '2') - - def test_next_siblings(self): - self.assertSelectsIDs(self.start.find_next_siblings("span"), - ['2', '3', '4']) - - self.assertSelectsIDs(self.start.find_next_siblings(id='3'), ['3']) - - def test_next_sibling_for_text_element(self): - soup = self.soup("Foobarbaz") - start = soup.find(text="Foo") - self.assertEqual(start.next_sibling.name, 'b') - self.assertEqual(start.next_sibling.next_sibling, 'baz') - - self.assertSelects(start.find_next_siblings('b'), ['bar']) - self.assertEqual(start.find_next_sibling(text="baz"), "baz") - self.assertEqual(start.find_next_sibling(text="nonesuch"), None) - - -class TestPreviousSibling(SiblingTest): - - def setUp(self): - super(TestPreviousSibling, self).setUp() - self.end = self.tree.find(id="4") - - def test_previous_sibling_of_root_is_none(self): - self.assertEqual(self.tree.previous_sibling, None) - - def test_previous_sibling(self): - self.assertEqual(self.end.previous_sibling['id'], '3') - self.assertEqual(self.end.previous_sibling.previous_sibling['id'], '2') - - # Note the difference between previous_sibling and previous_element. - self.assertEqual(self.end.previous_element['id'], '3.1') - - def test_previous_sibling_may_not_exist(self): - self.assertEqual(self.tree.html.previous_sibling, None) - - nested_span = self.tree.find(id="1.1") - self.assertEqual(nested_span.previous_sibling, None) - - first_span = self.tree.find(id="1") - self.assertEqual(first_span.previous_sibling, None) - - def test_find_previous_sibling(self): - self.assertEqual(self.end.find_previous_sibling('span')['id'], '3') - - def test_previous_siblings(self): - self.assertSelectsIDs(self.end.find_previous_siblings("span"), - ['3', '2', '1']) - - self.assertSelectsIDs(self.end.find_previous_siblings(id='1'), ['1']) - - def test_previous_sibling_for_text_element(self): - soup = self.soup("Foobarbaz") - start = soup.find(text="baz") - self.assertEqual(start.previous_sibling.name, 'b') - self.assertEqual(start.previous_sibling.previous_sibling, 'Foo') - - self.assertSelects(start.find_previous_siblings('b'), ['bar']) - self.assertEqual(start.find_previous_sibling(text="Foo"), "Foo") - self.assertEqual(start.find_previous_sibling(text="nonesuch"), None) - - -class TestTagCreation(SoupTest): - """Test the ability to create new tags.""" - def test_new_tag(self): - soup = self.soup("") - new_tag = soup.new_tag("foo", bar="baz") - self.assertTrue(isinstance(new_tag, Tag)) - self.assertEqual("foo", new_tag.name) - self.assertEqual(dict(bar="baz"), new_tag.attrs) - self.assertEqual(None, new_tag.parent) - - def test_tag_inherits_self_closing_rules_from_builder(self): - if XML_BUILDER_PRESENT: - xml_soup = BeautifulSoup("", "lxml-xml") - xml_br = xml_soup.new_tag("br") - xml_p = xml_soup.new_tag("p") - - # Both the
      and

      tag are empty-element, just because - # they have no contents. - self.assertEqual(b"
      ", xml_br.encode()) - self.assertEqual(b"

      ", xml_p.encode()) - - html_soup = BeautifulSoup("", "html.parser") - html_br = html_soup.new_tag("br") - html_p = html_soup.new_tag("p") - - # The HTML builder users HTML's rules about which tags are - # empty-element tags, and the new tags reflect these rules. - self.assertEqual(b"
      ", html_br.encode()) - self.assertEqual(b"

      ", html_p.encode()) - - def test_new_string_creates_navigablestring(self): - soup = self.soup("") - s = soup.new_string("foo") - self.assertEqual("foo", s) - self.assertTrue(isinstance(s, NavigableString)) - - def test_new_string_can_create_navigablestring_subclass(self): - soup = self.soup("") - s = soup.new_string("foo", Comment) - self.assertEqual("foo", s) - self.assertTrue(isinstance(s, Comment)) - -class TestTreeModification(SoupTest): - - def test_attribute_modification(self): - soup = self.soup('') - soup.a['id'] = 2 - self.assertEqual(soup.decode(), self.document_for('')) - del(soup.a['id']) - self.assertEqual(soup.decode(), self.document_for('')) - soup.a['id2'] = 'foo' - self.assertEqual(soup.decode(), self.document_for('')) - - def test_new_tag_creation(self): - builder = builder_registry.lookup('html')() - soup = self.soup("", builder=builder) - a = Tag(soup, builder, 'a') - ol = Tag(soup, builder, 'ol') - a['href'] = 'http://foo.com/' - soup.body.insert(0, a) - soup.body.insert(1, ol) - self.assertEqual( - soup.body.encode(), - b'
        ') - - def test_append_to_contents_moves_tag(self): - doc = """

        Don't leave me here.

        -

        Don\'t leave!

        """ - soup = self.soup(doc) - second_para = soup.find(id='2') - bold = soup.b - - # Move the tag to the end of the second paragraph. - soup.find(id='2').append(soup.b) - - # The tag is now a child of the second paragraph. - self.assertEqual(bold.parent, second_para) - - self.assertEqual( - soup.decode(), self.document_for( - '

        Don\'t leave me .

        \n' - '

        Don\'t leave!here

        ')) - - def test_replace_with_returns_thing_that_was_replaced(self): - text = "" - soup = self.soup(text) - a = soup.a - new_a = a.replace_with(soup.c) - self.assertEqual(a, new_a) - - def test_unwrap_returns_thing_that_was_replaced(self): - text = "" - soup = self.soup(text) - a = soup.a - new_a = a.unwrap() - self.assertEqual(a, new_a) - - def test_replace_with_and_unwrap_give_useful_exception_when_tag_has_no_parent(self): - soup = self.soup("FooBar") - a = soup.a - a.extract() - self.assertEqual(None, a.parent) - self.assertRaises(ValueError, a.unwrap) - self.assertRaises(ValueError, a.replace_with, soup.c) - - def test_replace_tag_with_itself(self): - text = "Foo" - soup = self.soup(text) - c = soup.c - soup.c.replace_with(c) - self.assertEqual(soup.decode(), self.document_for(text)) - - def test_replace_tag_with_its_parent_raises_exception(self): - text = "" - soup = self.soup(text) - self.assertRaises(ValueError, soup.b.replace_with, soup.a) - - def test_insert_tag_into_itself_raises_exception(self): - text = "" - soup = self.soup(text) - self.assertRaises(ValueError, soup.a.insert, 0, soup.a) - - def test_replace_with_maintains_next_element_throughout(self): - soup = self.soup('

        onethree

        ') - a = soup.a - b = a.contents[0] - # Make it so the tag has two text children. - a.insert(1, "two") - - # Now replace each one with the empty string. - left, right = a.contents - left.replaceWith('') - right.replaceWith('') - - # The tag is still connected to the tree. - self.assertEqual("three", soup.b.string) - - def test_replace_final_node(self): - soup = self.soup("Argh!") - soup.find(text="Argh!").replace_with("Hooray!") - new_text = soup.find(text="Hooray!") - b = soup.b - self.assertEqual(new_text.previous_element, b) - self.assertEqual(new_text.parent, b) - self.assertEqual(new_text.previous_element.next_element, new_text) - self.assertEqual(new_text.next_element, None) - - def test_consecutive_text_nodes(self): - # A builder should never create two consecutive text nodes, - # but if you insert one next to another, Beautiful Soup will - # handle it correctly. - soup = self.soup("Argh!") - soup.b.insert(1, "Hooray!") - - self.assertEqual( - soup.decode(), self.document_for( - "Argh!Hooray!")) - - new_text = soup.find(text="Hooray!") - self.assertEqual(new_text.previous_element, "Argh!") - self.assertEqual(new_text.previous_element.next_element, new_text) - - self.assertEqual(new_text.previous_sibling, "Argh!") - self.assertEqual(new_text.previous_sibling.next_sibling, new_text) - - self.assertEqual(new_text.next_sibling, None) - self.assertEqual(new_text.next_element, soup.c) - - def test_insert_string(self): - soup = self.soup("") - soup.a.insert(0, "bar") - soup.a.insert(0, "foo") - # The string were added to the tag. - self.assertEqual(["foo", "bar"], soup.a.contents) - # And they were converted to NavigableStrings. - self.assertEqual(soup.a.contents[0].next_element, "bar") - - def test_insert_tag(self): - builder = self.default_builder - soup = self.soup( - "Findlady!", builder=builder) - magic_tag = Tag(soup, builder, 'magictag') - magic_tag.insert(0, "the") - soup.a.insert(1, magic_tag) - - self.assertEqual( - soup.decode(), self.document_for( - "Findthelady!")) - - # Make sure all the relationships are hooked up correctly. - b_tag = soup.b - self.assertEqual(b_tag.next_sibling, magic_tag) - self.assertEqual(magic_tag.previous_sibling, b_tag) - - find = b_tag.find(text="Find") - self.assertEqual(find.next_element, magic_tag) - self.assertEqual(magic_tag.previous_element, find) - - c_tag = soup.c - self.assertEqual(magic_tag.next_sibling, c_tag) - self.assertEqual(c_tag.previous_sibling, magic_tag) - - the = magic_tag.find(text="the") - self.assertEqual(the.parent, magic_tag) - self.assertEqual(the.next_element, c_tag) - self.assertEqual(c_tag.previous_element, the) - - def test_append_child_thats_already_at_the_end(self): - data = "" - soup = self.soup(data) - soup.a.append(soup.b) - self.assertEqual(data, soup.decode()) - - def test_move_tag_to_beginning_of_parent(self): - data = "" - soup = self.soup(data) - soup.a.insert(0, soup.d) - self.assertEqual("", soup.decode()) - - def test_insert_works_on_empty_element_tag(self): - # This is a little strange, since most HTML parsers don't allow - # markup like this to come through. But in general, we don't - # know what the parser would or wouldn't have allowed, so - # I'm letting this succeed for now. - soup = self.soup("
        ") - soup.br.insert(1, "Contents") - self.assertEqual(str(soup.br), "
        Contents
        ") - - def test_insert_before(self): - soup = self.soup("foobar") - soup.b.insert_before("BAZ") - soup.a.insert_before("QUUX") - self.assertEqual( - soup.decode(), self.document_for("QUUXfooBAZbar")) - - soup.a.insert_before(soup.b) - self.assertEqual( - soup.decode(), self.document_for("QUUXbarfooBAZ")) - - def test_insert_after(self): - soup = self.soup("foobar") - soup.b.insert_after("BAZ") - soup.a.insert_after("QUUX") - self.assertEqual( - soup.decode(), self.document_for("fooQUUXbarBAZ")) - soup.b.insert_after(soup.a) - self.assertEqual( - soup.decode(), self.document_for("QUUXbarfooBAZ")) - - def test_insert_after_raises_exception_if_after_has_no_meaning(self): - soup = self.soup("") - tag = soup.new_tag("a") - string = soup.new_string("") - self.assertRaises(ValueError, string.insert_after, tag) - self.assertRaises(NotImplementedError, soup.insert_after, tag) - self.assertRaises(ValueError, tag.insert_after, tag) - - def test_insert_before_raises_notimplementederror_if_before_has_no_meaning(self): - soup = self.soup("") - tag = soup.new_tag("a") - string = soup.new_string("") - self.assertRaises(ValueError, string.insert_before, tag) - self.assertRaises(NotImplementedError, soup.insert_before, tag) - self.assertRaises(ValueError, tag.insert_before, tag) - - def test_replace_with(self): - soup = self.soup( - "

        There's no business like show business

        ") - no, show = soup.find_all('b') - show.replace_with(no) - self.assertEqual( - soup.decode(), - self.document_for( - "

        There's business like no business

        ")) - - self.assertEqual(show.parent, None) - self.assertEqual(no.parent, soup.p) - self.assertEqual(no.next_element, "no") - self.assertEqual(no.next_sibling, " business") - - def test_replace_first_child(self): - data = "" - soup = self.soup(data) - soup.b.replace_with(soup.c) - self.assertEqual("", soup.decode()) - - def test_replace_last_child(self): - data = "" - soup = self.soup(data) - soup.c.replace_with(soup.b) - self.assertEqual("", soup.decode()) - - def test_nested_tag_replace_with(self): - soup = self.soup( - """Wereservetherighttorefuseservice""") - - # Replace the entire tag and its contents ("reserve the - # right") with the tag ("refuse"). - remove_tag = soup.b - move_tag = soup.f - remove_tag.replace_with(move_tag) - - self.assertEqual( - soup.decode(), self.document_for( - "Werefusetoservice")) - - # The tag is now an orphan. - self.assertEqual(remove_tag.parent, None) - self.assertEqual(remove_tag.find(text="right").next_element, None) - self.assertEqual(remove_tag.previous_element, None) - self.assertEqual(remove_tag.next_sibling, None) - self.assertEqual(remove_tag.previous_sibling, None) - - # The tag is now connected to the tag. - self.assertEqual(move_tag.parent, soup.a) - self.assertEqual(move_tag.previous_element, "We") - self.assertEqual(move_tag.next_element.next_element, soup.e) - self.assertEqual(move_tag.next_sibling, None) - - # The gap where the tag used to be has been mended, and - # the word "to" is now connected to the tag. - to_text = soup.find(text="to") - g_tag = soup.g - self.assertEqual(to_text.next_element, g_tag) - self.assertEqual(to_text.next_sibling, g_tag) - self.assertEqual(g_tag.previous_element, to_text) - self.assertEqual(g_tag.previous_sibling, to_text) - - def test_unwrap(self): - tree = self.soup(""" -

        Unneeded formatting is unneeded

        - """) - tree.em.unwrap() - self.assertEqual(tree.em, None) - self.assertEqual(tree.p.text, "Unneeded formatting is unneeded") - - def test_wrap(self): - soup = self.soup("I wish I was bold.") - value = soup.string.wrap(soup.new_tag("b")) - self.assertEqual(value.decode(), "I wish I was bold.") - self.assertEqual( - soup.decode(), self.document_for("I wish I was bold.")) - - def test_wrap_extracts_tag_from_elsewhere(self): - soup = self.soup("I wish I was bold.") - soup.b.next_sibling.wrap(soup.b) - self.assertEqual( - soup.decode(), self.document_for("I wish I was bold.")) - - def test_wrap_puts_new_contents_at_the_end(self): - soup = self.soup("I like being bold.I wish I was bold.") - soup.b.next_sibling.wrap(soup.b) - self.assertEqual(2, len(soup.b.contents)) - self.assertEqual( - soup.decode(), self.document_for( - "I like being bold.I wish I was bold.")) - - def test_extract(self): - soup = self.soup( - 'Some content. More content.') - - self.assertEqual(len(soup.body.contents), 3) - extracted = soup.find(id="nav").extract() - - self.assertEqual( - soup.decode(), "Some content. More content.") - self.assertEqual(extracted.decode(), '') - - # The extracted tag is now an orphan. - self.assertEqual(len(soup.body.contents), 2) - self.assertEqual(extracted.parent, None) - self.assertEqual(extracted.previous_element, None) - self.assertEqual(extracted.next_element.next_element, None) - - # The gap where the extracted tag used to be has been mended. - content_1 = soup.find(text="Some content. ") - content_2 = soup.find(text=" More content.") - self.assertEqual(content_1.next_element, content_2) - self.assertEqual(content_1.next_sibling, content_2) - self.assertEqual(content_2.previous_element, content_1) - self.assertEqual(content_2.previous_sibling, content_1) - - def test_extract_distinguishes_between_identical_strings(self): - soup = self.soup("
        foobar") - foo_1 = soup.a.string - bar_1 = soup.b.string - foo_2 = soup.new_string("foo") - bar_2 = soup.new_string("bar") - soup.a.append(foo_2) - soup.b.append(bar_2) - - # Now there are two identical strings in the tag, and two - # in the tag. Let's remove the first "foo" and the second - # "bar". - foo_1.extract() - bar_2.extract() - self.assertEqual(foo_2, soup.a.string) - self.assertEqual(bar_2, soup.b.string) - - def test_extract_multiples_of_same_tag(self): - soup = self.soup(""" - - - - - - - - - -""") - [soup.script.extract() for i in soup.find_all("script")] - self.assertEqual("\n\n\n", str(soup.body)) - - - def test_extract_works_when_element_is_surrounded_by_identical_strings(self): - soup = self.soup( - '\n' - 'hi\n' - '') - soup.find('body').extract() - self.assertEqual(None, soup.find('body')) - - - def test_clear(self): - """Tag.clear()""" - soup = self.soup("

        String Italicized and another

        ") - # clear using extract() - a = soup.a - soup.p.clear() - self.assertEqual(len(soup.p.contents), 0) - self.assertTrue(hasattr(a, "contents")) - - # clear using decompose() - em = a.em - a.clear(decompose=True) - self.assertEqual(0, len(em.contents)) - - def test_string_set(self): - """Tag.string = 'string'""" - soup = self.soup(" ") - soup.a.string = "foo" - self.assertEqual(soup.a.contents, ["foo"]) - soup.b.string = "bar" - self.assertEqual(soup.b.contents, ["bar"]) - - def test_string_set_does_not_affect_original_string(self): - soup = self.soup("foobar") - soup.b.string = soup.c.string - self.assertEqual(soup.a.encode(), b"barbar") - - def test_set_string_preserves_class_of_string(self): - soup = self.soup("") - cdata = CData("foo") - soup.a.string = cdata - self.assertTrue(isinstance(soup.a.string, CData)) - -class TestElementObjects(SoupTest): - """Test various features of element objects.""" - - def test_len(self): - """The length of an element is its number of children.""" - soup = self.soup("123") - - # The BeautifulSoup object itself contains one element: the - # tag. - self.assertEqual(len(soup.contents), 1) - self.assertEqual(len(soup), 1) - - # The tag contains three elements: the text node "1", the - # tag, and the text node "3". - self.assertEqual(len(soup.top), 3) - self.assertEqual(len(soup.top.contents), 3) - - def test_member_access_invokes_find(self): - """Accessing a Python member .foo invokes find('foo')""" - soup = self.soup('') - self.assertEqual(soup.b, soup.find('b')) - self.assertEqual(soup.b.i, soup.find('b').find('i')) - self.assertEqual(soup.a, None) - - def test_deprecated_member_access(self): - soup = self.soup('') - with warnings.catch_warnings(record=True) as w: - tag = soup.bTag - self.assertEqual(soup.b, tag) - self.assertEqual( - '.bTag is deprecated, use .find("b") instead.', - str(w[0].message)) - - def test_has_attr(self): - """has_attr() checks for the presence of an attribute. - - Please note note: has_attr() is different from - __in__. has_attr() checks the tag's attributes and __in__ - checks the tag's chidlren. - """ - soup = self.soup("") - self.assertTrue(soup.foo.has_attr('attr')) - self.assertFalse(soup.foo.has_attr('attr2')) - - - def test_attributes_come_out_in_alphabetical_order(self): - markup = '' - self.assertSoupEquals(markup, '') - - def test_string(self): - # A tag that contains only a text node makes that node - # available as .string. - soup = self.soup("foo") - self.assertEqual(soup.b.string, 'foo') - - def test_empty_tag_has_no_string(self): - # A tag with no children has no .stirng. - soup = self.soup("") - self.assertEqual(soup.b.string, None) - - def test_tag_with_multiple_children_has_no_string(self): - # A tag with no children has no .string. - soup = self.soup("foo") - self.assertEqual(soup.b.string, None) - - soup = self.soup("foobar
        ") - self.assertEqual(soup.b.string, None) - - # Even if all the children are strings, due to trickery, - # it won't work--but this would be a good optimization. - soup = self.soup("foo
        ") - soup.a.insert(1, "bar") - self.assertEqual(soup.a.string, None) - - def test_tag_with_recursive_string_has_string(self): - # A tag with a single child which has a .string inherits that - # .string. - soup = self.soup("foo") - self.assertEqual(soup.a.string, "foo") - self.assertEqual(soup.string, "foo") - - def test_lack_of_string(self): - """Only a tag containing a single text node has a .string.""" - soup = self.soup("feo") - self.assertFalse(soup.b.string) - - soup = self.soup("") - self.assertFalse(soup.b.string) - - def test_all_text(self): - """Tag.text and Tag.get_text(sep=u"") -> all child text, concatenated""" - soup = self.soup("ar t ") - self.assertEqual(soup.a.text, "ar t ") - self.assertEqual(soup.a.get_text(strip=True), "art") - self.assertEqual(soup.a.get_text(","), "a,r, , t ") - self.assertEqual(soup.a.get_text(",", strip=True), "a,r,t") - - def test_get_text_ignores_comments(self): - soup = self.soup("foobar") - self.assertEqual(soup.get_text(), "foobar") - - self.assertEqual( - soup.get_text(types=(NavigableString, Comment)), "fooIGNOREbar") - self.assertEqual( - soup.get_text(types=None), "fooIGNOREbar") - - def test_all_strings_ignores_comments(self): - soup = self.soup("foobar") - self.assertEqual(['foo', 'bar'], list(soup.strings)) - -class TestCDAtaListAttributes(SoupTest): - - """Testing cdata-list attributes like 'class'. - """ - def test_single_value_becomes_list(self): - soup = self.soup("") - self.assertEqual(["foo"],soup.a['class']) - - def test_multiple_values_becomes_list(self): - soup = self.soup("") - self.assertEqual(["foo", "bar"], soup.a['class']) - - def test_multiple_values_separated_by_weird_whitespace(self): - soup = self.soup("") - self.assertEqual(["foo", "bar", "baz"],soup.a['class']) - - def test_attributes_joined_into_string_on_output(self): - soup = self.soup("") - self.assertEqual(b'', soup.a.encode()) - - def test_accept_charset(self): - soup = self.soup('
        ') - self.assertEqual(['ISO-8859-1', 'UTF-8'], soup.form['accept-charset']) - - def test_cdata_attribute_applying_only_to_one_tag(self): - data = '' - soup = self.soup(data) - # We saw in another test that accept-charset is a cdata-list - # attribute for the tag. But it's not a cdata-list - # attribute for any other tag. - self.assertEqual('ISO-8859-1 UTF-8', soup.a['accept-charset']) - - def test_string_has_immutable_name_property(self): - string = self.soup("s").string - self.assertEqual(None, string.name) - def t(): - string.name = 'foo' - self.assertRaises(AttributeError, t) - -class TestPersistence(SoupTest): - "Testing features like pickle and deepcopy." - - def setUp(self): - super(TestPersistence, self).setUp() - self.page = """ - - - -Beautiful Soup: We called him Tortoise because he taught us. - - - - - - -foo -bar - -""" - self.tree = self.soup(self.page) - - def test_pickle_and_unpickle_identity(self): - # Pickling a tree, then unpickling it, yields a tree identical - # to the original. - dumped = pickle.dumps(self.tree, 2) - loaded = pickle.loads(dumped) - self.assertEqual(loaded.__class__, BeautifulSoup) - self.assertEqual(loaded.decode(), self.tree.decode()) - - def test_deepcopy_identity(self): - # Making a deepcopy of a tree yields an identical tree. - copied = copy.deepcopy(self.tree) - self.assertEqual(copied.decode(), self.tree.decode()) - - def test_unicode_pickle(self): - # A tree containing Unicode characters can be pickled. - html = "\N{SNOWMAN}" - soup = self.soup(html) - dumped = pickle.dumps(soup, pickle.HIGHEST_PROTOCOL) - loaded = pickle.loads(dumped) - self.assertEqual(loaded.decode(), soup.decode()) - - def test_copy_navigablestring_is_not_attached_to_tree(self): - html = "FooBar" - soup = self.soup(html) - s1 = soup.find(string="Foo") - s2 = copy.copy(s1) - self.assertEqual(s1, s2) - self.assertEqual(None, s2.parent) - self.assertEqual(None, s2.next_element) - self.assertNotEqual(None, s1.next_sibling) - self.assertEqual(None, s2.next_sibling) - self.assertEqual(None, s2.previous_element) - - def test_copy_navigablestring_subclass_has_same_type(self): - html = "" - soup = self.soup(html) - s1 = soup.string - s2 = copy.copy(s1) - self.assertEqual(s1, s2) - self.assertTrue(isinstance(s2, Comment)) - - def test_copy_entire_soup(self): - html = "
        FooBar
        end" - soup = self.soup(html) - soup_copy = copy.copy(soup) - self.assertEqual(soup, soup_copy) - - def test_copy_tag_copies_contents(self): - html = "
        FooBar
        end" - soup = self.soup(html) - div = soup.div - div_copy = copy.copy(div) - - # The two tags look the same, and evaluate to equal. - self.assertEqual(str(div), str(div_copy)) - self.assertEqual(div, div_copy) - - # But they're not the same object. - self.assertFalse(div is div_copy) - - # And they don't have the same relation to the parse tree. The - # copy is not associated with a parse tree at all. - self.assertEqual(None, div_copy.parent) - self.assertEqual(None, div_copy.previous_element) - self.assertEqual(None, div_copy.find(string='Bar').next_element) - self.assertNotEqual(None, div.find(string='Bar').next_element) - -class TestSubstitutions(SoupTest): - - def test_default_formatter_is_minimal(self): - markup = "<<Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>>" - soup = self.soup(markup) - decoded = soup.decode(formatter="minimal") - # The < is converted back into < but the e-with-acute is left alone. - self.assertEqual( - decoded, - self.document_for( - "<<Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>>")) - - def test_formatter_html(self): - markup = "<<Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>>" - soup = self.soup(markup) - decoded = soup.decode(formatter="html") - self.assertEqual( - decoded, - self.document_for("<<Sacré bleu!>>")) - - def test_formatter_minimal(self): - markup = "<<Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>>" - soup = self.soup(markup) - decoded = soup.decode(formatter="minimal") - # The < is converted back into < but the e-with-acute is left alone. - self.assertEqual( - decoded, - self.document_for( - "<<Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>>")) - - def test_formatter_null(self): - markup = "<<Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>>" - soup = self.soup(markup) - decoded = soup.decode(formatter=None) - # Neither the angle brackets nor the e-with-acute are converted. - # This is not valid HTML, but it's what the user wanted. - self.assertEqual(decoded, - self.document_for("<>")) - - def test_formatter_custom(self): - markup = "<foo>bar" - soup = self.soup(markup) - decoded = soup.decode(formatter = lambda x: x.upper()) - # Instead of normal entity conversion code, the custom - # callable is called on every string. - self.assertEqual( - decoded, - self.document_for("BAR")) - - def test_formatter_is_run_on_attribute_values(self): - markup = 'e' - soup = self.soup(markup) - a = soup.a - - expect_minimal = 'e' - - self.assertEqual(expect_minimal, a.decode()) - self.assertEqual(expect_minimal, a.decode(formatter="minimal")) - - expect_html = 'e' - self.assertEqual(expect_html, a.decode(formatter="html")) - - self.assertEqual(markup, a.decode(formatter=None)) - expect_upper = 'E' - self.assertEqual(expect_upper, a.decode(formatter=lambda x: x.upper())) - - def test_formatter_skips_script_tag_for_html_documents(self): - doc = """ - -""" - encoded = BeautifulSoup(doc, 'html.parser').encode() - self.assertTrue(b"< < hey > >" in encoded) - - def test_formatter_skips_style_tag_for_html_documents(self): - doc = """ - -""" - encoded = BeautifulSoup(doc, 'html.parser').encode() - self.assertTrue(b"< < hey > >" in encoded) - - def test_prettify_leaves_preformatted_text_alone(self): - soup = self.soup("
        foo
          \tbar\n  \n  
        baz ") - # Everything outside the
         tag is reformatted, but everything
        -        # inside is left alone.
        -        self.assertEqual(
        -            '
        \n foo\n
          \tbar\n  \n  
        \n baz\n
        ', - soup.div.prettify()) - - def test_prettify_accepts_formatter(self): - soup = BeautifulSoup("foo", 'html.parser') - pretty = soup.prettify(formatter = lambda x: x.upper()) - self.assertTrue("FOO" in pretty) - - def test_prettify_outputs_unicode_by_default(self): - soup = self.soup("") - self.assertEqual(str, type(soup.prettify())) - - def test_prettify_can_encode_data(self): - soup = self.soup("") - self.assertEqual(bytes, type(soup.prettify("utf-8"))) - - def test_html_entity_substitution_off_by_default(self): - markup = "Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!" - soup = self.soup(markup) - encoded = soup.b.encode("utf-8") - self.assertEqual(encoded, markup.encode('utf-8')) - - def test_encoding_substitution(self): - # Here's the tag saying that a document is - # encoded in Shift-JIS. - meta_tag = ('') - soup = self.soup(meta_tag) - - # Parse the document, and the charset apprears unchanged. - self.assertEqual(soup.meta['content'], 'text/html; charset=x-sjis') - - # Encode the document into some encoding, and the encoding is - # substituted into the meta tag. - utf_8 = soup.encode("utf-8") - self.assertTrue(b"charset=utf-8" in utf_8) - - euc_jp = soup.encode("euc_jp") - self.assertTrue(b"charset=euc_jp" in euc_jp) - - shift_jis = soup.encode("shift-jis") - self.assertTrue(b"charset=shift-jis" in shift_jis) - - utf_16_u = soup.encode("utf-16").decode("utf-16") - self.assertTrue("charset=utf-16" in utf_16_u) - - def test_encoding_substitution_doesnt_happen_if_tag_is_strained(self): - markup = ('
        foo
        ') - - # Beautiful Soup used to try to rewrite the meta tag even if the - # meta tag got filtered out by the strainer. This test makes - # sure that doesn't happen. - strainer = SoupStrainer('pre') - soup = self.soup(markup, parse_only=strainer) - self.assertEqual(soup.contents[0].name, 'pre') - -class TestEncoding(SoupTest): - """Test the ability to encode objects into strings.""" - - def test_unicode_string_can_be_encoded(self): - html = "\N{SNOWMAN}" - soup = self.soup(html) - self.assertEqual(soup.b.string.encode("utf-8"), - "\N{SNOWMAN}".encode("utf-8")) - - def test_tag_containing_unicode_string_can_be_encoded(self): - html = "\N{SNOWMAN}" - soup = self.soup(html) - self.assertEqual( - soup.b.encode("utf-8"), html.encode("utf-8")) - - def test_encoding_substitutes_unrecognized_characters_by_default(self): - html = "\N{SNOWMAN}" - soup = self.soup(html) - self.assertEqual(soup.b.encode("ascii"), b"☃") - - def test_encoding_can_be_made_strict(self): - html = "\N{SNOWMAN}" - soup = self.soup(html) - self.assertRaises( - UnicodeEncodeError, soup.encode, "ascii", errors="strict") - - def test_decode_contents(self): - html = "\N{SNOWMAN}" - soup = self.soup(html) - self.assertEqual("\N{SNOWMAN}", soup.b.decode_contents()) - - def test_encode_contents(self): - html = "\N{SNOWMAN}" - soup = self.soup(html) - self.assertEqual( - "\N{SNOWMAN}".encode("utf8"), soup.b.encode_contents( - encoding="utf8")) - - def test_deprecated_renderContents(self): - html = "\N{SNOWMAN}" - soup = self.soup(html) - self.assertEqual( - "\N{SNOWMAN}".encode("utf8"), soup.b.renderContents()) - - def test_repr(self): - html = "\N{SNOWMAN}" - soup = self.soup(html) - if PY3K: - self.assertEqual(html, repr(soup)) - else: - self.assertEqual(b'\\u2603', repr(soup)) - -class TestNavigableStringSubclasses(SoupTest): - - def test_cdata(self): - # None of the current builders turn CDATA sections into CData - # objects, but you can create them manually. - soup = self.soup("") - cdata = CData("foo") - soup.insert(1, cdata) - self.assertEqual(str(soup), "") - self.assertEqual(soup.find(text="foo"), "foo") - self.assertEqual(soup.contents[0], "foo") - - def test_cdata_is_never_formatted(self): - """Text inside a CData object is passed into the formatter. - - But the return value is ignored. - """ - - self.count = 0 - def increment(*args): - self.count += 1 - return "BITTER FAILURE" - - soup = self.soup("") - cdata = CData("<><><>") - soup.insert(1, cdata) - self.assertEqual( - b"<><>]]>", soup.encode(formatter=increment)) - self.assertEqual(1, self.count) - - def test_doctype_ends_in_newline(self): - # Unlike other NavigableString subclasses, a DOCTYPE always ends - # in a newline. - doctype = Doctype("foo") - soup = self.soup("") - soup.insert(1, doctype) - self.assertEqual(soup.encode(), b"\n") - - def test_declaration(self): - d = Declaration("foo") - self.assertEqual("", d.output_ready()) - -class TestSoupSelector(TreeTest): - - HTML = """ - - - -The title - - - -Hello there. -
        -
        -

        An H1

        -

        Some text

        -

        Some more text

        -

        An H2

        -

        Another

        -Bob -

        Another H2

        -me - -span1a1 -span1a2 test - -span2a1 - - - -
        - -
        - - - - - - - - -

        English

        -

        English UK

        -

        English US

        -

        French

        -
        - - -""" - - def setUp(self): - self.soup = BeautifulSoup(self.HTML, 'html.parser') - - def assertSelects(self, selector, expected_ids): - el_ids = [el['id'] for el in self.soup.select(selector)] - el_ids.sort() - expected_ids.sort() - self.assertEqual(expected_ids, el_ids, - "Selector %s, expected [%s], got [%s]" % ( - selector, ', '.join(expected_ids), ', '.join(el_ids) - ) - ) - - assertSelect = assertSelects - - def assertSelectMultiple(self, *tests): - for selector, expected_ids in tests: - self.assertSelect(selector, expected_ids) - - def test_one_tag_one(self): - els = self.soup.select('title') - self.assertEqual(len(els), 1) - self.assertEqual(els[0].name, 'title') - self.assertEqual(els[0].contents, ['The title']) - - def test_one_tag_many(self): - els = self.soup.select('div') - self.assertEqual(len(els), 4) - for div in els: - self.assertEqual(div.name, 'div') - - el = self.soup.select_one('div') - self.assertEqual('main', el['id']) - - def test_select_one_returns_none_if_no_match(self): - match = self.soup.select_one('nonexistenttag') - self.assertEqual(None, match) - - - def test_tag_in_tag_one(self): - els = self.soup.select('div div') - self.assertSelects('div div', ['inner', 'data1']) - - def test_tag_in_tag_many(self): - for selector in ('html div', 'html body div', 'body div'): - self.assertSelects(selector, ['data1', 'main', 'inner', 'footer']) - - def test_tag_no_match(self): - self.assertEqual(len(self.soup.select('del')), 0) - - def test_invalid_tag(self): - self.assertRaises(ValueError, self.soup.select, 'tag%t') - - def test_select_dashed_tag_ids(self): - self.assertSelects('custom-dashed-tag', ['dash1', 'dash2']) - - def test_select_dashed_by_id(self): - dashed = self.soup.select('custom-dashed-tag[id=\"dash2\"]') - self.assertEqual(dashed[0].name, 'custom-dashed-tag') - self.assertEqual(dashed[0]['id'], 'dash2') - - def test_dashed_tag_text(self): - self.assertEqual(self.soup.select('body > custom-dashed-tag')[0].text, 'Hello there.') - - def test_select_dashed_matches_find_all(self): - self.assertEqual(self.soup.select('custom-dashed-tag'), self.soup.find_all('custom-dashed-tag')) - - def test_header_tags(self): - self.assertSelectMultiple( - ('h1', ['header1']), - ('h2', ['header2', 'header3']), - ) - - def test_class_one(self): - for selector in ('.onep', 'p.onep', 'html p.onep'): - els = self.soup.select(selector) - self.assertEqual(len(els), 1) - self.assertEqual(els[0].name, 'p') - self.assertEqual(els[0]['class'], ['onep']) - - def test_class_mismatched_tag(self): - els = self.soup.select('div.onep') - self.assertEqual(len(els), 0) - - def test_one_id(self): - for selector in ('div#inner', '#inner', 'div div#inner'): - self.assertSelects(selector, ['inner']) - - def test_bad_id(self): - els = self.soup.select('#doesnotexist') - self.assertEqual(len(els), 0) - - def test_items_in_id(self): - els = self.soup.select('div#inner p') - self.assertEqual(len(els), 3) - for el in els: - self.assertEqual(el.name, 'p') - self.assertEqual(els[1]['class'], ['onep']) - self.assertFalse(els[0].has_attr('class')) - - def test_a_bunch_of_emptys(self): - for selector in ('div#main del', 'div#main div.oops', 'div div#main'): - self.assertEqual(len(self.soup.select(selector)), 0) - - def test_multi_class_support(self): - for selector in ('.class1', 'p.class1', '.class2', 'p.class2', - '.class3', 'p.class3', 'html p.class2', 'div#inner .class2'): - self.assertSelects(selector, ['pmulti']) - - def test_multi_class_selection(self): - for selector in ('.class1.class3', '.class3.class2', - '.class1.class2.class3'): - self.assertSelects(selector, ['pmulti']) - - def test_child_selector(self): - self.assertSelects('.s1 > a', ['s1a1', 's1a2']) - self.assertSelects('.s1 > a span', ['s1a2s1']) - - def test_child_selector_id(self): - self.assertSelects('.s1 > a#s1a2 span', ['s1a2s1']) - - def test_attribute_equals(self): - self.assertSelectMultiple( - ('p[class="onep"]', ['p1']), - ('p[id="p1"]', ['p1']), - ('[class="onep"]', ['p1']), - ('[id="p1"]', ['p1']), - ('link[rel="stylesheet"]', ['l1']), - ('link[type="text/css"]', ['l1']), - ('link[href="blah.css"]', ['l1']), - ('link[href="no-blah.css"]', []), - ('[rel="stylesheet"]', ['l1']), - ('[type="text/css"]', ['l1']), - ('[href="blah.css"]', ['l1']), - ('[href="no-blah.css"]', []), - ('p[href="no-blah.css"]', []), - ('[href="no-blah.css"]', []), - ) - - def test_attribute_tilde(self): - self.assertSelectMultiple( - ('p[class~="class1"]', ['pmulti']), - ('p[class~="class2"]', ['pmulti']), - ('p[class~="class3"]', ['pmulti']), - ('[class~="class1"]', ['pmulti']), - ('[class~="class2"]', ['pmulti']), - ('[class~="class3"]', ['pmulti']), - ('a[rel~="friend"]', ['bob']), - ('a[rel~="met"]', ['bob']), - ('[rel~="friend"]', ['bob']), - ('[rel~="met"]', ['bob']), - ) - - def test_attribute_startswith(self): - self.assertSelectMultiple( - ('[rel^="style"]', ['l1']), - ('link[rel^="style"]', ['l1']), - ('notlink[rel^="notstyle"]', []), - ('[rel^="notstyle"]', []), - ('link[rel^="notstyle"]', []), - ('link[href^="bla"]', ['l1']), - ('a[href^="http://"]', ['bob', 'me']), - ('[href^="http://"]', ['bob', 'me']), - ('[id^="p"]', ['pmulti', 'p1']), - ('[id^="m"]', ['me', 'main']), - ('div[id^="m"]', ['main']), - ('a[id^="m"]', ['me']), - ('div[data-tag^="dashed"]', ['data1']) - ) - - def test_attribute_endswith(self): - self.assertSelectMultiple( - ('[href$=".css"]', ['l1']), - ('link[href$=".css"]', ['l1']), - ('link[id$="1"]', ['l1']), - ('[id$="1"]', ['data1', 'l1', 'p1', 'header1', 's1a1', 's2a1', 's1a2s1', 'dash1']), - ('div[id$="1"]', ['data1']), - ('[id$="noending"]', []), - ) - - def test_attribute_contains(self): - self.assertSelectMultiple( - # From test_attribute_startswith - ('[rel*="style"]', ['l1']), - ('link[rel*="style"]', ['l1']), - ('notlink[rel*="notstyle"]', []), - ('[rel*="notstyle"]', []), - ('link[rel*="notstyle"]', []), - ('link[href*="bla"]', ['l1']), - ('[href*="http://"]', ['bob', 'me']), - ('[id*="p"]', ['pmulti', 'p1']), - ('div[id*="m"]', ['main']), - ('a[id*="m"]', ['me']), - # From test_attribute_endswith - ('[href*=".css"]', ['l1']), - ('link[href*=".css"]', ['l1']), - ('link[id*="1"]', ['l1']), - ('[id*="1"]', ['data1', 'l1', 'p1', 'header1', 's1a1', 's1a2', 's2a1', 's1a2s1', 'dash1']), - ('div[id*="1"]', ['data1']), - ('[id*="noending"]', []), - # New for this test - ('[href*="."]', ['bob', 'me', 'l1']), - ('a[href*="."]', ['bob', 'me']), - ('link[href*="."]', ['l1']), - ('div[id*="n"]', ['main', 'inner']), - ('div[id*="nn"]', ['inner']), - ('div[data-tag*="edval"]', ['data1']) - ) - - def test_attribute_exact_or_hypen(self): - self.assertSelectMultiple( - ('p[lang|="en"]', ['lang-en', 'lang-en-gb', 'lang-en-us']), - ('[lang|="en"]', ['lang-en', 'lang-en-gb', 'lang-en-us']), - ('p[lang|="fr"]', ['lang-fr']), - ('p[lang|="gb"]', []), - ) - - def test_attribute_exists(self): - self.assertSelectMultiple( - ('[rel]', ['l1', 'bob', 'me']), - ('link[rel]', ['l1']), - ('a[rel]', ['bob', 'me']), - ('[lang]', ['lang-en', 'lang-en-gb', 'lang-en-us', 'lang-fr']), - ('p[class]', ['p1', 'pmulti']), - ('[blah]', []), - ('p[blah]', []), - ('div[data-tag]', ['data1']) - ) - - def test_unsupported_pseudoclass(self): - self.assertRaises( - NotImplementedError, self.soup.select, "a:no-such-pseudoclass") - - self.assertRaises( - NotImplementedError, self.soup.select, "a:nth-of-type(a)") - - - def test_nth_of_type(self): - # Try to select first paragraph - els = self.soup.select('div#inner p:nth-of-type(1)') - self.assertEqual(len(els), 1) - self.assertEqual(els[0].string, 'Some text') - - # Try to select third paragraph - els = self.soup.select('div#inner p:nth-of-type(3)') - self.assertEqual(len(els), 1) - self.assertEqual(els[0].string, 'Another') - - # Try to select (non-existent!) fourth paragraph - els = self.soup.select('div#inner p:nth-of-type(4)') - self.assertEqual(len(els), 0) - - # Pass in an invalid value. - self.assertRaises( - ValueError, self.soup.select, 'div p:nth-of-type(0)') - - def test_nth_of_type_direct_descendant(self): - els = self.soup.select('div#inner > p:nth-of-type(1)') - self.assertEqual(len(els), 1) - self.assertEqual(els[0].string, 'Some text') - - def test_id_child_selector_nth_of_type(self): - self.assertSelects('#inner > p:nth-of-type(2)', ['p1']) - - def test_select_on_element(self): - # Other tests operate on the tree; this operates on an element - # within the tree. - inner = self.soup.find("div", id="main") - selected = inner.select("div") - # The
        tag was selected. The