From patchwork Fri Sep 8 08:49:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Mikanovich X-Patchwork-Id: 3045 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 08 Sep 2023 10:49:30 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f189.google.com (mail-lj1-f189.google.com [209.85.208.189]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 3888nSpL023770 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 8 Sep 2023 10:49:29 +0200 Received: by mail-lj1-f189.google.com with SMTP id 38308e7fff4ca-2bcc03ed694sf22182121fa.2 for ; Fri, 08 Sep 2023 01:49:29 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1694162963; cv=pass; d=google.com; s=arc-20160816; b=FqxXVghtlu52HMNiPQM6Wwp8VfR1IBeuvKhdpBUEGJ1pmvJS/wRHOvngeAXTgEekKj AmGglcBdTsIxjQp6XiQwqaVWflaTpJQ1k+5gShjuLV1gieEgLQ0CVXl72X1PNrt2iKBE t5A6f34gnk7mlxtyQvsw6s/AO6wDRqclIDohmLj5DrQu1OU++YGNB4Hnv3onwvUI4cAK sqqRkRNj2TNAOWMXxXi28Ku234hrLMICeP4mP/T/HfKPuYbmkF67Ob6GRe6edWiXx27f QQ5125ZMQ1hFHVMJKkCODTjRYlCCndu2rI8xbzgeO5pTbF1AnpUozFyyAmZIHRk/Tuuf /wiQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=S7IjKAeKlLhDB+J2Dzc3Wdsr34RBVxTar6qdabql3+M=; fh=/h9QQkzJ8EboVkWg45aWwpaUro6WMavIVd2OhN45RtE=; b=IBFY22BQjrd4FaRGiBkm3mzyznjHm/hYyWjMSgyAtk0nfln6tLwIspG87QjoKE8AH5 cgL3/vr3qquVCD4lMkjOs1Xey6EDE4XmH8R+uw8PsEwiakdE2Wb5TcnytOOVFT3UcVsD uNtdAx86PlYBVCMB+RT4/XCQuJ04WrS1b7Fkn88ptBYCdWu9LJV+ZCO7wJCRVvEL27hb vsRA6t9gICu7ztI6eTgATubNka9CT1CZWRzBi2gRiSQXEiBAaGYM+vy3bcQOvOYzmElC uKS/r9Hw3rM6GhldvrJ7Rm3+fT+70WKk7Uqv9omzM6hfe4om5FcGez8yyjM3zpGzBsW3 KZ1w== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1694162963; x=1694767763; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:message-id:date:subject:cc:to:from :sender:from:to:cc:subject:date:message-id:reply-to; bh=S7IjKAeKlLhDB+J2Dzc3Wdsr34RBVxTar6qdabql3+M=; b=PznbTX/kukyLwAag8ssKbs+cQDI1n/BWhhfUr2h7GQj9uRroSks2occkJabrksojKR D9uOmystKbIGVRhCnQvO9riDRrkXuhJEQT7ki5abNbdRVzQbFFgw2f/8QPiBOHqaZAYK YE+jq2Uf26ZzvN61rljBoEz30b1umSD0S34E9Kzb4Cqyi14xMJX6LnjINERTTtpZp6RZ Gz3wFsnZnfwSKgQSnRFdxn9DyPqVwZ8zDmMfrVwvgLkwixLueQ5cktQ/FXXDdCEhZSgV bqkOZut6CrW7s5Ak7LoIsI4LwwOYsZswr8zz7DNkXY8FW16blxpFf8j6x4DPRtoLqqg+ 8nfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694162963; x=1694767763; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:x-beenthere:x-gm-message-state :sender:from:to:cc:subject:date:message-id:reply-to; bh=S7IjKAeKlLhDB+J2Dzc3Wdsr34RBVxTar6qdabql3+M=; b=Fwhwc9F++PPUboGa+OB6ZMpxPmqXsiGUH7GErLipFUIgWq1uWxjRJMlZltOAFLUkqY eTgSr3OdUwTIU6cTYvG+CLDJNi/xVMty4Ji72HUx5CxHa4aUhMchg1xmhZ7xON9Y/w6w 7quuaOV7nI87sYa8vAJcpFRNZ2rf0xmfHasCGsLCI76kTBdfCgeXFVTJuihs1NkADp3E +uNixKQ/eIHU+RDKrPWGi5XY4RJFacBX3L6vbwhwmDbNarELIaAKFiZ9lIlKj8Nimq+b W1tYWNCVCdU1iGrhWILH8pskI2bFOVmGXaf/SzPZHFjBfzO/WAEvtVaUyndFF75JVeSn TMHQ== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOJu0YwGZjQbAa6SfBOkZvOROx/4g+1A1zn8/MVVBkziB2opdJd/mCVD /OInLQKQ9ZT6A+k1lguHqv4= X-Google-Smtp-Source: AGHT+IH08i2dR8qvI0OOfoXHnBqRaFM2IgsXk+VshLydA40A0Qoy5sgUY/CzQK10lJ87dErRlZAOig== X-Received: by 2002:a2e:9c4c:0:b0:2bc:e3a5:57aa with SMTP id t12-20020a2e9c4c000000b002bce3a557aamr1134773ljj.0.1694162962882; Fri, 08 Sep 2023 01:49:22 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:b2a:b0:2b9:5184:f05f with SMTP id b42-20020a05651c0b2a00b002b95184f05fls402146ljr.1.-pod-prod-01-eu; Fri, 08 Sep 2023 01:49:21 -0700 (PDT) X-Received: by 2002:a2e:7a0a:0:b0:2b5:7f93:b3b0 with SMTP id v10-20020a2e7a0a000000b002b57f93b3b0mr1204206ljc.17.1694162960853; Fri, 08 Sep 2023 01:49:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694162960; cv=none; d=google.com; s=arc-20160816; b=MRklmgjFl4nOuK5QtTuFeiRrHL/gQG75dfDkw/TeWjhEwG1yA/HTNn8neXSElRinlx GZ8QhyPaCjFpGFNTHkyLatYH2Buj8UYgXY6FzYtb3zwt3DbAzxydBhHVrCBpwSD/Pz+B 2SCxmA6XFDAxALAUhq7lP2XDBFBmL//JCF8kY32yimD3MmEvqz3DoHzFwACDzvHxWlWb 2RSvoetTEkHANMKiUpWPkh3x9gbYjztf1WK1sr1e9XnN8+bnQJ0ru5yVh/wAFNq9hCuV GFCAAwu+8bzAQLaRfm+fvGz1zHg8AYLTJ7ufTPlPF/IkPSZU0CHCxZx+XcaU+Vzj06jQ Ciqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=qikJnHf44vTjfVsQftz9MNa9X9PIoIvUrsBew6SA2lI=; fh=/h9QQkzJ8EboVkWg45aWwpaUro6WMavIVd2OhN45RtE=; b=triv26gvqs6ZXkEQJM73yd393FwKwIom+fFRu/HTz4ze/psZPf0fEL/GrC7Nar9TcE HVbrbIRAbP4olcIe+JnE7SFY/brJGHhWIrWdRPnGVfs2kQRuPfpOiVAhiQcaYB0qW+YM jOucqsO8rNDKvP8JrzQhpnekO2vHdiVlCBiK8bDPHSjcHzKDSHLKqdiD8P5DPJlgg2Ef CBr+Jqj6ziKFmxlQ1bGS2a25w9pR6CyRYiMze5usbdZ55sMe5I5FAGrY1qNHD0ze2z41 EtSd3PAj8HIVXBIr/dbtagXhO2RLcq+OMHhON5MZjmk93HroWjXCms4WPAns0DWPpXgL U0Ug== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id k11-20020a2ea26b000000b002b96150efccsi80016ljm.0.2023.09.08.01.49.20 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 08 Sep 2023 01:49:20 -0700 (PDT) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (r154-240-38-77-broadband.btv.lv [77.38.240.154] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 3888nIKv023758 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 8 Sep 2023 10:49:19 +0200 From: Anton Mikanovich To: isar-users@googlegroups.com Cc: Anton Mikanovich Subject: [PATCH] bitbake: update to Bitbake 2.0.12 Date: Fri, 8 Sep 2023 11:49:12 +0300 Message-Id: <20230908084912.2656302-1-amikan@ilbers.de> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: amikan@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= Upstream commit 41b6684489d0261753344956042be2cc4adb0159. Signed-off-by: Anton Mikanovich --- bitbake/bin/bitbake | 3 +- bitbake/bin/bitbake-diffsigs | 49 +++++++++-------- bitbake/bin/bitbake-layers | 4 +- bitbake/bin/bitbake-server | 5 +- bitbake/bin/bitbake-worker | 3 +- .../bitbake-user-manual-fetching.rst | 4 +- bitbake/lib/bb/cookerdata.py | 17 +----- bitbake/lib/bb/fetch2/git.py | 19 +++++-- bitbake/lib/bb/fetch2/gitsm.py | 2 +- bitbake/lib/bb/runqueue.py | 52 ++++++++++++------- bitbake/lib/bb/siggen.py | 11 ++-- bitbake/lib/bb/utils.py | 47 ++++++++++++----- bitbake/lib/ply/yacc.py | 7 +++ 13 files changed, 132 insertions(+), 91 deletions(-) diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake index 042c9180..0b9cc629 100755 --- a/bitbake/bin/bitbake +++ b/bitbake/bin/bitbake @@ -25,8 +25,7 @@ except RuntimeError as exc: from bb import cookerdata from bb.main import bitbake_main, BitBakeConfigParameters, BBMainException -if sys.getfilesystemencoding() != "utf-8": - sys.exit("Please use a locale setting which supports UTF-8 (such as LANG=en_US.UTF-8).\nPython can't change the filesystem locale after loading so we need a UTF-8 when Python starts or things won't work.") +bb.utils.check_system_locale() __version__ = "2.0.0" diff --git a/bitbake/bin/bitbake-diffsigs b/bitbake/bin/bitbake-diffsigs index cf4cc706..fe0f33ee 100755 --- a/bitbake/bin/bitbake-diffsigs +++ b/bitbake/bin/bitbake-diffsigs @@ -11,6 +11,7 @@ import os import sys import warnings + warnings.simplefilter("default") import argparse import logging @@ -27,6 +28,7 @@ logger = bb.msg.logger_create(myname) is_dump = myname == 'bitbake-dumpsig' + def find_siginfo(tinfoil, pn, taskname, sigs=None): result = None tinfoil.set_event_mask(['bb.event.FindSigInfoResult', @@ -52,6 +54,7 @@ def find_siginfo(tinfoil, pn, taskname, sigs=None): sys.exit(2) return result + def find_siginfo_task(bbhandler, pn, taskname, sig1=None, sig2=None): """ Find the most recent signature files for the specified PN/task """ @@ -63,10 +66,10 @@ def find_siginfo_task(bbhandler, pn, taskname, sig1=None, sig2=None): if not sigfiles: logger.error('No sigdata files found matching %s %s matching either %s or %s' % (pn, taskname, sig1, sig2)) sys.exit(1) - elif not sig1 in sigfiles: + elif sig1 not in sigfiles: logger.error('No sigdata files found matching %s %s with signature %s' % (pn, taskname, sig1)) sys.exit(1) - elif not sig2 in sigfiles: + elif sig2 not in sigfiles: logger.error('No sigdata files found matching %s %s with signature %s' % (pn, taskname, sig2)) sys.exit(1) latestfiles = [sigfiles[sig1], sigfiles[sig2]] @@ -88,9 +91,9 @@ def recursecb(key, hash1, hash2): recout = [] if not hashfiles: recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2)) - elif not hash1 in hashfiles: + elif hash1 not in hashfiles: recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash1)) - elif not hash2 in hashfiles: + elif hash2 not in hashfiles: recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2)) else: out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, color=color) @@ -110,36 +113,36 @@ parser.add_argument('-D', '--debug', if is_dump: parser.add_argument("-t", "--task", - help="find the signature data file for the last run of the specified task", - action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname')) + help="find the signature data file for the last run of the specified task", + action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname')) parser.add_argument("sigdatafile1", - help="Signature file to dump. Not used when using -t/--task.", - action="store", nargs='?', metavar="sigdatafile") + help="Signature file to dump. Not used when using -t/--task.", + action="store", nargs='?', metavar="sigdatafile") else: parser.add_argument('-c', '--color', - help='Colorize the output (where %(metavar)s is %(choices)s)', - choices=['auto', 'always', 'never'], default='auto', metavar='color') + help='Colorize the output (where %(metavar)s is %(choices)s)', + choices=['auto', 'always', 'never'], default='auto', metavar='color') parser.add_argument('-d', '--dump', - help='Dump the last signature data instead of comparing (equivalent to using bitbake-dumpsig)', - action='store_true') + help='Dump the last signature data instead of comparing (equivalent to using bitbake-dumpsig)', + action='store_true') parser.add_argument("-t", "--task", - help="find the signature data files for the last two runs of the specified task and compare them", - action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname')) + help="find the signature data files for the last two runs of the specified task and compare them", + action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname')) parser.add_argument("-s", "--signature", - help="With -t/--task, specify the signatures to look for instead of taking the last two", - action="store", dest="sigargs", nargs=2, metavar=('fromsig', 'tosig')) + help="With -t/--task, specify the signatures to look for instead of taking the last two", + action="store", dest="sigargs", nargs=2, metavar=('fromsig', 'tosig')) parser.add_argument("sigdatafile1", - help="First signature file to compare (or signature file to dump, if second not specified). Not used when using -t/--task.", - action="store", nargs='?') + help="First signature file to compare (or signature file to dump, if second not specified). Not used when using -t/--task.", + action="store", nargs='?') parser.add_argument("sigdatafile2", - help="Second signature file to compare", - action="store", nargs='?') + help="Second signature file to compare", + action="store", nargs='?') options = parser.parse_args() if is_dump: @@ -157,7 +160,8 @@ if options.taskargs: with bb.tinfoil.Tinfoil() as tinfoil: tinfoil.prepare(config_only=True) if not options.dump and options.sigargs: - files = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1], options.sigargs[0], options.sigargs[1]) + files = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1], options.sigargs[0], + options.sigargs[1]) else: files = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1]) @@ -166,7 +170,8 @@ if options.taskargs: output = bb.siggen.dump_sigfile(files[-1]) else: if len(files) < 2: - logger.error('Only one matching sigdata file found for the specified task (%s %s)' % (options.taskargs[0], options.taskargs[1])) + logger.error('Only one matching sigdata file found for the specified task (%s %s)' % ( + options.taskargs[0], options.taskargs[1])) sys.exit(1) # Recurse into signature comparison diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers index 449434d4..d4b1d1aa 100755 --- a/bitbake/bin/bitbake-layers +++ b/bitbake/bin/bitbake-layers @@ -68,11 +68,11 @@ def main(): registered = False for plugin in plugins: + if hasattr(plugin, 'tinfoil_init'): + plugin.tinfoil_init(tinfoil) if hasattr(plugin, 'register_commands'): registered = True plugin.register_commands(subparsers) - if hasattr(plugin, 'tinfoil_init'): - plugin.tinfoil_init(tinfoil) if not registered: logger.error("No commands registered - missing plugins?") diff --git a/bitbake/bin/bitbake-server b/bitbake/bin/bitbake-server index f53f88b6..d00bb068 100755 --- a/bitbake/bin/bitbake-server +++ b/bitbake/bin/bitbake-server @@ -12,8 +12,9 @@ warnings.simplefilter("default") import logging sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib')) -if sys.getfilesystemencoding() != "utf-8": - sys.exit("Please use a locale setting which supports UTF-8 (such as LANG=en_US.UTF-8).\nPython can't change the filesystem locale after loading so we need a UTF-8 when Python starts or things won't work.") +import bb + +bb.utils.check_system_locale() # Users shouldn't be running this code directly if len(sys.argv) != 10 or not sys.argv[1].startswith("decafbad"): diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker index 2f3e9f72..5e62bc20 100755 --- a/bitbake/bin/bitbake-worker +++ b/bitbake/bin/bitbake-worker @@ -24,8 +24,7 @@ import subprocess from multiprocessing import Lock from threading import Thread -if sys.getfilesystemencoding() != "utf-8": - sys.exit("Please use a locale setting which supports UTF-8 (such as LANG=en_US.UTF-8).\nPython can't change the filesystem locale after loading so we need a UTF-8 when Python starts or things won't work.") +bb.utils.check_system_locale() # Users shouldn't be running this code directly if len(sys.argv) != 2 or not sys.argv[1].startswith("decafbad"): diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst index 9c269ca8..519aec9a 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst @@ -424,8 +424,8 @@ This fetcher supports the following parameters: - *"nobranch":* Tells the fetcher to not check the SHA validation for the branch when set to "1". The default is "0". Set this option for - the recipe that refers to the commit that is valid for a tag instead - of the branch. + the recipe that refers to the commit that is valid for any namespace + (branch, tag, ...) instead of the branch. - *"bareclone":* Tells the fetcher to clone a bare clone into the destination directory without checking out a working tree. Only the diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py index fe5696c7..ec3741cc 100644 --- a/bitbake/lib/bb/cookerdata.py +++ b/bitbake/lib/bb/cookerdata.py @@ -160,12 +160,7 @@ def catch_parse_error(func): def wrapped(fn, *args): try: return func(fn, *args) - except IOError as exc: - import traceback - parselog.critical(traceback.format_exc()) - parselog.critical("Unable to parse %s: %s" % (fn, exc)) - raise bb.BBHandledException() - except bb.data_smart.ExpansionError as exc: + except Exception as exc: import traceback bbdir = os.path.dirname(__file__) + os.sep @@ -177,9 +172,6 @@ def catch_parse_error(func): break parselog.critical("Unable to parse %s" % fn, exc_info=(exc_class, exc, tb)) raise bb.BBHandledException() - except bb.parse.ParseError as exc: - parselog.critical(str(exc)) - raise bb.BBHandledException() return wrapped @catch_parse_error @@ -301,14 +293,9 @@ class CookerDataBuilder(object): bb.event.fire(bb.event.MultiConfigParsed(self.mcdata), self.data) self.data_hash = data_hash.hexdigest() - except (SyntaxError, bb.BBHandledException): - raise bb.BBHandledException() except bb.data_smart.ExpansionError as e: logger.error(str(e)) raise bb.BBHandledException() - except Exception: - logger.exception("Error parsing configuration files") - raise bb.BBHandledException() # Handle obsolete variable names @@ -435,7 +422,7 @@ class CookerDataBuilder(object): msg += (" and bitbake did not find a conf/bblayers.conf file in" " the expected location.\nMaybe you accidentally" " invoked bitbake from the wrong directory?") - raise SystemExit(msg) + bb.fatal(msg) if not data.getVar("TOPDIR"): data.setVar("TOPDIR", os.path.abspath(os.getcwd())) diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index f0df6fb6..4d6e57ad 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -44,7 +44,8 @@ Supported SRC_URI options are: - nobranch Don't check the SHA validation for branch. set this option for the recipe - referring to commit which is valid in tag instead of branch. + referring to commit which is valid in any namespace (branch, tag, ...) + instead of branch. The default is "0", set nobranch=1 if needed. - usehead @@ -358,9 +359,13 @@ class Git(FetchMethod): # If the repo still doesn't exist, fallback to cloning it if not os.path.exists(ud.clonedir): - # We do this since git will use a "-l" option automatically for local urls where possible + # We do this since git will use a "-l" option automatically for local urls where possible, + # but it doesn't work when git/objects is a symlink, only works when it is a directory. if repourl.startswith("file://"): - repourl = repourl[7:] + repourl_path = repourl[7:] + objects = os.path.join(repourl_path, 'objects') + if os.path.isdir(objects) and not os.path.islink(objects): + repourl = repourl_path clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, shlex.quote(repourl), ud.clonedir) if ud.proto.lower() != 'file': bb.fetch2.check_network_access(d, clone_cmd, ud.url) @@ -374,7 +379,11 @@ class Git(FetchMethod): runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir) runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=ud.clonedir) - fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl)) + + if ud.nobranch: + fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl)) + else: + fetch_cmd = "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*" % (ud.basecmd, shlex.quote(repourl)) if ud.proto.lower() != 'file': bb.fetch2.check_network_access(d, fetch_cmd, ud.url) progresshandler = GitProgressHandler(d) @@ -731,7 +740,7 @@ class Git(FetchMethod): Compute the HEAD revision for the url """ if not d.getVar("__BBSEENSRCREV"): - raise bb.fetch2.FetchError("Recipe uses a floating tag/branch without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).") + raise bb.fetch2.FetchError("Recipe uses a floating tag/branch '%s' for repo '%s' without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE)." % (ud.unresolvedrev[name], ud.host+ud.path)) # Ensure we mark as not cached bb.fetch2.get_autorev(d) diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py index 25d5db0e..c5f7c03c 100644 --- a/bitbake/lib/bb/fetch2/gitsm.py +++ b/bitbake/lib/bb/fetch2/gitsm.py @@ -115,7 +115,7 @@ class GitSM(Git): # This has to be a file reference proto = "file" url = "gitsm://" + uris[module] - if "{}{}".format(ud.host, ud.path) in url: + if url.endswith("{}{}".format(ud.host, ud.path)): raise bb.fetch2.FetchError("Submodule refers to the parent repository. This will cause deadlock situation in current version of Bitbake." \ "Consider using git fetcher instead.") diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index 48e25401..56952070 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -1974,11 +1974,19 @@ class RunQueueExecute: self.setbuildable(revdep) logger.debug("Marking task %s as buildable", revdep) - for t in self.sq_deferred.copy(): + found = None + for t in sorted(self.sq_deferred.copy()): if self.sq_deferred[t] == task: - logger.debug2("Deferred task %s now buildable" % t) - del self.sq_deferred[t] - update_scenequeue_data([t], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False) + # Allow the next deferred task to run. Any other deferred tasks should be deferred after that task. + # We shouldn't allow all to run at once as it is prone to races. + if not found: + bb.note("Deferred task %s now buildable" % t) + del self.sq_deferred[t] + update_scenequeue_data([t], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False) + found = t + else: + bb.note("Deferring %s after %s" % (t, found)) + self.sq_deferred[t] = found def task_complete(self, task): self.stats.taskCompleted() @@ -2489,17 +2497,6 @@ class RunQueueExecute: self.sq_buildable.remove(tid) if tid in self.sq_running: self.sq_running.remove(tid) - harddepfail = False - for t in self.sqdata.sq_harddeps: - if tid in self.sqdata.sq_harddeps[t] and t in self.scenequeue_notcovered: - harddepfail = True - break - if not harddepfail and self.sqdata.sq_revdeps[tid].issubset(self.scenequeue_covered | self.scenequeue_notcovered): - if tid not in self.sq_buildable: - self.sq_buildable.add(tid) - if not self.sqdata.sq_revdeps[tid]: - self.sq_buildable.add(tid) - if tid in self.sqdata.outrightfail: self.sqdata.outrightfail.remove(tid) if tid in self.scenequeue_notcovered: @@ -2518,21 +2515,36 @@ class RunQueueExecute: if tid in self.build_stamps: del self.build_stamps[tid] - update_tasks.append((tid, harddepfail, tid in self.sqdata.valid)) + update_tasks.append(tid) + + update_tasks2 = [] + for tid in update_tasks: + harddepfail = False + for t in self.sqdata.sq_harddeps: + if tid in self.sqdata.sq_harddeps[t] and t in self.scenequeue_notcovered: + harddepfail = True + break + if not harddepfail and self.sqdata.sq_revdeps[tid].issubset(self.scenequeue_covered | self.scenequeue_notcovered): + if tid not in self.sq_buildable: + self.sq_buildable.add(tid) + if not self.sqdata.sq_revdeps[tid]: + self.sq_buildable.add(tid) + + update_tasks2.append((tid, harddepfail, tid in self.sqdata.valid)) - if update_tasks: + if update_tasks2: self.sqdone = False for mc in sorted(self.sqdata.multiconfigs): - for tid in sorted([t[0] for t in update_tasks]): + for tid in sorted([t[0] for t in update_tasks2]): if mc_from_tid(tid) != mc: continue h = pending_hash_index(tid, self.rqdata) if h in self.sqdata.hashes and tid != self.sqdata.hashes[h]: self.sq_deferred[tid] = self.sqdata.hashes[h] bb.note("Deferring %s after %s" % (tid, self.sqdata.hashes[h])) - update_scenequeue_data([t[0] for t in update_tasks], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False) + update_scenequeue_data([t[0] for t in update_tasks2], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False) - for (tid, harddepfail, origvalid) in update_tasks: + for (tid, harddepfail, origvalid) in update_tasks2: if tid in self.sqdata.valid and not origvalid: hashequiv_logger.verbose("Setscene task %s became valid" % tid) if harddepfail: diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py index 9a20fc8e..0a9ce0ed 100644 --- a/bitbake/lib/bb/siggen.py +++ b/bitbake/lib/bb/siggen.py @@ -329,19 +329,19 @@ class SignatureGeneratorBasic(SignatureGenerator): data = self.basehash[tid] for dep in self.runtaskdeps[tid]: - data = data + self.get_unihash(dep) + data += self.get_unihash(dep) for (f, cs) in self.file_checksum_values[tid]: if cs: if "/./" in f: - data = data + "./" + f.split("/./")[1] - data = data + cs + data += "./" + f.split("/./")[1] + data += cs if tid in self.taints: if self.taints[tid].startswith("nostamp:"): - data = data + self.taints[tid][8:] + data += self.taints[tid][8:] else: - data = data + self.taints[tid] + data += self.taints[tid] h = hashlib.sha256(data.encode("utf-8")).hexdigest() self.taskhash[tid] = h @@ -1028,6 +1028,7 @@ def compare_sigfiles(a, b, recursecb=None, color=False, collapsed=False): # If a dependent hash changed, might as well print the line above and then defer to the changes in # that hash since in all likelyhood, they're the same changes this task also saw. output = [output[-1]] + recout + break a_taint = a_data.get('taint', None) b_taint = b_data.get('taint', None) diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py index 92d44c52..3f7f82d1 100644 --- a/bitbake/lib/bb/utils.py +++ b/bitbake/lib/bb/utils.py @@ -13,6 +13,7 @@ import errno import logging import bb import bb.msg +import locale import multiprocessing import fcntl import importlib @@ -545,7 +546,12 @@ def md5_file(filename): Return the hex string representation of the MD5 checksum of filename. """ import hashlib - return _hasher(hashlib.new('MD5', usedforsecurity=False), filename) + try: + sig = hashlib.new('MD5', usedforsecurity=False) + except TypeError: + # Some configurations don't appear to support two arguments + sig = hashlib.new('MD5') + return _hasher(sig, filename) def sha256_file(filename): """ @@ -601,6 +607,21 @@ def preserved_envvars(): ] return v + preserved_envvars_exported() +def check_system_locale(): + """Make sure the required system locale are available and configured""" + default_locale = locale.getlocale(locale.LC_CTYPE) + + try: + locale.setlocale(locale.LC_CTYPE, ("en_US", "UTF-8")) + except: + sys.exit("Please make sure locale 'en_US.UTF-8' is available on your system") + else: + locale.setlocale(locale.LC_CTYPE, default_locale) + + if sys.getfilesystemencoding() != "utf-8": + sys.exit("Please use a locale setting which supports UTF-8 (such as LANG=en_US.UTF-8).\n" + "Python can't change the filesystem locale after loading so we need a UTF-8 when Python starts or things won't work.") + def filter_environment(good_vars): """ Create a pristine environment for bitbake. This will remove variables that @@ -985,6 +1006,9 @@ def to_boolean(string, default=None): if not string: return default + if isinstance(string, int): + return string != 0 + normalized = string.lower() if normalized in ("y", "yes", "1", "true"): return True @@ -1635,23 +1659,20 @@ def disable_network(uid=None, gid=None): def export_proxies(d): """ export common proxies variables from datastore to environment """ - import os variables = ['http_proxy', 'HTTP_PROXY', 'https_proxy', 'HTTPS_PROXY', 'ftp_proxy', 'FTP_PROXY', 'no_proxy', 'NO_PROXY', - 'GIT_PROXY_COMMAND'] - exported = False + 'GIT_PROXY_COMMAND', 'SSL_CERT_FILE', 'SSL_CERT_DIR'] - for v in variables: - if v in os.environ.keys(): - exported = True - else: - v_proxy = d.getVar(v) - if v_proxy is not None: - os.environ[v] = v_proxy - exported = True + origenv = d.getVar("BB_ORIGENV") + + for name in variables: + value = d.getVar(name) + if not value and origenv: + value = origenv.getVar(name) + if value: + os.environ[name] = value - return exported def load_plugins(logger, plugins, pluginpath): diff --git a/bitbake/lib/ply/yacc.py b/bitbake/lib/ply/yacc.py index 767c4e46..381b50cf 100644 --- a/bitbake/lib/ply/yacc.py +++ b/bitbake/lib/ply/yacc.py @@ -2798,7 +2798,14 @@ class ParserReflect(object): def signature(self): try: import hashlib + except ImportError: + raise RuntimeError("Unable to import hashlib") + try: sig = hashlib.new('MD5', usedforsecurity=False) + except TypeError: + # Some configurations don't appear to support two arguments + sig = hashlib.new('MD5') + try: if self.start: sig.update(self.start.encode('latin-1')) if self.prec: