From patchwork Fri Jun 21 15:08:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Mikanovich X-Patchwork-Id: 3628 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 21 Jun 2024 17:08:38 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f58.google.com (mail-lf1-f58.google.com [209.85.167.58]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 45LF8aWl006227 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 21 Jun 2024 17:08:37 +0200 Received: by mail-lf1-f58.google.com with SMTP id 2adb3069b0e04-52c805e6f38sf1708166e87.0 for ; Fri, 21 Jun 2024 08:08:37 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1718982510; cv=pass; d=google.com; s=arc-20160816; b=0ME5Msra8K8L2pot3YUwESv2LPZ1ip2hkM/E2G8QTjKf2gxxKojKVtWDU7K7IFy73X xLTJoN4p9nUMIDrUglblpVrSpe2hRPmtle9JlK/zktsq62zjIR4eMazW8mbHeb2SwduH sgaCXH1bJLCxD7Boz8K9Rdht65YVF/vIOMUlF6UWX2S0TUceCQ8LGLKwdd4gJbrIDWdK z6X7UAe5wgsQpaPQAV3Wdlyfz/wk6u+KQRMqyasxeZwssCLjxLDPH+i+Y/Mcj3Lrhsk3 x4y7xsH8mfRuRWFeVCdHDL6sTGRI+/C1stL1yJ9a3sluv3c8OZE4CRrKc/0yCuC10TZN 1glQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=LNivfEa5oZS35Fux4/cRKVMpln8H1gjip2LbM+Ih97s=; fh=FZGH+NqNtSsECqCQoJNaq8cRES7+ZiCtaWwkFBikIg0=; b=jcIUEOPx3i6QhrfNkjZrpHAa3gNh6b0GnLKQPw+bqRuzerZXH5JLcyBQNSaVg7Oifh SB090NYpxDNJV9y5R6IcBUsDtYDnTMie/6/I4LubingEl0lU7fWFjEqNdOm9+MaAg4TY 3gsq9UaOPVkANgvEuIV7AukMvnabhTIke56YgxvZVdAfqF86R1SjQrQBoYUAXyesFq9e Z8f9PgSjFX5emsSQUoA+BP+qQXz3En5Ni+TxG6YnamIk77teZFu+Edvy7yadF/QPrmS8 Q24TmjN3MFNDQw+NAFa/11Zz1HYK8cZm0Crtxscw9yeyk1+8Szgcxv58HOouVPGrrXMy 5mMA==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1718982510; x=1719587310; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=LNivfEa5oZS35Fux4/cRKVMpln8H1gjip2LbM+Ih97s=; b=ioDgsD3A+JoqgyaVevjXzm/kivZpKufmfJgRt82upjGJMN98fQf8JZXem6/TyJb3WM SkzRYXeawbEWSwbQ7h1Nd5n2lfsgJopBkG8EKDzyvJ7qXyCOum11HtV7EOTzvUGepoQT VwXb188FpmfKWd/z4z8Wzp1ufxRq/cOFwB4elPQwqTtroAQ/9qsgiEwxPc0F70JwB575 TIK/xG9frdfYncQwBMfEfASYMf2xiK7WGI/1tVCGOIVXZaghvf8PCABeQ6di6E92IvXI d8trlQHx0IjToLicWiDASjFJufX/pseydaAjq6YhdyfDmDz1FnmyM9cEmJod3e0afCYQ SGWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718982510; x=1719587310; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :x-beenthere:x-gm-message-state:sender:from:to:cc:subject:date :message-id:reply-to; bh=LNivfEa5oZS35Fux4/cRKVMpln8H1gjip2LbM+Ih97s=; b=a4y+bI8JfVpWkLB3it7Z+MRfxqlr02Cv7Mpv31Q2vN+tuJ/go/J5J9gtdDJeQB5FjZ T+iu4eUM76xeKWNnC3TI+/4V4v0cqzNzXdXRnpGw1KQxYVelIoqDsWZtWpTdep6j/zHn xevKr0EPgzGPbMPEvBKZTJZBzT5LTKpSIRSPTFqRZG90zY9E/djibeZQ4/6TWxTxY94B 2ZmgAeTVZuCSLFoXtxU4TsM0efgxTkdhuMhTNJ7gZWjwiJaLFFMuwd3SpXoJoPoAKzvo iQXNHrk6rtmdtM6TEnd6KvqfKaGdm/F2mXHQrXnuAwSJTELxJMSdFimRmigPT8kQQd/m LcZA== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCXw45sH9sxcrvgVnc02Gkt0Q+Xyr/dWPYwwsgSj8oBlGi4mCd7O54nB4EAc6FCsk6ozKT6FoXuZUAXpHOR6stKyAznd0gQ= X-Gm-Message-State: AOJu0YxbZD7iuZiC3m3p9w4abJQDi15g4eijSm+6lOdRhPDMgkgpp6H9 +BMF38RTSRdacAZN383qh4OjYbq0h1IiukFE5NEBQ24Q15MY0qy9 X-Google-Smtp-Source: AGHT+IEJGt+xQD2aqARxww1okWe/vOkDT4KGXyIvQXmQAkFxqkUDGXO+QVQ3Qxebo8/SOri0eav0lA== X-Received: by 2002:a19:644c:0:b0:52c:dbe6:f5f9 with SMTP id 2adb3069b0e04-52cdbe6f808mr399648e87.12.1718982509647; Fri, 21 Jun 2024 08:08:29 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:2206:b0:52c:a105:90ee with SMTP id 2adb3069b0e04-52ccf08ea71ls164306e87.1.-pod-prod-01-eu; Fri, 21 Jun 2024 08:08:28 -0700 (PDT) X-Received: by 2002:ac2:489b:0:b0:52c:d56f:b2cc with SMTP id 2adb3069b0e04-52cd56fb3bfmr1421759e87.58.1718982507620; Fri, 21 Jun 2024 08:08:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1718982507; cv=none; d=google.com; s=arc-20160816; b=YlEFlGHUzXH0cxRe+jEQZ0I21onVVkku/dSSTJTao8Uamxy1U5bERY+F+ujv5BTgR6 fq7VnLwyQ0vOuR6XAO65CE9r1TQIQNxeoEXCpSKMjwG/Uk+cZQ1PdgehGwhy6vrrqPwO eI89orGc4bcAEo7Soyl6cGWN/Skbqpj27WXM4fShrIy1HIt6ZNzylHh4R91m3vEsCkrt BjajtNkJSzRUJk3Ah6MnNJ7AFEA0vorJSciQZ4tjkyeHYLaWCWBNWt7vWzLOyo4+fDuS kj4m2DQdoNzFVAKLqazsR2BvSt0SFaVr/qj5ERwg1V8svy/LUDoF8WCO4ceaasazRJmk PwIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=GQDMB7shaswAgAwX4lgZ4QgmvgefyZhHCIVp2Od8fxk=; fh=/h9QQkzJ8EboVkWg45aWwpaUro6WMavIVd2OhN45RtE=; b=YH1dPrB/GdbJ08MXNr38UvxAsbHHFMlUO7RPsBCeJSpFHHoN8KAD0v/NlyAs83rKRz LbwWtZ04Jkp+fJLZi50KZKlGjv4yyONfeqwH4/X5xq2s83uO93eKnAnIAkSY5SHFEW+g xOSKZqzAVGXV0VIh/kquJU+JFjzuMb71kkzC3JN5OY/QZnmDGPkLAxZ5mhY5ffUhH7Kq aZA3mHGgltKENKzdiqF5mPGs+ooGfuY0tETkvP5AB3RC57E4xYJPJfk8UTKdLz4gaCTR kDdD2hYMKP797gBIPFFkA59bGfHx8A9dLvzViWvs+9emikyYPf1ubh5zUy19bmJA9STm V1CA==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-4247194de3bsi6211355e9.0.2024.06.21.08.08.27 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 21 Jun 2024 08:08:27 -0700 (PDT) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from user-B660.promwad.corp ([159.148.83.114]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 45LF8ObF006158 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Jun 2024 17:08:26 +0200 From: Anton Mikanovich To: isar-users@googlegroups.com Cc: Anton Mikanovich Subject: [PATCH v3 4/5] CI: Allow external usage for some APIs Date: Fri, 21 Jun 2024 18:08:13 +0300 Message-Id: <20240621150814.189288-5-amikan@ilbers.de> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240621150814.189288-1-amikan@ilbers.de> References: <20240621150814.189288-1-amikan@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: amikan@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= Provide utilities class for external usage to be imported from outside of Avocado framework. Signed-off-by: Anton Mikanovich --- testsuite/cibase.py | 8 +++-- testsuite/cibuilder.py | 52 ++++----------------------------- testsuite/citest.py | 4 ++- testsuite/repro-build-test.py | 4 ++- testsuite/utils.py | 55 +++++++++++++++++++++++++++++++++++ 5 files changed, 72 insertions(+), 51 deletions(-) create mode 100755 testsuite/utils.py diff --git a/testsuite/cibase.py b/testsuite/cibase.py index 0f6997af..fd571c29 100755 --- a/testsuite/cibase.py +++ b/testsuite/cibase.py @@ -8,6 +8,8 @@ import tempfile import time from cibuilder import CIBuilder, isar_root +from utils import CIUtils + from avocado.utils import process class CIBaseTest(CIBuilder): @@ -238,13 +240,13 @@ class CIBaseTest(CIBuilder): for target in targets: sfiles[target] = dict() package = target.rsplit(':', 1)[-1] - isar_apt = self.getVars('REPO_ISAR_DB_DIR', target=target) + isar_apt = CIUtils.getVars('REPO_ISAR_DB_DIR', target=target) fpath = f'{package}/{package}*.tar.gz' targz = set(glob.glob(f'{isar_apt}/../apt/*/pool/*/*/{fpath}')) if len(targz) < 1: self.fail('No source packages found') - for filename in targz: - sfiles[target][filename] = self.get_tar_content(filename) + for fname in targz: + sfiles[target][fname] = CIUtils.get_tar_content(fname) return sfiles self.configure(**kwargs) diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py index 87d71387..a20e88f9 100755 --- a/testsuite/cibuilder.py +++ b/testsuite/cibuilder.py @@ -9,11 +9,11 @@ import shutil import signal import subprocess import sys -import tarfile import time import tempfile import start_vm +from utils import CIUtils from avocado import Test from avocado.utils import path @@ -22,7 +22,6 @@ from avocado.utils import process sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)) + '/../bitbake/lib') import bb -import bb.tinfoil DEF_VM_TO_SEC = 600 @@ -253,38 +252,6 @@ class CIBuilder(Test): except FileNotFoundError: self.log.warn(path + backup_prefix + ' not exist') - def getVars(self, *vars, target=None): - self.check_init() - def fixStream(stream): - # fix stream objects to emulate _io.TextIOWrapper - stream.isatty = lambda: False - stream.fileno = lambda: False - stream.encoding = sys.getdefaultencoding() - - sl = target is not None - fixStream(sys.stdout) - fixStream(sys.stderr) - - # wait until previous bitbake will be finished - lockfile = os.path.join(self.build_dir, 'bitbake.lock') - checks = 0 - while os.path.exists(lockfile) and checks < 5: - time.sleep(1) - checks += 1 - - with bb.tinfoil.Tinfoil(setup_logging=sl) as tinfoil: - values = () - if target: - tinfoil.prepare(quiet=2) - d = tinfoil.parse_recipe(target) - for var in vars: - values = values + (d.getVar(var, True) or 'None',) - else: - tinfoil.prepare(config_only=True, quiet=2) - for var in vars: - values = values + (tinfoil.config_data.getVar(var, True) or 'None',) - return values if len(values) > 1 else values[0] - def create_tmp_layer(self): tmp_layer_dir = os.path.join(isar_root, 'meta-tmp') @@ -314,13 +281,6 @@ BBPATH .= ":${LAYERDIR}"\ bb.utils.edit_bblayers_conf(bblayersconf_file, None, tmp_layer_dir) bb.utils.prunedir(tmp_layer_dir) - def get_tar_content(self, filename): - try: - tar = tarfile.open(filename) - return tar.getnames() - except Exception: - return [] - def get_ssh_cmd_prefix(self, user, host, port, priv_key): cmd_prefix = 'ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=no '\ '-p %s -o IdentityFile=%s %s@%s ' \ @@ -525,11 +485,11 @@ BBPATH .= ":${LAYERDIR}"\ module_output = b'Just an example' resize_output = None image_fstypes, \ - wks_file, \ - bbdistro = self.getVars('IMAGE_FSTYPES', \ - 'WKS_FILE', \ - 'DISTRO', \ - target=multiconfig) + wks_file, \ + bbdistro = CIUtils.getVars('IMAGE_FSTYPES', + 'WKS_FILE', + 'DISTRO', + target=multiconfig) # only the first type will be tested in start_vm if image_fstypes.split()[0] == 'wic': diff --git a/testsuite/citest.py b/testsuite/citest.py index 42d44f6a..b84ae0e1 100755 --- a/testsuite/citest.py +++ b/testsuite/citest.py @@ -5,6 +5,7 @@ import os from avocado import skipUnless from avocado.utils import path from cibase import CIBaseTest +from utils import CIUtils UMOCI_AVAILABLE = True SKOPEO_AVAILABLE = True @@ -17,6 +18,7 @@ try: except path.CmdNotFoundError: SKOPEO_AVAILABLE = False + class DevTest(CIBaseTest): """ @@ -46,7 +48,7 @@ class DevTest(CIBaseTest): def test_dev_rebuild(self): self.init() - layerdir_core = self.getVars('LAYERDIR_core') + layerdir_core = CIUtils.getVars('LAYERDIR_core') dpkgbase_file = layerdir_core + '/classes/dpkg-base.bbclass' diff --git a/testsuite/repro-build-test.py b/testsuite/repro-build-test.py index 843f3bc6..04e4ddc7 100755 --- a/testsuite/repro-build-test.py +++ b/testsuite/repro-build-test.py @@ -1,6 +1,7 @@ #!/usr/bin/env python3 from cibuilder import CIBuilder +from utils import CIUtils from avocado.utils import process @@ -29,7 +30,8 @@ class ReproBuild(CIBuilder): def get_image_path(self, target_name): image_dir = "tmp/deploy/images" - machine, image_name = self.getVars("MACHINE", "IMAGE_FULLNAME", target=target_name) + machine, image_name = CIUtils.getVars('MACHINE', 'IMAGE_FULLNAME', + target=target_name) return f"{image_dir}/{machine}/{image_name}.tar.gz" def build_repro_image( diff --git a/testsuite/utils.py b/testsuite/utils.py new file mode 100755 index 00000000..ffcb5cc7 --- /dev/null +++ b/testsuite/utils.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 + +import os +import sys +import tarfile +import time + +sys.path.append(os.path.join(os.path.dirname(__file__), '../bitbake/lib')) + +import bb +import bb.tinfoil + + +class CIUtils(): + @staticmethod + def getVars(*vars, target=None): + def fixStream(stream): + # fix stream objects to emulate _io.TextIOWrapper + stream.isatty = lambda: False + stream.fileno = lambda: False + stream.encoding = sys.getdefaultencoding() + + sl = target is not None + if not hasattr(sys.stdout, 'isatty'): + fixStream(sys.stdout) + if not hasattr(sys.stderr, 'isatty'): + fixStream(sys.stderr) + + # wait until previous bitbake will be finished + lockfile = os.path.join(os.getcwd(), 'bitbake.lock') + checks = 0 + while os.path.exists(lockfile) and checks < 5: + time.sleep(1) + checks += 1 + + with bb.tinfoil.Tinfoil(setup_logging=sl) as tinfoil: + values = () + if target: + tinfoil.prepare(quiet=2) + d = tinfoil.parse_recipe(target) + for var in vars: + values = values + (d.getVar(var, True) or '',) + else: + tinfoil.prepare(config_only=True, quiet=2) + for var in vars: + values = values + (tinfoil.config_data.getVar(var) or '',) + return values if len(values) > 1 else values[0] + + @staticmethod + def get_tar_content(filename): + try: + tar = tarfile.open(filename) + return tar.getnames() + except Exception: + return []