From patchwork Thu Feb 24 21:40:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Mikanovich X-Patchwork-Id: 1534 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3) with LMTPA; Fri, 25 Feb 2022 08:41:04 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f56.google.com (mail-wm1-f56.google.com [209.85.128.56]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPS id 21P7f3es001633 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 25 Feb 2022 08:41:03 +0100 Received: by mail-wm1-f56.google.com with SMTP id v125-20020a1cac83000000b0037e3d70e7e1sf1186700wme.1 for ; Thu, 24 Feb 2022 23:41:03 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1645774858; cv=pass; d=google.com; s=arc-20160816; b=To0JNR+6YCIgauCh7/SP2S/6JSy/bXrbfBV37JqRexqU3r7kb+bSy52es42fEKcCkc T/ZVLqoGwD8Ex7lA5sI6dxjuhlqL44UzV7R8unnIQ1NNg7HXz4/lkZDWobjzdkfzM0Wu F3gEc9bp34cRmmPHENkC8T7LTTymDFGUx++PfMgQrT1BFMnJc6SPqm+p1j7nHCYe4Hsb hGEhx/D3kVPDvi3ukRk4Ir76WSKSLawEuauehqPg1rVH9R8vrwnbfJLdxh/QfW+Nl/5J 11m/+j3cE+6kG+BGqNvwESv3hgYJF6vx7bTMrO6reFNz6tVANlVoH7G9+ZPX0tU/kK3L 3fXw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=SO1IRaZREGL4jDLx2nDwyt0pvyzmkPawpM9LLZLilXE=; b=nLXG79c+FxPFTFM5W1wl3PulP4uNEDt5ITkH1t+K4GC5BJpkMW5lEE11KySevJhpUu NbBl9ked5wkbjB0sH5UY3qTxLU77ilMcDD0AacYbe6APLGVKcA1QxxUH25MV6ifDvD1m 01LdbVg95tSxj5FWi8o3/g6Z0zCK4HXVhzMSNiDc0/oZhD8rfcRI0b1xhnLYmOINjiPW bz8kkT+2CaYX9RJFlOCWJ3YDvJPrgTayuE6Z1g61Z9JKHMnDs2u1+xMfiQSF7VxTwcu7 sXR3LodUHhS2PlIqQ4jSYlI0ieoKbg+4OX1qCsry+aH5vKzup3dhWjfwi4nsMzA2DONM xzSQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=SO1IRaZREGL4jDLx2nDwyt0pvyzmkPawpM9LLZLilXE=; b=FLd3vVZM0QhEf+Kq7grJqMFJilNNTGBJtximAwc9lTWoVnm7hAYATur2BlyV70lcQ1 5QAP14F2s467Hq2LjdR52mPYErTvrQHIk2707mBkahqngRBoNFllRvCmc2UnAG9kCjyG javiaoo/dr2MB7q/fXRzI7EQEU/vzNNxPHkL/+q6Isvw7UGi0DZTfgJO1ohBBwJV0OZp Xz9ZReptNd6kmGeboKRstYMPI5RX5eI8jBJgkDcSz0u/InJw8Tl1Hw93onmRi7Y+Vsg+ DODmpVM0pG/ZATN83f6H/4jhV4yecgHqnV/je0DwFHj0bUKWVst3WiG6gPzvHqac5dhR 5VRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=SO1IRaZREGL4jDLx2nDwyt0pvyzmkPawpM9LLZLilXE=; b=VKO/FBBz6A3SuTs6KadQ6/hIDrqOB/uPIiIAJjCHRiqXG2VDMa7mV490aZSWaLuRPW iwa8iRWUz6cBK91y1rFsPwUumUwPJHCMirLmA/VFoVzcYVDIF9IcgaIn7M+/R/6tkq+3 O3iw972widhneZNDlRNkh7/ZzTEaMxbj3f4KHxPFeBfQSfl/ppJI+J2oEApKd/jrlWm8 T0eo8himK7FMrqjBjjxoxA98+lwKloFmTIYgekgFtIHnBHz7x8cVZZTAdfZinSnJlp9F 32ro+ldoUryHgm0gsZ7EjGiI4KoSruAegfNnNK5tfANZ8/GhlSrQ1lbqDiLLQCAlX7qS m5+g== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM532YAj2WdtpU12IHqS/MNJ1ikVHWJPCE7FM2AZsAVGWD7V23nK6L g8UF3kxs1Uv71eqoJGEzn6Y= X-Google-Smtp-Source: ABdhPJwqhtgGyZxIwzav2XDl722AO9b1bHNYyov1oK3NfLgl7fZ48o9gUVOjVQsvJ9qmg5YJxnp2JQ== X-Received: by 2002:adf:e94a:0:b0:1ea:9bdf:d2c4 with SMTP id m10-20020adfe94a000000b001ea9bdfd2c4mr5089102wrn.176.1645774858537; Thu, 24 Feb 2022 23:40:58 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:1554:b0:37c:d848:5b20 with SMTP id f20-20020a05600c155400b0037cd8485b20ls34865wmg.1.gmail; Thu, 24 Feb 2022 23:40:57 -0800 (PST) X-Received: by 2002:a5d:440c:0:b0:1ee:1c9a:193e with SMTP id z12-20020a5d440c000000b001ee1c9a193emr4252883wrq.523.1645774857643; Thu, 24 Feb 2022 23:40:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1645774857; cv=none; d=google.com; s=arc-20160816; b=SgwxfDQ68vg6QKVsC+nKQuFg6XEpaRdfkmgW4zms7+edX7ZH4NKrNEDUN9OJF1d+Vu Cf3DFDIOu0MrHqKBPITS24HkyNe25q9bkwUPYxQ23yuzgwl+WQ5nMn0fX77ccxB03pAd +LU2uAPxkIY6+a2o0XK1eXK7AmEJ4/vciNWpSF5qhHanS7Ij/pjzJqI9ec2bBsdG68l1 N/uQdwA81EWY2bLlCOTXA9Gu/spDkaiBBKm6d4aFWQloBSPRy/MUyKkr94Hb3xflyUy8 vOV8lXjBly3l4topsG993yI0GovN3o2L1b2/aUrvhGWHixkALs7dJcr4AOX7jKYQrNXn tqZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=qR7otDF91wcDWnrpPms9Ewmm1uYiJtuKtFkbpC7nz/0=; b=kI+cFSGrEif8jQhMJYuDSLFlUC1IBZ/HoooUctLa/4MO/464jlCbenhctA4I6fzUbZ 1VWlC2ug8Qd9rEnoBjdSRcMLRAT8lnfAhcnAxCeVjhesD1CCjfxOUv93sYXjefS/jVxo WB3hVzAeWJhb+J7xmTl4lp9e9Uv/QHxmyBRTh1AJqV9R5z1daf1uF3g7SMhhhmAo+Z6+ FLQycBTBXzoIi8VlA0tEyo+PvuNg28hofPoFJ6m/e+lie8UClLU+pax9AzX+PdooOTKY aDuHZEaW74IcmFHPO0HObEazfrZYktXwqEmrNm0mHZWUkRdF6Q1U/9zOgAklvuH2r/xk 15mw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id bg3-20020a05600c3c8300b0037e391f947bsi474809wmb.4.2022.02.24.23.40.57 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 24 Feb 2022 23:40:57 -0800 (PST) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (mm-5-27-214-37.mgts.dynamic.pppoe.byfly.by [37.214.27.5] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 21P7emqt001558 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 25 Feb 2022 08:40:56 +0100 From: Anton Mikanovich To: isar-users@googlegroups.com Cc: Anton Mikanovich Subject: [RFC 5/5] isar: Add external builder Date: Fri, 25 Feb 2022 10:40:40 +0300 Message-Id: <20220225074040.20975-6-amikan@ilbers.de> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220225074040.20975-1-amikan@ilbers.de> References: <20220225074040.20975-1-amikan@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: amikan@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: INBOX Build Isar with external tool to parce all Debian package dependencies. Support only one target build for now, mc:qemuamd64-bullseye:isar-image-base is used by default. Usage (inside build dir): $ ../isar_builder.py Signed-off-by: Anton Mikanovich --- isar_builder.py | 142 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100755 isar_builder.py diff --git a/isar_builder.py b/isar_builder.py new file mode 100755 index 0000000..cf99c52 --- /dev/null +++ b/isar_builder.py @@ -0,0 +1,142 @@ +#!/usr/bin/env python3 + +import os +import sys +import apt_pkg +import apt.progress.base + +sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)) + '/bitbake/lib') + +import bb.tinfoil + +class IsarBuilder: + def __init__(self): + self.tinfoil = bb.tinfoil.Tinfoil() + self.tinfoil.prepare() + + def create_apt_rootfs(self, rootfs): + repo_isar_dir = '%s/tmp/deploy/isar-apt/%s-%s/apt' % (os.path.abspath(os.getcwd()), self.distro, self.arch) + mirror = 'file://%s/%s' % (repo_isar_dir, self.distro) + codename = 'isar' + + if not os.path.exists(rootfs + '/var/lib/dpkg'): + os.makedirs(rootfs + '/var/lib/dpkg') + open(rootfs + '/var/lib/dpkg/status', 'w').close() + + if not os.path.exists(rootfs + '/etc/apt/preferences.d'): + os.makedirs(rootfs + '/etc/apt/preferences.d') + with open(rootfs + '/etc/apt/sources.list', 'w') as f: + f.write('deb [arch=%s] %s %s main\n' % (self.arch, mirror, codename)) + f.write('deb-src [arch=%s] %s %s main\n' % (self.arch, mirror, codename)) + + if not os.path.exists(rootfs + '/var/cache/apt/archives/partial'): + os.makedirs(rootfs + '/var/cache/apt/archives/partial') + + def apt_config(self): + apt_pkg.init() + + rootfs = os.path.abspath(os.getcwd()) + '/tmp/deps_poc_rootfs/%s-%s' % (self.distro, self.arch) + + if not os.path.isdir(rootfs): + self.create_apt_rootfs(rootfs) + + apt_pkg.config.set('APT::Architecture', self.arch) + apt_pkg.config.set('Dir', rootfs) + apt_pkg.config.set('Dir::Cache', rootfs + '/var/cache/apt') + apt_pkg.config.set('Dir::State::status', rootfs + '/var/lib/dpkg/status') + + apt_pkg.config.set("Acquire::AllowInsecureRepositories", "1") + + def isar_apt_update(self): + sources = apt_pkg.SourceList() + sources.read_main_list() + + progress = apt.progress.text.AcquireProgress() + + self.cache = apt_pkg.Cache() + self.cache.update(progress, sources) + self.cache = apt_pkg.Cache() + + def srcpackage_lookup(self, pkg): + if pkg in self.cache: + return + if self.tinfoil.find_best_provider(pkg)[3] is None: + return + if pkg not in self.target_deps: + self.need_deb.append(pkg) + self.sr.restart() + if self.sr.lookup(pkg) is None: + self.need_source.append(pkg) + return + if 'Build-Depends' not in self.sr.build_depends: + return + for dep in self.sr.build_depends['Build-Depends']: + child = str(dep[0][0]) + if child not in self.checkdeps: + self.checkdeps.append(child) + self.srcpackage_lookup(child) + + def bitbake(self, mc, targets, task='', justbuild=[]): + targets = [targets] if isinstance(targets, str) else targets + target_str = '' + for pn in targets: + target_str += 'mc:%s:%s ' % (mc, pn if not task else ':'.join([pn, task])) + targets.clear() + for pn in justbuild: + target_str += 'mc:%s:%s ' % (mc, pn) + justbuild.clear() + if target_str != '': + print('Building %s' % target_str) + self.tinfoil.build_targets(target_str) + + def build(self, mc, target): + d = self.tinfoil.parse_recipe('multiconfig:%s:%s' % (mc, target)) + self.distro = d.getVar('DISTRO', expand=True) + self.arch = d.getVar('DISTRO_ARCH', expand=True) + + recipecache = self.tinfoil.cooker.recipecaches[mc] + provider = self.tinfoil.find_best_provider('multiconfig:%s:%s' % (mc, target)) + self.target_deps = recipecache.deps[provider[3]] + + kernel_image = d.getVar('KERNEL_IMAGE_PKG', expand=True) or '' + if kernel_image in self.target_deps: + kernel = self.tinfoil.find_best_provider('multiconfig:%s:%s' % (mc, kernel_image)) + if kernel[3].endswith('linux-distro.bb'): + self.target_deps.remove(kernel_image) + print('Remove %s from checking' % kernel_image) + + self.need_source = self.target_deps.copy() + self.need_deb = [] + + self.apt_config() + + maxdepth = 3 + while maxdepth > 0: + maxdepth -= 1 + + self.bitbake(mc, self.need_source, 'do_deploy_source', self.need_deb) + + self.isar_apt_update() + self.sr = apt_pkg.SourceRecords() + + self.checkdeps = [] + for dep in self.target_deps: + if dep not in self.checkdeps: + self.checkdeps.append(dep) + self.srcpackage_lookup(dep) + + if not self.need_source: + break + + if self.need_source: + print('Following packages still left unchecked: ' + str(self.need_source)) + + if self.need_deb: + print('Additional Debian dependencies found:' + str(self.need_deb)) + self.bitbake(mc, self.need_deb) + + # start final build + self.bitbake(mc, target) + +ib = IsarBuilder() +ib.build('qemuamd64-bullseye', 'isar-image-base')