From patchwork Fri Jun 14 11:34:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Mikanovich X-Patchwork-Id: 3606 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 14 Jun 2024 13:34:47 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f56.google.com (mail-wm1-f56.google.com [209.85.128.56]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 45EBYjdO032088 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 14 Jun 2024 13:34:45 +0200 Received: by mail-wm1-f56.google.com with SMTP id 5b1f17b1804b1-42180d2a0d6sf3244395e9.1 for ; Fri, 14 Jun 2024 04:34:45 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1718364880; cv=pass; d=google.com; s=arc-20160816; b=RgtYclReBVTnflvPguAJ+JUaoQGFJv1lG/aZlISHq+fxAJyD97sIGHUT00AP78V1qW qOsH5aYEoX1+HqyISZEcix7yfkWCZm0SPVLtVg6uaVhyqhrUEZXnAN5kEYEio39mPtk8 TouCkqRYxwRCF43KuUUFpsewdNuhCPJKiyVuoglDk+5qMITUmmO2lwmwgVeI04xL1F/l TVSOp9OkjSdHho+ThZusijeAvHtC01f5V2Wd8k7Dj1dxZlp+RBa0LR10ZncvfxWeh72V lhLFLxQjdGHYluJYkkLYaik1tIYnTUUzPKHPxITBsivIVXecoZeAG1R8AZ4n9ejmxWoW 2IQQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=cG+GDJ8kiICOrYECAkGZxrE8cZvAd3Qn5iD2wrjO/Dk=; fh=Ukc4hVcxY/3Vx7/ZE2P3H8PwmW8Lrg9SjqOYXL0jFm8=; b=JEDrj2O5YdmBQBLrc34pKc5Ej/qK4Zj1kKPGd0/1Ts7BjyN6X3WYrooft1DIemnnc6 vtPBlVCX/QGBdgwp1Cj/GUTHiYj3aKeXjAouiGpddritdGm7uggLUUuEEsKYS3S7qw7u Q0aAq3pX0io2T5rmFIZmhVOvOphfSOE4BrEgEOJk8NRNxCEqE/FuRul7kagNzHNu5dWV xc9amAZLJ7Q0SZuL4ist41ndKttvNjBVBo81Dm/N33AuEuHXfFynV33k+kryXTY4bCWm HRGF15UDZTP087G6n5VUioRK/2FdT3jgISIKyfkEykKMd5lo+X9jIkyTtZ4DA6U+3W0T 4N4A==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1718364880; x=1718969680; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=cG+GDJ8kiICOrYECAkGZxrE8cZvAd3Qn5iD2wrjO/Dk=; b=r3jOWd2ITj8bsTYkMvhc7xoCta9t7+tzoPEzIkPq/xImrpYIQ914tfRbq4HDeILLlg Kf0ZKjxUQbsTwxMTuAsVDbn31aWUjovUrEqJG1NrppEDaawIPuarZuN88ZC/ycD2FJhT +Kb7ba9X7trL2lmHvjY6QRVUr+5Tobx3u6wjVOBE48+ILo/qBreXaW+/F8SBbBh+EKdC 91sXbHENhvktnwQAWfIfe+MkG51o0df9UMlqebKh6+/lzUW2SKKK4LXhvkMrzg/nuswB 8bjEcuBLBQS8BAdioUkv06YOX7jwWrDP1B9r6Fs7yJLWyVWh1bo2kkzvuYFQqG+Nj0Pr HUQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718364880; x=1718969680; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :x-beenthere:x-gm-message-state:sender:from:to:cc:subject:date :message-id:reply-to; bh=cG+GDJ8kiICOrYECAkGZxrE8cZvAd3Qn5iD2wrjO/Dk=; b=GYGKI/y3dhl3sC0u/HDCjUtm5MxXo124JtvDfj9bFjhPP5qXHVedu9zEQgWya/teYQ bvy8hkAhVropDSBT+vwT2fZiLNG5VhxciIU8NxCGQg1kWGrbGyDazquwulB5q4TgezxC 2HL/GjpOKZX35oFXXN4DG01m+IDRgT6llJ3Hcg1z27FD0LNknveP/CmrRrTkSyUopdT8 lEWng6uMwXyvRI0UgBxsI3q/15BY1h5ytmXeXh/MwQmjErqkgx7eIYchGBRQye2OWsCy p7eTpU+gB1Qfj/3Qino3jA/A3+kHB3EvrgmVwrtAerx6JiuhIIcKnxyOrFhZ1VnBwuTu SfRQ== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCWqDueP5G6ksye8Qdn13q+8OOtGhgEGTdnItTVII3B3czxMO8PSd1/lRrWJlIbiOI0hMY5xrahLTef82nJQSq9pRSyd2Gc= X-Gm-Message-State: AOJu0Yz9zWmgU0PK9mNNU7sp6IJY+RJGmAuOYqrtUhydZ1aakpdEAHxJ wkkPOb3tEjdXWjgaKTwo4++eRy4ysuJeAdUloYb9vqvMTT4aJMPd X-Google-Smtp-Source: AGHT+IF1xwHVUgvPJvTZnp0N4gEonuPuWAMGQK+D/+oX35uJhzf/qHs/VxOwwLoVRknYirjX4oJ4tw== X-Received: by 2002:a5d:6da4:0:b0:35f:2287:7b97 with SMTP id ffacd0b85a97d-3607a7804efmr1835116f8f.4.1718364879529; Fri, 14 Jun 2024 04:34:39 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:5704:0:b0:357:d332:f77 with SMTP id ffacd0b85a97d-3607194f3ddls898197f8f.1.-pod-prod-05-eu; Fri, 14 Jun 2024 04:34:37 -0700 (PDT) X-Received: by 2002:adf:f54d:0:b0:35f:c82:e7b5 with SMTP id ffacd0b85a97d-3607a768bbemr2264638f8f.42.1718364877531; Fri, 14 Jun 2024 04:34:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1718364877; cv=none; d=google.com; s=arc-20160816; b=zB6sG/+NQZyxZ5IIEqQ3SizJkFcxBZ+AR6eYmgnR5e+QVsn3nwXnjk1jYnTmeC8nB8 a7LhPFZniPBjLG19M3joQQPc7NPXkCxml1UPZLBCf3dXGdGYHpS8P3eUWkBPTkk4ZuUU 4oUsMzTbMuJeHqmtEcIpuNQMmJ6zt9noEDNZcEGJcKlLHqrgoI6b9rHi8vwGQnsDbwcB 9xVAdqOuAAnUt+f5QXL3lkLXc926N5FX0h+tI0b2HyFHAj7jzmjDnad3QIU8/q48prxU cpC8rKrnmXCo6JM+YJpbJdPjzZi5nJQo7DnLJE2qmqR/RWf3iZDUcGsbG36Ymr4PDsPf 2JQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=+YZUR7d/Gifuq1dptnHD4e1ThZoIgNK0nMRC7vEtAMs=; fh=/h9QQkzJ8EboVkWg45aWwpaUro6WMavIVd2OhN45RtE=; b=Bf/+iZP/kYdHTVxUtQOmRBg5sLFEgEEhfYgV/Nmc3lX0BMW8ogSS9zbbmaTx2uqpnJ INvCbOTKkx3eOIly6UEKvPGbyLF+M5XwqXFhoh2iEvRNnLTXwDWvHTuvu10e/ScQ3m/R CZGotuh/X7VzKTtDyIFM2Jo2IzFJQOVfHb1yiH4H+Qb3T/QK3QBiS0Cvp5OpuELmui91 wEkq+w3tUzItKe8f2ROV6gWECEOWS1TZcuUvVerqSIY/mBneaM6UtImDv1SSepxxE8sa ufNCb3u3oPEUGESoIkLRWy9oBUW+wt7GYwOxi5lJSRiwMkBsH5VekawPv74HvGL0SAC7 1K2g==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id ffacd0b85a97d-36074e0e07fsi74396f8f.0.2024.06.14.04.34.37 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 14 Jun 2024 04:34:37 -0700 (PDT) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (r154-240-38-77-broadband.btv.lv [77.38.240.154] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 45EBYZOw032043 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 14 Jun 2024 13:34:36 +0200 From: Anton Mikanovich To: isar-users@googlegroups.com Cc: Anton Mikanovich Subject: [PATCH v9 1/8] isar-bootstrap: Move common parts to bbclass Date: Fri, 14 Jun 2024 14:34:18 +0300 Message-Id: <20240614113425.122722-2-amikan@ilbers.de> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240614113425.122722-1-amikan@ilbers.de> References: <20240614113425.122722-1-amikan@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: amikan@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= To allow implementing other bootstrap recipes move some functionality supposed to be reusable to bbclass. Signed-off-by: Anton Mikanovich --- meta/classes/bootstrap.bbclass | 228 ++++++++++++++++++ .../isar-bootstrap/isar-bootstrap.inc | 221 +---------------- 2 files changed, 230 insertions(+), 219 deletions(-) create mode 100644 meta/classes/bootstrap.bbclass diff --git a/meta/classes/bootstrap.bbclass b/meta/classes/bootstrap.bbclass new file mode 100644 index 00000000..b392677a --- /dev/null +++ b/meta/classes/bootstrap.bbclass @@ -0,0 +1,228 @@ +# This software is a part of Isar. +# Copyright (c) Siemens AG, 2018-2021 +# Copyright (C) 2024 ilbers GmbH +# +# SPDX-License-Identifier: MIT + +FILESEXTRAPATHS:append = ":${BBPATH}" + +SRC_URI = " \ + file://locale \ + file://chroot-setup.sh \ + ${DISTRO_BOOTSTRAP_KEYS} \ + ${THIRD_PARTY_APT_KEYS}" + +BOOTSTRAP_FOR_HOST ?= "0" + +APTPREFS = "${WORKDIR}/apt-preferences" +APTSRCS = "${WORKDIR}/apt-sources" +APTSRCS_INIT = "${WORKDIR}/apt-sources-init" +DISTRO_BOOTSTRAP_KEYFILES = "" +THIRD_PARTY_APT_KEYFILES = "" +DISTRO_BOOTSTRAP_KEYS ?= "" +THIRD_PARTY_APT_KEYS ?= "" +DEPLOY_ISAR_BOOTSTRAP ?= "" +DISTRO_BOOTSTRAP_BASE_PACKAGES ??= "" +DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else ''}" +BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}" +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}" + +# reproducible builds, only enabled if ISAR_USE_APT_SNAPSHOT +ISAR_APT_SNAPSHOT_MIRROR ??= "" +APT_SNAPSHOT_DATE = "${@ get_apt_snapshot_date(d)}" + +python () { + distro_bootstrap_keys = (d.getVar("DISTRO_BOOTSTRAP_KEYS") or "").split() + third_party_apt_keys = (d.getVar("THIRD_PARTY_APT_KEYS") or "").split() + topdir = d.getVar("TOPDIR") + + # The cached repo key can be both for bootstrapping and apt package + # installation afterwards. However, bootstrap will include the key into + # the rootfs automatically thus the right place is distro_bootstrap_keys. + + if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): + own_pub_key = d.getVar("BASE_REPO_KEY") + if own_pub_key: + distro_bootstrap_keys += own_pub_key.split() + + for key in distro_bootstrap_keys: + fetcher = bb.fetch2.Fetch([key], d) + filename = os.path.relpath(fetcher.localpath(key), topdir) + d.appendVar("DISTRO_BOOTSTRAP_KEYFILES", " ${TOPDIR}/%s" % filename) + + for key in third_party_apt_keys: + fetcher = bb.fetch2.Fetch([key], d) + filename = os.path.relpath(fetcher.localpath(key), topdir) + d.appendVar("THIRD_PARTY_APT_KEYFILES", " ${TOPDIR}/%s" % filename) + + distro_apt_sources = get_aptsources_list(d) + for file in distro_apt_sources: + d.appendVar("SRC_URI", " file://%s" % file) + + distro_apt_preferences = d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_PREFERENCES") or "" + for file in distro_apt_preferences.split(): + d.appendVar("SRC_URI", " file://%s" % file) +} + +def aggregate_files(d, file_list, file_out): + import shutil + + with open(file_out, "wb") as out_fd: + for entry in file_list: + entry_real = bb.parse.resolve_file(entry, d) + with open(entry_real, "rb") as in_fd: + shutil.copyfileobj(in_fd, out_fd, 1024*1024*10) + out_fd.write("\n".encode()) + +def parse_aptsources_list_line(source_list_line): + import re + + s = source_list_line.strip() + + if not s or s.startswith("#"): + return None + + type, s = re.split("\s+", s, maxsplit=1) + if type not in ["deb", "deb-src"]: + return None + + options = "" + options_match = re.match("\[\s*(\S+=\S+(?=\s))*\s*(\S+=\S+)\s*\]\s+", s) + if options_match: + options = options_match.group(0).strip() + s = s[options_match.end():] + + source, s = re.split("\s+", s, maxsplit=1) + + if s.startswith("/"): + suite = "" + else: + suite, s = re.split("\s+", s, maxsplit=1) + + components = " ".join(s.split()) + + return [type, options, source, suite, components] + +def get_apt_snapshot_date(d): + import time + source_date_epoch = d.getVar('ISAR_APT_SNAPSHOT_TIMESTAMP') + return time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch))) + +def get_apt_source_mirror(d, aptsources_entry_list): + import re + + # this is executed during parsing. No error checking possible + use_snapshot = bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) + snapshot_mirror = d.getVar('DISTRO_APT_SNAPSHOT_PREMIRROR') + if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): + premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" + elif use_snapshot and snapshot_mirror: + premirrors = snapshot_mirror + else: + premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or "" + mirror_list = [entry.split() + for entry in premirrors.split('\\n') + if any(entry)] + + for regex, replace in mirror_list: + match = re.search(regex, aptsources_entry_list[2]) + + if match: + new_aptsources_entry_list = aptsources_entry_list.copy() + new_aptsources_entry_list[2] = re.sub(regex, replace, + aptsources_entry_list[2], + count = 1) + if use_snapshot: + new_aptsources_entry_list[1] = "[check-valid-until=no]" + return new_aptsources_entry_list + + return aptsources_entry_list + +def aggregate_aptsources_list(d, file_list, file_out): + import shutil + + with open(file_out, "wb") as out_fd: + for entry in file_list: + entry_real = bb.parse.resolve_file(entry, d) + with open(entry_real, "r") as in_fd: + for line in in_fd: + parsed = parse_aptsources_list_line(line) + if parsed: + parsed = get_apt_source_mirror(d, parsed) + out_fd.write(" ".join(parsed).encode()) + else: + out_fd.write(line.encode()) + out_fd.write("\n".encode()) + out_fd.write("\n".encode()) + +def get_aptsources_list(d): + import errno + from collections import OrderedDict + apt_sources_var = d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_SOURCES" + apt_sources_list = list(OrderedDict.fromkeys((d.getVar(apt_sources_var) or "").split())) + for p in apt_sources_list: + try: + bb.parse.resolve_file(p, d) + except FileNotFoundError as e: + bb.fatal(os.strerror(errno.ENOENT) + ' "' + p + '"') + return apt_sources_list + +def generate_distro_sources(d): + apt_sources_list = get_aptsources_list(d) + for entry in apt_sources_list: + with open(bb.parse.resolve_file(entry, d), "r") as in_fd: + for line in in_fd: + parsed = parse_aptsources_list_line(line) + if parsed: + parsed = get_apt_source_mirror(d, parsed) + yield parsed + +def get_distro_primary_source_entry(d): + for source in generate_distro_sources(d): + if source[0] == "deb": + return source[2:] + bb.fatal('Invalid apt sources list') + +def get_distro_source(d): + return get_distro_primary_source_entry(d)[0] + +def get_distro_suite(d): + return get_distro_primary_source_entry(d)[1] + +def get_distro_components_argument(d): + components = get_distro_primary_source_entry(d)[2] + if components and components.strip(): + return "--components=" + ",".join(components.split()) + else: + return "" + +do_apt_config_prepare[dirs] = "${WORKDIR}" +do_apt_config_prepare[vardeps] += " \ + APTPREFS \ + ${DISTRO_VARS_PREFIX}DISTRO_APT_PREFERENCES \ + DEBDISTRONAME \ + APTSRCS \ + ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \ + DEPLOY_ISAR_BOOTSTRAP \ + ${@'DISTRO_APT_SNAPSHOT_PREMIRROR' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''} \ + " +python do_apt_config_prepare() { + apt_preferences_out = d.getVar("APTPREFS") + apt_preferences_list = ( + d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_PREFERENCES") or "" + ).split() + aggregate_files(d, apt_preferences_list, apt_preferences_out) + + apt_sources_out = d.getVar("APTSRCS") + apt_sources_init_out = d.getVar("APTSRCS_INIT") + apt_sources_list = get_aptsources_list(d) + + aggregate_files(d, apt_sources_list, apt_sources_init_out) + aggregate_aptsources_list(d, apt_sources_list, apt_sources_out) +} +addtask apt_config_prepare before do_bootstrap after do_unpack + +CLEANFUNCS = "clean_deploy" +clean_deploy() { + rm -f "${DEPLOY_ISAR_BOOTSTRAP}" +} diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index faf22a50..5c9eb5c8 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -5,193 +5,20 @@ # # SPDX-License-Identifier: MIT +inherit bootstrap + LICENSE = "gpl-2.0" LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" -FILESPATH:prepend := "${THISDIR}/files:" -SRC_URI = " \ - file://locale \ - file://chroot-setup.sh \ - ${DISTRO_BOOTSTRAP_KEYS} \ - ${THIRD_PARTY_APT_KEYS}" PV = "1.0" -BOOTSTRAP_FOR_HOST ?= "0" - DEBOOTSTRAP ?= "qemu-debootstrap" ROOTFSDIR = "${WORKDIR}/rootfs" -APTPREFS = "${WORKDIR}/apt-preferences" -APTSRCS = "${WORKDIR}/apt-sources" -APTSRCS_INIT = "${WORKDIR}/apt-sources-init" -DISTRO_BOOTSTRAP_KEYFILES = "" -THIRD_PARTY_APT_KEYFILES = "" -DISTRO_BOOTSTRAP_KEYS ?= "" -THIRD_PARTY_APT_KEYS ?= "" -DEPLOY_ISAR_BOOTSTRAP ?= "" DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales" DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = ",gnupg" DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates" -DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else ''}" -BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}" -BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}" -FILESEXTRAPATHS:append = ":${BBPATH}" -# reproducible builds, only enabled if ISAR_USE_APT_SNAPSHOT -ISAR_APT_SNAPSHOT_MIRROR ??= "" -APT_SNAPSHOT_DATE = "${@ get_apt_snapshot_date(d)}" inherit deb-dl-dir -python () { - distro_bootstrap_keys = (d.getVar("DISTRO_BOOTSTRAP_KEYS") or "").split() - third_party_apt_keys = (d.getVar("THIRD_PARTY_APT_KEYS") or "").split() - topdir = d.getVar("TOPDIR") - - # The cached repo key can be both for bootstrapping and apt package - # installation afterwards. However, debootstrap will include the key into - # the rootfs automatically thus the right place is distro_bootstrap_keys. - - if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): - own_pub_key = d.getVar("BASE_REPO_KEY") - if own_pub_key: - distro_bootstrap_keys += own_pub_key.split() - - for key in distro_bootstrap_keys: - fetcher = bb.fetch2.Fetch([key], d) - filename = os.path.relpath(fetcher.localpath(key), topdir) - d.appendVar("DISTRO_BOOTSTRAP_KEYFILES", " ${TOPDIR}/%s" % filename) - - for key in third_party_apt_keys: - fetcher = bb.fetch2.Fetch([key], d) - filename = os.path.relpath(fetcher.localpath(key), topdir) - d.appendVar("THIRD_PARTY_APT_KEYFILES", " ${TOPDIR}/%s" % filename) - - distro_apt_sources = get_aptsources_list(d) - for file in distro_apt_sources: - d.appendVar("SRC_URI", " file://%s" % file) - - distro_apt_preferences = d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_PREFERENCES") or "" - for file in distro_apt_preferences.split(): - d.appendVar("SRC_URI", " file://%s" % file) -} - -def aggregate_files(d, file_list, file_out): - import shutil - - with open(file_out, "wb") as out_fd: - for entry in file_list: - entry_real = bb.parse.resolve_file(entry, d) - with open(entry_real, "rb") as in_fd: - shutil.copyfileobj(in_fd, out_fd, 1024*1024*10) - out_fd.write("\n".encode()) - -def parse_aptsources_list_line(source_list_line): - import re - - s = source_list_line.strip() - - if not s or s.startswith("#"): - return None - - type, s = re.split("\s+", s, maxsplit=1) - if type not in ["deb", "deb-src"]: - return None - - options = "" - options_match = re.match("\[\s*(\S+=\S+(?=\s))*\s*(\S+=\S+)\s*\]\s+", s) - if options_match: - options = options_match.group(0).strip() - s = s[options_match.end():] - - source, s = re.split("\s+", s, maxsplit=1) - - if s.startswith("/"): - suite = "" - else: - suite, s = re.split("\s+", s, maxsplit=1) - - components = " ".join(s.split()) - - return [type, options, source, suite, components] - -def get_apt_snapshot_date(d): - import time - source_date_epoch = d.getVar('ISAR_APT_SNAPSHOT_TIMESTAMP') - return time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch))) - -def get_apt_source_mirror(d, aptsources_entry_list): - import re - - # this is executed during parsing. No error checking possible - use_snapshot = bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) - snapshot_mirror = d.getVar('DISTRO_APT_SNAPSHOT_PREMIRROR') - if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): - premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" - elif use_snapshot and snapshot_mirror: - premirrors = snapshot_mirror - else: - premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or "" - mirror_list = [entry.split() - for entry in premirrors.split('\\n') - if any(entry)] - - for regex, replace in mirror_list: - match = re.search(regex, aptsources_entry_list[2]) - - if match: - new_aptsources_entry_list = aptsources_entry_list.copy() - new_aptsources_entry_list[2] = re.sub(regex, replace, - aptsources_entry_list[2], - count = 1) - if use_snapshot: - new_aptsources_entry_list[1] = "[check-valid-until=no]" - return new_aptsources_entry_list - - return aptsources_entry_list - -def aggregate_aptsources_list(d, file_list, file_out): - import shutil - - with open(file_out, "wb") as out_fd: - for entry in file_list: - entry_real = bb.parse.resolve_file(entry, d) - with open(entry_real, "r") as in_fd: - for line in in_fd: - parsed = parse_aptsources_list_line(line) - if parsed: - parsed = get_apt_source_mirror(d, parsed) - out_fd.write(" ".join(parsed).encode()) - else: - out_fd.write(line.encode()) - out_fd.write("\n".encode()) - out_fd.write("\n".encode()) - -def get_aptsources_list(d): - import errno - from collections import OrderedDict - apt_sources_var = d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_SOURCES" - apt_sources_list = list(OrderedDict.fromkeys((d.getVar(apt_sources_var) or "").split())) - for p in apt_sources_list: - try: - bb.parse.resolve_file(p, d) - except FileNotFoundError as e: - bb.fatal(os.strerror(errno.ENOENT) + ' "' + p + '"') - return apt_sources_list - -def generate_distro_sources(d): - apt_sources_list = get_aptsources_list(d) - for entry in apt_sources_list: - with open(bb.parse.resolve_file(entry, d), "r") as in_fd: - for line in in_fd: - parsed = parse_aptsources_list_line(line) - if parsed: - parsed = get_apt_source_mirror(d, parsed) - yield parsed - -def get_distro_primary_source_entry(d): - for source in generate_distro_sources(d): - if source[0] == "deb": - return source[2:] - bb.fatal('Invalid apt sources list') - def get_distro_have_https_source(d): return any(source[2].startswith("https://") for source in generate_distro_sources(d)) @@ -213,19 +40,6 @@ def get_distro_needs_gpg_support(d): OVERRIDES:append = ":${@get_distro_needs_gpg_support(d)}" -def get_distro_source(d): - return get_distro_primary_source_entry(d)[0] - -def get_distro_suite(d): - return get_distro_primary_source_entry(d)[1] - -def get_distro_components_argument(d): - components = get_distro_primary_source_entry(d)[2] - if components and components.strip(): - return "--components=" + ",".join(components.split()) - else: - return "" - APT_KEYS_DIR = "${WORKDIR}/aptkeys" DISTRO_BOOTSTRAP_KEYRING = "${WORKDIR}/distro-keyring.gpg" @@ -249,32 +63,6 @@ do_generate_keyrings() { } addtask generate_keyrings before do_build after do_unpack -do_apt_config_prepare[dirs] = "${WORKDIR}" -do_apt_config_prepare[vardeps] += " \ - APTPREFS \ - ${DISTRO_VARS_PREFIX}DISTRO_APT_PREFERENCES \ - DEBDISTRONAME \ - APTSRCS \ - ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \ - DEPLOY_ISAR_BOOTSTRAP \ - ${@'DISTRO_APT_SNAPSHOT_PREMIRROR' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''} \ - " -python do_apt_config_prepare() { - apt_preferences_out = d.getVar("APTPREFS") - apt_preferences_list = ( - d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_PREFERENCES") or "" - ).split() - aggregate_files(d, apt_preferences_list, apt_preferences_out) - - apt_sources_out = d.getVar("APTSRCS") - apt_sources_init_out = d.getVar("APTSRCS_INIT") - apt_sources_list = get_aptsources_list(d) - - aggregate_files(d, apt_sources_list, apt_sources_init_out) - aggregate_aptsources_list(d, apt_sources_list, apt_sources_out) -} -addtask apt_config_prepare before do_bootstrap after do_unpack - def get_host_release(): import platform rel = platform.release() @@ -446,8 +234,3 @@ python do_bootstrap_setscene() { addtask do_bootstrap_setscene do_bootstrap_setscene[dirs] = "${DEPLOY_DIR_BOOTSTRAP}" - -CLEANFUNCS = "clean_deploy" -clean_deploy() { - rm -f "${DEPLOY_ISAR_BOOTSTRAP}" -}