From patchwork Thu Mar 14 07:27:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3402 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 14 Mar 2024 08:31:16 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f185.google.com (mail-lj1-f185.google.com [209.85.208.185]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 42E7V1Ov021547 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:31:01 +0100 Received: by mail-lj1-f185.google.com with SMTP id 38308e7fff4ca-2d46c51b340sf5376611fa.1 for ; Thu, 14 Mar 2024 00:31:01 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710401456; cv=pass; d=google.com; s=arc-20160816; b=NgXX6Nq//kj5xKowBDGiWBYhUqAASKI6SnMYEyCHzjdh5FFfDlWEwR3batBF8vK7Vz S/2hOBr8gLHlGYcsaWJeJJNxj0ZanquALruHRcqlyQjiub4u2y6KWFvLWmeMU23KyL3g UzaEp9ii+X9+GvQ7Ct14311jBOWEgBNZc7U7LADxBSE7voZceJiykXmkNvNOiXcAnQOS TAifqCIAVinBzhfzcH4gvtZ9iQjyxqe9Zl+2UttmBfYOXUNdHVCuVYWExk5ka3/wY7qV YZBTFEPBRXU4uQholUM4bsL7vgq6FOnDbOv4t6BqSliECQY3ksIMJiu3CztWcG8fdO4h pTqQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=t+1psRWZotLZ+FiXL32aj1d6gO1nfljyev/lCTmevOc=; fh=78TC2WMwWRa05i+nhWeXMgE7R9ZLu1PnvWklkqa6ulI=; b=mlz/72CjXFCjwuXs2f9EyZ7rtJYuN0tHiRaRQsfZSSSubwnHz1gcTR8tXMNOUg2fKY nE6Jku62nSHKImukcobJDWFCa0fRh/KEJgrRFyCccBkBDTji4CSxntWy5/oX8DJwXQ3n +0/0XMpLyM1VlnXVXS8Xyar46QMUQ3YO0EqWEBEeQ2zNLaIGghWArXwCZAkDrPIxN6KS X1STxbp9nMRUvcR0bDleImZ5ovPz/DOQEraGec7lgT8HqGzR4/S1JFHjql9ukEOTzxg0 Bd7coHWYe0IMmZKFRORmJZUosu26cgLcQPQO15755TiF92R4oEov5Pwcz4YQ3z4iJKZr r5BQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1710401456; x=1711006256; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=t+1psRWZotLZ+FiXL32aj1d6gO1nfljyev/lCTmevOc=; b=kpo3R7WZajL8bK1DA+/Iz9WFSogxomp7UH+l8hSLVCA7U0YZW51eGYP/p507tWuY/Q QGbticESU2KsfbDoXJIjg0iKCDD7XJ6zIUu2dtiHizpMFt8Pcz2v99+V9sbIhZLPU/3+ EDiEC1NlNQBJxm9w9wEHeV6zTYnEas1PzP3ZOHuKqruIDfd5l+tTNgOJBI8fpNlZ2RAM e3eo/eoxXR0KKjFwGZshDx7lrDjcfFC+uhK2AJGSvbjqMgpfSA1mFyEY+KkbURyxbJVx RcnSqm99PxyGU0hsJCBqtuDIvtKSCIYYxoplbM7udjD4z7WIL+sHUSJnfwNnYpcEuBC4 eKJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710401456; x=1711006256; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=t+1psRWZotLZ+FiXL32aj1d6gO1nfljyev/lCTmevOc=; b=hrvH6RuwmFB+rHvZk8K1MWRSjLoGnV6VhJVkfK6wJTw/uTGlqioK+Qok2gSVU2B6eI iuZW9cBHcfEKyuvoLl3L3MDmwEqwwhNR/+hi/ccnpe0XD2T1B9H0CmGGh011Xk9YTRf/ T/WUBuwUO9h0MYtCO/E2vp2HxLuCKrT1OO9zjWfH7l/+qnhKRRQKb9/S/WWMtUFD9S6P eJiLet5YxnGedZ8Ss2FoBhFVAbrE5Mo4SQM3GU0X5Eip8soYL2VlniK00ZTAFWWcakYE 0DMI9QOGT9b3xMorMlTKwQfNoXTFd33aO3/u7zhRybBoijwFE8JmZXCm2X1lIEb+QF2a 1uqQ== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCVwQeUcFij4aEZNq606uS5Sz28NWnoGpE+LOua/LqWKNSO4U9vetOMv6DzLvRO8I1EKAPxyrYyQI/4f/2xTaoGcPHbkV/A= X-Gm-Message-State: AOJu0YwBs19bjpfcRFb2nLFhKYOHzeFUN5FDDxZtBYzHwtMLHAbF44S3 ejJOj+p/N7bxyZAETzSjgpOsVu/HiYiBb0KnjM4Eg7Kj6uhtOSBY X-Google-Smtp-Source: AGHT+IGyiGqwRJfIsxtOHny+Gaai2/HMWb9mh/tMfyXiIjZ+C3ay1N7LY+Dtc6Dshv86ORQ/lcnpFA== X-Received: by 2002:a2e:98c2:0:b0:2d4:6aba:f19f with SMTP id s2-20020a2e98c2000000b002d46abaf19fmr535210ljj.32.1710401455697; Thu, 14 Mar 2024 00:30:55 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:2281:0:b0:2d3:914b:406e with SMTP id i123-20020a2e2281000000b002d3914b406els54146lji.1.-pod-prod-08-eu; Thu, 14 Mar 2024 00:30:53 -0700 (PDT) X-Received: by 2002:a05:651c:782:b0:2d3:9354:fe07 with SMTP id g2-20020a05651c078200b002d39354fe07mr507054lje.5.1710401452988; Thu, 14 Mar 2024 00:30:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1710401452; cv=none; d=google.com; s=arc-20160816; b=QyHX55F9DcIu7Cu5k87wC2YzYBvOed+nNTqgySWoSm6+wxqRrDk1cEhi0U87sgtRCq t62wMOV8T9x65EdYz5bw4RVDEFI0Guqx0eCOApnbQ3Ut2SQhJ3qfg8l49s98/9l+X4LA 89HvBamj8Z2vvb6rpKq8AMFquCFNdJh3DXVk99Ej18Nc5JvhXPiZvdh0BHBj8PIphkpC rRwL/smVWAbsOdtaSBj2Pudc/7yYYH5X2rG4VmEJdsAx8ZaXoCQWpyYuWXKotGFNcKKK Hpa0xPGqI1/c6iBd8VUWHhF2WCaK5mNpsT3OVK/1+Ks/FW2dd1+odOrrmZx0bgDGnVK4 /Ntw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=ltVJFb9ivX1hE93WZirM0aUtWvKSwKvJn7KCu3ua5ig=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=SBlLU/2bjGXjLj03yDRe/4nVZ6d29ZR9Va8Xb3t5yU9CiYS8zuOJ7MsDYCHiJjZEEV CSErVXd1Y8cM3gSMPCSr82GdNa2hTDKdXzZNXrf6N+iKec2VK+ZRoB5I491IpA/6uMBl ftIG2JkWrvVe+9cIInhhKAkk4CBckiKwwPu0qa6A8ypmCoxp9YMxCASpOEGDeKAn3u7j ZhL9hW9X0HkL0xO/tzGUsdIzZVMD5Lu2uLtmnyqkT821A5iCUHnbZ8JtWrVRRu5IKr91 cXyVkZ/YBAwFAA5YJrlAT/vq155Tg8F5pdrjWsXRdr1ocE1CQNr4HpyMIpbtDMP6UbMY hUQQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id q9-20020a2e9149000000b002d449d21a98si8297ljg.1.2024.03.14.00.30.52 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Mar 2024 00:30:52 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from hp15.localdomain (mm-209-185-213-37.mogilev.dynamic.pppoe.byfly.by [37.213.185.209]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 42E7Uo9u021482 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:30:51 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v6 01/11] scripts: Add debrepo python script handling base-apt Date: Thu, 14 Mar 2024 10:27:18 +0300 Message-ID: <20240314073047.29465-2-ubely@ilbers.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240314073047.29465-1-ubely@ilbers.de> References: <20240314073047.29465-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This is the main utility responsible for prefetching packages into local `base-apt` repo from external Debian mirrors. It uses python-apt module and requires some kind of minimal `rootfs` to work (let's call it "debrepo context"). Once initialized with `--init --workdir=`, it stores the initial configuration in `repo.opts` file inside the context and uses it at futher calls. In future, the logic `debrepo` script implements could be directly implemented inside bitbake classes. Signed-off-by: Uladzimir Bely --- scripts/debrepo | 571 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 571 insertions(+) create mode 100755 scripts/debrepo diff --git a/scripts/debrepo b/scripts/debrepo new file mode 100755 index 00000000..e089cdb1 --- /dev/null +++ b/scripts/debrepo @@ -0,0 +1,571 @@ +#!/usr/bin/env python3 + +""" +# This software is a part of Isar. +# Copyright (C) 2024 ilbers GmbH + +# debrepo: build Debian-like repo using "python3-apt" library. + +When building the image, Isar downloads required Debian packages from external +mirrors. After build completed, it can pick all downloaded packages from DL_DIR +and build local 'base-apt' Debian-like repo from them. + +This tool allows to download packages and create local repo in advance. So, +Isar just uses this local repository and does not interact with external +mirrors. Such approach makes deb-dl import/export functionality redundant. + +Script `debrepo` works in so-called "context" directory. It means some +rootfs-like directory with bare minimum of directories/files required for +python3-apt to work. + +Context directory path is passed with "--workdir " command-line option. +On context creating, all passed parameters are stored in the context directory +and picked every time the context is used again. + +1. Repo for building Debian system +``` +debrepo --workdir=d12 --init locales gnupg +``` +Initialize the context in "d12" directory and create a repository in +"d12/repo/apt" directory sufficient to debootstrap default system +(e.g., debian-bookworm-amd64). Additionally, packages "locales" and "gnupg" +with all their dependencies will be available in this repo. + +``` +debrepo --workdir=d12 docbook-to-man +``` +Adds "docbook-to-man" packages with its dependencies to earlier created repo. + +``` +debrepo --workdir=d12 --srcmode docbook-to-man +``` +Downloads source package for "docbook-to-man" and adds it to the repo + +2. Repo for building Ubuntu system +``` +debrepo --init --workdir=uf \ +--distro=ubuntu --codename=focal --arch=arm64 \ +--aptsrcsfile=/work/isar/meta-isar/conf/distro/ubuntu-focal-ports.list \ +--repodir=repo/apt --repodbdir=repo/db \ +--mirror=http://ports.ubuntu.com/ubuntu-ports \ +locales gnupg +``` +Initialize the context in "uf" directory and create a repository in "repo/apt" +directory sufficient deboostraup ubuntu-focal arm64 system. Mirror to use and +source list are specified by corresponding arguments. Packages "locales" and +" gnupg" with the dependencies will be also placed to the repo. + +``` +debrepo --workdir=uf gnupg,locales +``` +Add "gnupg" and "locales" packages with their dependencies to earlier created +ubuntu repo. Other parameters (distro, codename, arch) are ommited since they +are picked from the context. + +3. Repo for cross-building Debian system +``` +debrepo --init --workdir=d11 --codename=bullseye--arch=amd64 --crossarch=armhf +``` +Initialize the context in "d11" directory sufficient to deboostrap Debian +Bullseye (amd64) with foreign "armhf" architecture support + +``` +debrepo --workdir=d11 gcc +``` +Add "gcc" package (amd64 version) to earlier created repo. + +``` +debrepo --workdir=d11 --crossbuild gcc +``` +Add "gcc" package (armhf version) to earlier created repo. +""" + +import os +import sys +import fcntl + +import argparse +import shutil +import subprocess +import pickle +import urllib.parse + +import apt_pkg +import apt.progress.base + + +class DebRepo(object): + class DebRepoCtx(object): + def __init__(self, workdir): + self.distro = "debian" + self.codename = "bullseye" + self.arch = "amd64" + self.mirror = "http://deb.debian.org/debian" + + self.repodir = f"{workdir}/repo/apt" + self.repodbdir = f"{workdir}/repo/db" + + self.crossarch = self.arch + self.compatarch = None + self.keydir = "/etc/apt/trusted.gpg.d" + + def __init__(self, args): + self.workdir = os.path.abspath(args.workdir) + os.makedirs(self.workdir, exist_ok=True) + self.ctx = self.DebRepoCtx(self.workdir) + + self.cache = None + self.depcache = None + self.sr = None + self.extrarepo = None + + self.ctx_load() + self.ctx_update(args) + self.ctx_save() + + print( + f"ctx workdir: {self.workdir}\n" + f" distro: {self.ctx.distro}\n" + f" codename: {self.ctx.codename}\n" + f" arch: {self.ctx.arch}\n" + f" mirror: {self.ctx.mirror}\n" + f" repodir: {self.ctx.repodir}\n" + f" repodbdir: {self.ctx.repodbdir}\n" + f" crossarch: {self.ctx.crossarch}\n" + f" compatarch: {self.ctx.compatarch}\n" + f" keydir: {self.ctx.keydir}" + ) + + if args.extrarepo: + self.extrarepo = os.path.abspath(args.extrarepo) + + def ctx_load(self): + ctxfile = f"{self.workdir}/debrepo.ctx" + + if os.path.isfile(ctxfile): + with open(ctxfile, 'rb') as f: + self.ctx = pickle.load(f) + + def ctx_save(self): + ctxfile = f"{self.workdir}/debrepo.ctx" + + with open(ctxfile, 'wb') as f: + pickle.dump(self.ctx, f) + + def ctx_update(self, args): + if args.distro: + self.ctx.distro = args.distro + if args.codename: + self.ctx.codename = args.codename + if args.arch: + self.ctx.arch = args.arch + if args.mirror: + self.ctx.mirror = args.mirror + + if args.repodir: + self.ctx.repodir = os.path.abspath(args.repodir) + if args.repodbdir: + self.ctx.repodbdir = os.path.abspath(args.repodbdir) + + if args.crossarch: + self.ctx.crossarch = args.crossarch + if args.compatarch: + self.ctx.compatarch = args.compatarch + if args.keydir: + self.ctx.keydir = args.keydir + + def create_rootfs(self, aptsrcsfile): + os.makedirs(f"{self.workdir}/var/lib/dpkg", exist_ok=True) + with open(f"{self.workdir}/var/lib/dpkg/status", "w"): + pass + + os.makedirs(f"{self.workdir}/etc/apt/sources.list.d", exist_ok=True) + + srcfile = f"{self.workdir}/etc/apt/sources.list.d/bootstrap.list" + if aptsrcsfile and os.path.exists(aptsrcsfile): + shutil.copy(aptsrcsfile, srcfile) + else: + with open(srcfile, "w") as f: + repo = f"{self.ctx.mirror} {self.ctx.codename} main" + f.write(f"deb {repo}\n") + f.write(f"deb-src {repo}\n") + + dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}" + os.makedirs(f"{self.workdir}/{dir_cache}/archives/partial", + exist_ok=True) + + os.makedirs(f"{self.workdir}/tmp", exist_ok=True) + + def create_repo_dist(self): + conf_dir = f"{self.ctx.repodir}/{self.ctx.distro}/conf" + os.makedirs(conf_dir, exist_ok=True) + if not os.path.exists(f"{conf_dir}/distributions"): + with open(f"{conf_dir}/distributions", "w") as f: + f.write(f"Codename: {self.ctx.codename}\n") + f.write( + "Architectures: " + "i386 armhf arm64 amd64 mipsel riscv64 source\n") + f.write("Components: main\n") + + def apt_config(self, init, crossbuild): + # Configure apt to work with empty directory + if not init and self.ctx.arch != self.ctx.crossarch: + apt_pkg.config["APT::Architectures::"] = self.ctx.crossarch + apt_pkg.config["APT::Architectures::"] = self.ctx.arch + + if not init and self.ctx.compatarch: + apt_pkg.config["APT::Architectures::"] = self.ctx.compatarch + + apt_pkg.config.set("APT::Architecture", self.ctx.arch) + + apt_pkg.config.set("Dir", self.workdir) + + dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}" + apt_pkg.config.set("Dir::Cache", f"{self.workdir}/{dir_cache}") + apt_pkg.config.set("Dir::State::status", + f"{self.workdir}/var/lib/dpkg/status") + + apt_pkg.config.set("APT::Install-Recommends", "0") + apt_pkg.config.set("APT::Install-Suggests", "0") + + # Use host keys for authentification + apt_pkg.config.set("Dir::Etc::TrustedParts", self.ctx.keydir) + + # Allow using repositories without keys + apt_pkg.config.set("Acquire::AllowInsecureRepositories", "1") + + def mark_essential(self): + for pkg in self.cache.packages: + if pkg.architecture == self.ctx.arch: + if pkg.essential: + self.depcache.mark_install(pkg) + + def mark_by_prio(self, priority): + for pkg in self.cache.packages: + if pkg.architecture == self.ctx.arch: + ver = self.depcache.get_candidate_ver(pkg) + if ver and ver.priority <= priority: + self.depcache.mark_install(pkg) + + def mark_pkg(self, name, crossbuild): + pkgname = name + + if pkgname and (pkgname not in self.cache): + # Try for cross arch + if (pkgname, self.ctx.crossarch) in self.cache: + pkgname += f":{self.ctx.crossarch}" + + if pkgname not in self.cache: + print(f"Error: package '{name}' not found") + return False + + pkg = self.cache[pkgname] + + if (not crossbuild) or (':' in pkgname) or (not pkg.has_versions): + if (pkg.has_provides) and (not pkg.has_versions): + print("pkgname is virtual package, selecting best provide") + # Select first provide + pkg_provide = pkg.provides_list[0][2] + # Find better provide with higher version + for provide in pkg.provides_list: + if apt_pkg.version_compare(provide[2].ver_str, + pkg_provide.ver_str) > 0: + pkg_provide = provide[2] + self.depcache.mark_install(pkg_provide.parent_pkg) + else: + self.depcache.mark_install(pkg) + else: + version = pkg.version_list[0] + if version.arch == "all": + self.depcache.mark_install(pkg) + else: + if version.multi_arch == version.MULTI_ARCH_FOREIGN: + if (pkgname, self.ctx.arch) in self.cache: + nativepkg = self.cache[pkgname, self.ctx.arch] + self.depcache.mark_install(nativepkg) + else: + return False + else: + if (pkgname, self.ctx.crossarch) in self.cache: + crosspkg = self.cache[pkgname, self.ctx.crossarch] + self.depcache.mark_install(crosspkg) + else: + return False + + return True + + def mark_list(self, pkglist, crossbuild): + ret = True + if pkglist: + for pkgname in pkglist: + ret = ret and self.mark_pkg(pkgname, crossbuild) + + return ret + + def handle_deb(self, item): + fd = open(f"{self.ctx.repodir}/repo.lock", 'w') + fcntl.flock(fd, fcntl.LOCK_EX) + subprocess.run([ + "reprepro", + "--dbdir", f"{self.ctx.repodbdir}/{self.ctx.distro}", + "--outdir", f"{self.ctx.repodir}/{self.ctx.distro}", + "--confdir", f"{self.ctx.repodir}/{self.ctx.distro}/conf", + "-C", "main", + "includedeb", + self.ctx.codename, + item.destfile + ], + stdout=subprocess.PIPE) + fd.close() + + def handle_repo(self, fetcher): + dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}" + fd = open(f"{self.workdir}/{dir_cache}.lock", "w") + fcntl.flock(fd, fcntl.LOCK_EX) + fetcher.run() + fd.close() + for item in fetcher.items: + if item.status == item.STAT_ERROR: + print("Some error ocured: '%s'" % item.error_text) + pass + else: + self.handle_deb(item) + + def get_filename(self, uri): + path = urllib.parse.urlparse(uri).path + unquoted_path = urllib.parse.unquote(path) + basename = os.path.basename(unquoted_path) + return basename + + def fetch_file(self, uri): + filename = self.get_filename(uri) + subprocess.run([ + "wget", + "-H", + "--timeout=30", + "--tries=3", + "-q", + uri, + "-O", + f"{self.workdir}/tmp/{filename}" + ], + stdout=subprocess.PIPE) + + def handle_dsc(self, uri): + filename = self.get_filename(uri) + fd = open(f"{self.ctx.repodir}/repo.lock", 'w') + fcntl.flock(fd, fcntl.LOCK_EX) + subprocess.run([ + "reprepro", + "--dbdir", f"{self.ctx.repodbdir}/{self.ctx.distro}", + "--outdir", f"{self.ctx.repodir}/{self.ctx.distro}", + "--confdir", f"{self.ctx.repodir}/{self.ctx.distro}/conf", + "-C", "main", + "-S", "-", "-P" "source", + "--delete", + "includedsc", + self.ctx.codename, + os.path.realpath(f"{self.workdir}/tmp/{filename}") + ], + stdout=subprocess.PIPE) + fd.close() + + def handle_src_list(self, pkgs): + if pkgs: + for pkg in pkgs: + pkgname = pkg + pkgver = "" + if '=' in pkg: + pkgname = pkg.split("=")[0] + pkgver = pkg.split("=")[1] + + self.sr.restart() + while self.sr.lookup(pkgname): + if pkgver and pkgver != self.sr.version: + continue + + for sr_file in self.sr.files: + print(self.sr.index.archive_uri(sr_file[2])) + self.fetch_file(self.sr.index.archive_uri(sr_file[2])) + + dsc_uri = self.sr.index.archive_uri(self.sr.files[0][2]) + self.handle_dsc(dsc_uri) + break + + def apt_run(self, init, srcmode, pkgs, controlfile, crossbuild): + apt_pkg.init() + + if self.extrarepo: + srcfile = f"{self.workdir}/etc/apt/sources.list.d/extrarepo.list" + with open(srcfile, "w") as f: + distdir=os.path.join(self.extrarepo, "dists") + if os.path.isdir(distdir): + for dist in os.listdir(distdir): + repodir = os.path.join(distdir,dist) + if os.path.isdir(repodir): + for repo in os.listdir(repodir): + if os.path.isdir(os.path.join(repodir, repo)): + f.write(f"deb file://{self.extrarepo} " + f"{dist} {repo}\n") + + sources = apt_pkg.SourceList() + sources.read_main_list() + + progress = apt.progress.text.AcquireProgress() + + self.cache = apt_pkg.Cache() + self.cache.update(progress, sources) + self.cache = apt_pkg.Cache() + + if self.extrarepo: + os.remove(f"{self.workdir}/etc/apt/sources.list.d/extrarepo.list") + + self.depcache = apt_pkg.DepCache(self.cache) + self.sr = apt_pkg.SourceRecords() + + ret = True + + if init: + self.mark_essential() + # 1(required), 2(important), 3(standard), 4(optional), 5(extra) + self.mark_by_prio(1) + + pkgs = list(filter(None, ','.join(pkgs).split(','))) + if srcmode: + self.handle_src_list(pkgs) + else: + ret = self.mark_list(pkgs, crossbuild) + + if controlfile: + fobj = open(controlfile, "r") + + try: + tagfile = apt_pkg.TagFile(fobj) + while tagfile.step() == 1: + deps = tagfile.section.get("Build-Depends", "") + # Remove extra commas and spaces - apt_pkg.parse_depends + # doesnt like lines like ", device-tree-compiler" + deps = ', '.join( + [s.strip() for s in deps.split(',') if s.strip()] + ) + print(f"parsed deps: {deps}") + for item in apt_pkg.parse_depends(deps, False): + pkgname = item[0][0] + self.mark_pkg(pkgname, crossbuild) + + finally: + fobj.close() + + if not ret: + sys.exit("Some of requested packages not found") + + if init or not srcmode: + fetcher = apt_pkg.Acquire(progress) + pm = apt_pkg.PackageManager(self.depcache) + + recs = apt_pkg.PackageRecords(self.cache) + pm.get_archives(fetcher, sources, recs) + + self.handle_repo(fetcher) + + +def parse_arguments(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--init", + default=False, action="store_true", + help="initialize context in WORKDIR") + parser.add_argument( + "--workdir", + type=str, required=True, + help="work directory storing debrepo context") + parser.add_argument( + "--aptsrcsfile", + type=str, metavar="PATH", + help="sources.list file to use when init") + parser.add_argument( + "--srcmode", + default=False, action="store_true", + help="add source packages instead of debs") + parser.add_argument( + "--repodir", + type=str, metavar="REPO", + help="repository directory") + parser.add_argument( + "--repodbdir", + type=str, metavar="REPODB", + help="repository database directory") + parser.add_argument( + "--extrarepo", + type=str, metavar="REPO", + help="extra repository to consider") + parser.add_argument( + "--mirror", + type=str, + help="use custom distro mirror") + parser.add_argument( + "--distro", + type=str, + help="select distro to use") + parser.add_argument( + "--codename", + type=str, + help="distro codename") + parser.add_argument( + "--arch", + type=str, + help="distro arch") + parser.add_argument( + "--compatarch", + type=str, metavar="ARCH", + help="compat arch to use") + parser.add_argument( + "--crossarch", + type=str, metavar="ARCH", + help="cross-build arch") + parser.add_argument( + "--keydir", + type=str, + help="directory with distro keys") + parser.add_argument( + "--no-check-gpg", + default=False, action="store_true", + help="allow insecure repositories") + parser.add_argument( + "--controlfile", + type=str, metavar="PATH", + help="debian control file to parse") + parser.add_argument( + "--crossbuild", + default=False, action="store_true", + help="add packages with cross arch") + + parser.add_argument( + "packages", + nargs='*', type=str, + help="space- or comma-separated list of packages to add") + + args = parser.parse_args() + + return args + + +def main(): + args = parse_arguments() + + if not (args.init or args.packages or args.controlfile): + sys.exit("Nothing to do") + + debrepo = DebRepo(args) + + if args.init: + debrepo.create_rootfs(args.aptsrcsfile) + debrepo.create_repo_dist() + + debrepo.apt_config(args.init, args.crossbuild) + debrepo.apt_run(args.init, args.srcmode, args.packages, + args.controlfile, args.crossbuild) + + +if __name__ == "__main__": + main() From patchwork Thu Mar 14 07:27:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3406 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 14 Mar 2024 08:31:39 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f189.google.com (mail-lj1-f189.google.com [209.85.208.189]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 42E7V33o021574 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:31:03 +0100 Received: by mail-lj1-f189.google.com with SMTP id 38308e7fff4ca-2d46a089baasf5013391fa.3 for ; Thu, 14 Mar 2024 00:31:03 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710401456; cv=pass; d=google.com; s=arc-20160816; b=JP0VhZMBk3Nx3qsOwH9Mv74wgAJKl7S/ErYh6/RtwrSNbpiljpTmUZ+K/YIEEGdocA UOzLjc77gq2sW2uADNDBP/qFQSfsQ+cPdDYX/1QBkQM1OUiCYW/pK2HYn/1oyAgz9Ndf fYKVcbdGROpRW6Y49ZyruE0mfREM2I3sR5BA5hQvdA+XPwzJQIyOzTGe1Yrys6TXdvkS +Vp0WAmsGfeQ8f77PKBGdgCd8Q5BIz3F/fluZXIbP+apBsXuJunAiRqy+J1CA5RrWBXp k31p9LczUyXIP9wl3HQPpOnHRmjX+GkXGM46ccZJR6l7nqx/YXP4v0pSvHkTV1UnCjbR 2rVw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=TAJz8r8DL/KOIr47RzIOfKvkByzzLOlRncw5CVSL3lY=; fh=Ft9TC1xmjPd+3EMSlRTjDHcwi91wluztK98bHOtSaX8=; b=GuzArFaR0RX/jLaAv2TA0kl+BMjTtvzlwmbbAW+kNhGtozxQbkSOy6Nv59HaqyjoY2 X0dbyhxZ25hsq0layJlOYGBwjRXK0H/63egduiaJdN0C/TU9OFe/J9wyPtDyxI9eTJKH JO0O9anAA+KRpFYHFA+INxjkFR3gBXLoPIucRmesjaaQWj537KMQnQyHhUgy4PoSNOIR gTJFaF+QaQ3ibpfghEQK4aEYM6w1MbmnIYlqBEfdj98+xBZH9BP35PppFk6c65EWWUdu O3xYO9BmfHbaFvSVabAMi43QQdG3jku0dAlUO4Kcncan9sckJNlm1kfcrr/rTnuao6tz eynQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1710401456; x=1711006256; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=TAJz8r8DL/KOIr47RzIOfKvkByzzLOlRncw5CVSL3lY=; b=gvAddUn6rBsjAfs4zfaqU0w5TuYrGrfKdnXKwOBu0KTO4qexfn9Un5EXqpsQybNENA LZbnHudZSOos8tPmyxpAD2lch37ZPsoSqDac5BvU/7UNeXROKiYRFi6PfNQKHxojMuVo q+GP2ix3xaxLhab6enF31XKtPimJ+X0XlFeWgWifb5NImKQ+GmOmTuDLR82uN9gcpZS2 FO81pu/+kqD1Zhsas3j5rthmhhHF18DHeIm8UrYMhIvzicKEPrYAXtamB92Yn5s37aiH WhqXDSs7fh8xnAb1CXQvoIA0w9h5HTln+lPL7Nptnl754jz9ThTFNeLZyPpShZvBNwfW GTuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710401456; x=1711006256; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=TAJz8r8DL/KOIr47RzIOfKvkByzzLOlRncw5CVSL3lY=; b=pXYczsnzDPNIFDfO6RxfdsU+W9QOd78W0ZWMhs+S1Bs1YzENZLqR31yOBCBZObUkAm pU6kn3U/F9pXFMZl2YuvZvQnXvQ81iVUJscyN3RgsPPOHWJwY8KErk5F/CHXysvD8u9r e/bX8cMZzCSz6pSOiHoOMxKX4TfDoR4Yv+pgHeNfvBal8119zML+GgZe3wnDSL8WuimR wLaUBl+g8EV55fjT0gsOvokXq2z59W9F19s5SRJ0Ua+4pv7Pjt3dZUthUtynZEr051Kh CQEIVrSS1XiioMWBHexbBrtDyuvK+Z2RPsLFuBX+LaDWZK6w9PXSCbZcBsyfI46GrsAG Ajow== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCVU5QnL2vlmJvP5qe9QFvLs7rfhiGdnIu2Nve9WO730Ce/OOdaDB0E0CDE1/kIW0HrmFaLTuQJ6xgcObS/c42lZs4zIJ1M= X-Gm-Message-State: AOJu0Ywsh/RMv9p2hi0QDjjYuVN+eD0V6LHrRm525yrBhBQNaE9aw5Y0 SgEfoI6qUpj9X+OGwqgLApfGbiCBzEzsIe2FuaRV7Rrrh0wB767v X-Google-Smtp-Source: AGHT+IEGT0Rp6DLawW91UNx437x0oycg0Pp6dzWY3zWsrYTwuX0gU4kzDnCjzj6vVQGRc+Y5V20O/A== X-Received: by 2002:a2e:9a8a:0:b0:2d4:6e5b:3601 with SMTP id p10-20020a2e9a8a000000b002d46e5b3601mr599167lji.14.1710401455806; Thu, 14 Mar 2024 00:30:55 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:19ab:b0:2d4:3db9:66b7 with SMTP id bx43-20020a05651c19ab00b002d43db966b7ls339429ljb.1.-pod-prod-04-eu; Thu, 14 Mar 2024 00:30:53 -0700 (PDT) X-Received: by 2002:a2e:a9a6:0:b0:2d3:2a95:6f0b with SMTP id x38-20020a2ea9a6000000b002d32a956f0bmr658943ljq.12.1710401453303; Thu, 14 Mar 2024 00:30:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1710401453; cv=none; d=google.com; s=arc-20160816; b=cSC/gRludF4MTRnjvX0YT7vDgVuld8SDHjk+YpMbS7UhYLzwgi665gQX8wbKqHbltt 184iOpmV3vB79e8fAxtF0p0Tty9q2P/uE2Qgsi9f9T2WMKrgzdxYffrCfGWYdQ9DcVyv aVz2s7Ex05SHvQVd374Hn9wpBkiVuF0juiMTco20XhPp23E6UkbYmwAXTeohpg3AMgeY uZYImllhsTGGjwbKFDCaS4fs18K1FX42oGpPsjVqcMJyLpfcqkjziXaZ1BwI5haemsLJ 0qGMwpNPUmFmGgLIel61KUiNWwewV1i5PGLEzjBpcmRAztjHBjpvEq869/loKmkXBBF2 GcLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=+jomylU6eahElKSbHJziRBkdNW0tot2Clr2cQQ/gLzI=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=p8pTY1fg6EGbVCfOBd0BpFLboNyoNRF+n7kIHupHJYJcdleNNExvFrpRjmNc0TyGvN YmXCHnzr7vWWnwzqzduUaBxp/Fk4X0oaiAPKR434hcDp6hesSzi38jvi/03VdMGeLda8 teYi0kSmd8MRq4FPyNvbYXFez2pP39u5agBDrPF0gSaguxGw2D4SZg6W2li37vIEYqKi vaDrdvRUEtrbOcOB8lJ1c7pTgypMLbStxdB9k+PmEb/26/qvHzL4vm7AGZDmxTDdr0ex gdui9KDEB/KqzApay7l3/dz3t50Z+dWYq2Ctbc81efRRa3eV7tM484spLeZnMWlECBHR 6Wsg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id z20-20020a2e9b94000000b002d46764b564si6140lji.6.2024.03.14.00.30.53 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Mar 2024 00:30:53 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from hp15.localdomain (mm-209-185-213-37.mogilev.dynamic.pppoe.byfly.by [37.213.185.209]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 42E7Uo9v021482 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:30:52 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v6 02/11] meta: Add debrepo bbclass handling base-apt prefetching Date: Thu, 14 Mar 2024 10:27:19 +0300 Message-ID: <20240314073047.29465-3-ubely@ilbers.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240314073047.29465-1-ubely@ilbers.de> References: <20240314073047.29465-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This class uses 'scripts/debrepo' python script to prefetch given packages or sources to local base-apt repository. Signed-off-by: Uladzimir Bely --- RECIPE-API-CHANGELOG.md | 9 ++++ meta/classes/debrepo.bbclass | 81 ++++++++++++++++++++++++++++++++++++ meta/conf/bitbake.conf | 5 +++ 3 files changed, 95 insertions(+) create mode 100644 meta/classes/debrepo.bbclass diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index 2da7968d..cb466eb7 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -572,3 +572,12 @@ the module build recipe. Remove all uses of the function deb_compat. The functionality was replaced with a dependency to the package debhelper-compat. +### "Prefetch" mode for base-apt + +Originally, `base-apt` repo is created only during second build when variable +ISAR_USE_CACHED_BASE_REPO is set. The repo is populated with every package that +took part in the first build and was cached in DL_DIR. + +New ISAR_PREFETCH_BASE_APT variable changes the way `base-apt` is populated. +Packages added to the repo before running any task that need them. Separate +`debrepo` script is used for populating base-apt repo. diff --git a/meta/classes/debrepo.bbclass b/meta/classes/debrepo.bbclass new file mode 100644 index 00000000..6c135ef2 --- /dev/null +++ b/meta/classes/debrepo.bbclass @@ -0,0 +1,81 @@ +# This software is a part of Isar. +# Copyright (C) 2024 ilbers GmbH +# +# SPDX-License-Identifier: MIT + +DEBREPO_WORKDIR ?= "${DEBREPO_TARGET_DIR}" + +debrepo_update_apt_source_list() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + + chroot_dir=${1} + apt_list=${2} + + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + sudo -E chroot ${chroot_dir} /usr/bin/apt-get update \ + -o Dir::Etc::SourceList=\"sources.list.d/${apt_list}.list\" \ + -o Dir::Etc::SourceParts=\"-\" \ + -o APT::Get::List-Cleanup=\"0\" + " +} + +debrepo_add_packages() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return + + args="" + if [ "${1}" = "--srcmode" ]; then + args="${args} --srcmode" + shift + fi + + if [ "${1}" = "--isarapt" ]; then + args="${args} --extrarepo=${REPO_ISAR_DIR}/${DISTRO}" + shift + fi + + workdir="${1}" + args="${args} ${2}" + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + fi + + flock -x "${workdir}/repo.lock" -c " + ${SCRIPTSDIR}/debrepo \ + --workdir=\"${workdir}\" \ + ${args} + " +} + +debrepo_handle_controlfile() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return + + control_file="${1}" + args="" + + build_arch=${DISTRO_ARCH} + if [ "${ISAR_CROSS_COMPILE}" = "1" ]; then + build_arch=${HOST_ARCH} + fi + if [ "${PACKAGE_ARCH}" != "${build_arch}" ]; then + args="--crossbuild \ + crossbuild-essential-${PACKAGE_ARCH}:${build_arch} \ + dose-distcheck:${build_arch} \ + libc-dev:${PACKAGE_ARCH} \ + libstdc++-dev:${PACKAGE_ARCH} \ + " + fi + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + fi + + flock -x "${DEBREPO_WORKDIR}/repo.lock" -c " + ${SCRIPTSDIR}/debrepo \ + --workdir=\"${DEBREPO_WORKDIR}\" \ + --controlfile=\"${control_file}\" \ + ${args} + " +} diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 91c5c815..1bff8c8e 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -69,6 +69,11 @@ KERNEL_FILE:mipsel ?= "vmlinux" KERNEL_FILE:riscv64 ?= "vmlinux" KERNEL_FILE:arm64 ?= "vmlinux" +# debrepo config +DEBREPO_DIR = "${TOPDIR}/debrepo" +DEBREPO_HOST_DIR = "${DEBREPO_DIR}/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}" +DEBREPO_TARGET_DIR = "${DEBREPO_DIR}/${DISTRO}-${DISTRO_ARCH}" + MACHINEOVERRIDES ?= "${MACHINE}" DISTROOVERRIDES ?= "${DISTRO}" OVERRIDES = "${PACKAGE_ARCH}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${BASE_DISTRO_CODENAME}:forcevariable" From patchwork Thu Mar 14 07:27:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3403 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 14 Mar 2024 08:31:27 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-ed1-f55.google.com (mail-ed1-f55.google.com [209.85.208.55]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 42E7V2Y0021562 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:31:02 +0100 Received: by mail-ed1-f55.google.com with SMTP id 4fb4d7f45d1cf-5689df8fbcasf406438a12.0 for ; Thu, 14 Mar 2024 00:31:02 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710401456; cv=pass; d=google.com; s=arc-20160816; b=eD9yxpVn7pCPKp98W4+Fb2GHVdsm8lFr7/Am7ORi9ajWDt7ZZhgzzPg/YHZRadFS8R VzJQPr4YTXjlg+rvwlHBrk7lumkPbu5vyu4SqFaFHDfxIVAuh7+EN5Q/jXjdw7IRK8Yl OHs/uyQY/RI/QoLwkKctkK5gEKtw2EGtd7c0GiGpDf0Su5WwUnMFb3KXZxkDUUEgxMG8 ZbOH1EZBzH9M66sk82oCkLXUnyYYnkLbpC+xzCCbCly+QKHobuG/FC8kkhc4tEoRcK0H TYjhIxtkS3JPiGKRXkQZZ53H+rHh1tkXAwrW2+K+J0zYl8AFugE00cWOFpjB8XEMZ68J h47Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=zrHv2K/qJX5SN7tIpvu/qbmXRVnXIAo6ATaArTUJU2U=; fh=YIgrIgo5UWDsDfiG4kJwwYejdL1gl2eIAslZLqgsdyA=; b=HlE4KZkobxO4JBI3TnLTin1PqUNMQf0JSshsLiptaL0WjZFmLKYiGwbUbsLcpax9jH gOqTJ9LNsmzolJXkwsgS+BNE3+0+D9SJnh24m8WIqlCGssflzXI9WKt/0moGjsWEkgE7 ps9cFV9FHKd2lqMcaLSov5V2tQu1/VZ3LOzJ0ROUW+YWbs0IO0UR3APzlM2KiIgKsoFz bjYzYbOgCxPpF9+oH+iJV38eTkYL+APbC988v4p4froKGzAsiQEnLL+hq5AAWLPSA4DL ai3Fvfa11VoZEnorPa2dRSPyUSIoP+5hrh8dsYArLALclm+2SU3l3dfG58iAl4zmzk9K S5Tg==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1710401456; x=1711006256; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=zrHv2K/qJX5SN7tIpvu/qbmXRVnXIAo6ATaArTUJU2U=; b=ZD/5N1+RjPQXa0Po4i++cuqiMWeRg6mWPyp/a3QI/XTJT0LbvXOnb+5eN2WS+rTSl8 1T46Po89JUEhAVNgcMmUO4+1dtvKW2YjRUrj2tWMMeQQvmeI6DvxPrEfWLnGlhzEupcj uRFMQAop0QCyfwnRweiIi9qrdcW/Jrd8tDM4iwJ+dwmbw5tblUwNymJq7MtAlI3Z4Zi0 Yv886CyzomBVXmRlkEIDj6BwUj8kAbnvVrDf9OkfQ+nkAWxNFkFG7nvlVSeviI8OP+6p 3okeMOiIprPLmxmYbY5IrkK1AbIe2EeKZxrzF9RaL6obqEXTr/vYYtlkw+lQVjkXTrle legw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710401456; x=1711006256; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=zrHv2K/qJX5SN7tIpvu/qbmXRVnXIAo6ATaArTUJU2U=; b=rnSs5nWFKqwiJUlv6EKvsvbbbicm+DLCQDGmTlM/02Dlq4UgFXY8CAJoxe+XzXqJKF WbvOQ49wDqhA6itwTJvnW9zu8yYukbZXDKwI8CSrgdMgwGz8GoBD4puBpY9SmayAeA50 CPdDXmJQPfJWWZlBJvp+lZNVGqrD/GXZw7j6SyuG7XPbrdvMGfA0oD+QHUHi5KcsLnGd fyBqryZ1b4vNcWxeZHvtdMURmAmwe4YZCqIspPfYUjSsK3oVPWxXCL6hOMsJL+XZKL0W qj3JF3k5zW8wYwQG9CMfJNT10yjFplGlMGNac7il814dYw5COqYgnYMMmrc4DfE4Ze+P 0AKQ== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCUZkDmgxF9phdo782NbQKpHpYXe77K6PB60DtB5qOUaxY+3A++6Po2nuyD3ED/xq1arMD/KCOieEFjdLaexoj4L0Xo4S5I= X-Gm-Message-State: AOJu0YxGwGkVhJ1uYSpRBcvYUrN7n9AnEBmApVLH1ikcCaQYfJGzA+S/ vzgQGv+BsGdGrRPA7iYGdJUCU9TGKuKjBjbq7xfnC1+YJfGpcpVr X-Google-Smtp-Source: AGHT+IE6A1qhiy7AKN6uPweFioHAcZcbdGTKROe5PfCEJFYU9EsRML/Ias8m5oogR1uDQn8C0QbUbA== X-Received: by 2002:aa7:d998:0:b0:568:a088:e472 with SMTP id u24-20020aa7d998000000b00568a088e472mr579219eds.6.1710401456041; Thu, 14 Mar 2024 00:30:56 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6402:380e:b0:568:2764:b455 with SMTP id es14-20020a056402380e00b005682764b455ls451849edb.1.-pod-prod-00-eu; Thu, 14 Mar 2024 00:30:54 -0700 (PDT) X-Received: by 2002:a17:906:d0c7:b0:a46:6df5:3511 with SMTP id bq7-20020a170906d0c700b00a466df53511mr1296600ejb.4.1710401453640; Thu, 14 Mar 2024 00:30:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1710401453; cv=none; d=google.com; s=arc-20160816; b=q23RFpyE3S0qw9e5e8XcaauxWmYrB5PZlQAELZd65u4/C6y7PZ0VtZBBiI+my4c5zr BmnnudEUbCKvAHIE9kvuIHurP94R4daev2TLN+JV8qOgvhKNqI6ZlwRST7cSNoX3jVyx +YWqA5fTlzJs4GbcKc6Kp99f3du+zrwKqqAfdSDViBzJCDzIju7d+lCPUl0KjDE0+duE 1uT7l08Izfje81eGHaNpn3LJci7sXyAphy5GQEiBnPRV/Q4JPkzlz3gxi5O4vc5VrhPp W5JvoQ0sVx8IWDB+97DEDuIHd0Y1morI8tzQ2rUQyWnisGXWxPPW35/nScljvvPWG72q OY7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=UD5y/dXcMVuvvSNV4kUevqVtox3oeo0vurNRka5Fm2g=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=kLgoWUTO7aZFVzaLgJ/zjKhAwg52dy+eWvg/+yztpA8ytzY+54SDRGWg3tkB49dOOY iPQ4k7WKPuz5GXtwmgazge5smyKmCXIlcnFpu8KRZ4ojQ5c1yqFhsGZPks2xPglpg/fE pKxbzng0VA4fbTzHDyd/3yvR61bHv86dz5bZj5RNK4UJ+ao+9kRmPXrY9+X0btFRdW0d 6fHpl1DnY5ZL3V2gUCpNKXbgfDGbcHuJIDiFsespOK31LZXiZZNkEAkEAaKIfvhUG4Gh hoG1PNfHQfiTwqE9EGIXNgqtt9jsQmctzxvjVBG+DYQh5EG3lUgbqafMfXqMLYrRvdcm LQSg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id bc4-20020a056402204400b00568a42a7564si4948edb.0.2024.03.14.00.30.53 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Mar 2024 00:30:53 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from hp15.localdomain (mm-209-185-213-37.mogilev.dynamic.pppoe.byfly.by [37.213.185.209]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 42E7Uo9w021482 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:30:52 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v6 03/11] meta: Always use base-apt repo in local mode Date: Thu, 14 Mar 2024 10:27:20 +0300 Message-ID: <20240314073047.29465-4-ubely@ilbers.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240314073047.29465-1-ubely@ilbers.de> References: <20240314073047.29465-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This means only local URLs in apt sources.list* are present during the build. Any installation of packages is done from local base-apt. So, base-apt should be always mounted in *_do_mounts since now. Signed-off-by: Uladzimir Bely --- meta/classes/rootfs.bbclass | 5 +++-- .../isar-bootstrap/isar-bootstrap.inc | 12 +++++++---- meta/recipes-devtools/base-apt/base-apt.bb | 21 ++++++++++++------- 3 files changed, 25 insertions(+), 13 deletions(-) diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 498fbfd6..ddc5d12c 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -54,8 +54,9 @@ rootfs_do_mounts() { mount --bind '${REPO_ISAR_DIR}/${DISTRO}' '${ROOTFSDIR}/isar-apt' fi - # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set - if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ] + # Mount base-apt if 'ISAR_PREFETCH_BASE_APT' or 'ISAR_USE_CACHED_BASE_REPO' is set + if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')))}" = 'True' ] || \ + [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ] then mkdir -p '${ROOTFSDIR}/base-apt' mountpoint -q '${ROOTFSDIR}/base-apt' || \ diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 17f19fd8..05c80308 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -42,7 +42,8 @@ python () { # installation afterwards. However, debootstrap will include the key into # the rootfs automatically thus the right place is distro_bootstrap_keys. - if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): + if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \ + bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) : own_pub_key = d.getVar("BASE_REPO_KEY") if own_pub_key: distro_bootstrap_keys += own_pub_key.split() @@ -110,7 +111,8 @@ def parse_aptsources_list_line(source_list_line): def get_apt_source_mirror(d, aptsources_entry_list): import re - if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): + if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \ + bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) : premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" else: premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or "" @@ -283,7 +285,8 @@ do_bootstrap() { if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}" fi - if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" -a -z "${BASE_REPO_KEY}" ] || \ + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then debootstrap_args="$debootstrap_args --no-check-gpg" fi E="${@ isar_export_proxies(d)}" @@ -314,7 +317,8 @@ do_bootstrap() { install -v -m644 "${APTPREFS}" \ "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap" mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" - if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] || \ + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main" if [ -z "${BASE_REPO_KEY}" ]; then line="[trusted=yes] ${line}" diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb index ea885fe6..e62958d1 100644 --- a/meta/recipes-devtools/base-apt/base-apt.bb +++ b/meta/recipes-devtools/base-apt/base-apt.bb @@ -57,9 +57,12 @@ repo() { "${BASE_DISTRO_CODENAME}" \ "${WORKDIR}/distributions.in" \ "${KEYFILES}" - populate_base_apt "${BASE_DISTRO}" - repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" + if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \ + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then + populate_base_apt "${BASE_DISTRO}" + repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ + "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" + fi if [ '${BASE_DISTRO}' != '${HOST_BASE_DISTRO}' ]; then repo_create "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \ @@ -67,14 +70,18 @@ repo() { "${BASE_DISTRO_CODENAME}" \ "${WORKDIR}/distributions.in" \ "${KEYFILES}" - populate_base_apt "${HOST_BASE_DISTRO}" - repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \ - "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}" + if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \ + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then + populate_base_apt "${HOST_BASE_DISTRO}" + repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \ + "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}" + fi fi } python do_cache() { - if not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): + if not bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) and \ + not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): return 0 for key in d.getVar('BASE_REPO_KEY').split(): From patchwork Thu Mar 14 07:27:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3404 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 14 Mar 2024 08:31:28 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f55.google.com (mail-wr1-f55.google.com [209.85.221.55]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 42E7V2ku021561 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:31:02 +0100 Received: by mail-wr1-f55.google.com with SMTP id ffacd0b85a97d-33eca5c305asf15928f8f.3 for ; Thu, 14 Mar 2024 00:31:02 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710401456; cv=pass; d=google.com; s=arc-20160816; b=KnVPfkldw3138s4cwhqmtZ4aX8Jvu+2rmf9vYQF72p8igs87M623hATDPqLyxA/oji AnDF0S55ol4KrowAQsHst73S5VtZcFtycRu+xeyXiwlNibFzaSwqjkwu8lQvePoSWPn2 RHIKybJBQXHdDYcw/L0adYQ9cy1kUiCLSycgR4J1pbekm2vgt6OK9GBLnnyElvSQT4Ui 1FyxBiQdv1yDSolXzgqopXtj3DkT3RHU4REeXOUUeIM1OYyy9HjnnUqaAeevlavaMCqP Ong1OdKOyyngQgWdXDiljBdJ9wL+KuFAFLSjyz4Q13Yp+OTyWI18hQUqOxXHNdu2wDdh /WQA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=P9QKlBLKzqFuvZd0HAs+OI0qtD7dAteP2Lx09jQwyu4=; fh=h78iQD1B22Ky3tmFpqSOP4nWJqssUS12lSS9bJbF2Rg=; b=swc6l74dHN1gdLDmCEJBt23zt/yw1eauXFb/Dq4wAWubS4xxIXfg0WDeXz7QFQK70i OKGKL4RnHRgpzTlRB/7E8SrqFvZ48wcjJLTyClqWZoqM13u2UrMKO5iHSC7v4GXd0P3x ytohP3DZ/gitzNm37qZWOmmOzq2tZ+MTPgD4VeXUpewP+ESqfhzd9Wd/8bG5fHvcyhBg 0YCIH3Utp4eNizzKa7WUcu2MOHLcOj9Lo7uYoF+SS2q1YuM1O7Ii51Uch1Bs1lwgAoTZ Pjs0m8rwBYH2EBmXBWQYA9jl83B3ebo3FRhPSTafLqWXW06yIwuMJgRCpf/aECkbBlip QNpw==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1710401456; x=1711006256; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=P9QKlBLKzqFuvZd0HAs+OI0qtD7dAteP2Lx09jQwyu4=; b=Eis5pHZHqgDP1h35bwILbRdUlktn/OrziWBX7u46WVu+7YCIY3kSUAA68f2KkU8V14 isluipIM1ASbiUyZKPQ+xyuff/shu8RSAUmlg/FT+pC9+vXPDuiwZvr5uQLLcAL7LQYy /Ghh2JbQpQwGO+0UFxvsk4ep7aOjJqN19odTcuUsNVcHXEkCX6/TUUqYRlsVPQmKFOaW njpS1kqTaJsPWb8f9sWDvulrHntwYkbjw8OuBb8JX523XqdZOFfvpR8dT1li5xediXVS 1x8uQHoTFxoc6epNcuxFkbCYpyisiwYsESKtPimJnmixXCur9FufWYUC7StK0JhUf9rN ZZGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710401456; x=1711006256; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=P9QKlBLKzqFuvZd0HAs+OI0qtD7dAteP2Lx09jQwyu4=; b=KC7jCEGf5et0xxuyKd/rzFhiS1UyZ76YdBBpYJcTsI8YcnGZuBXC1TrwXoJojrsxUr TETh+6FvUGOVR9vTSBwtl2k8etf5j2E1rPK+YdN0vLKS9vzNESUs+qFHBLlevN6T4rTz VLrHTdTPClbgP0LZtHtDHgaD8tqgKVlY7mziPKnuitOMCWgurwSJOXx1W/l6LXJw56tJ +wRKq7/O8KU0Q3dhbwDIEPEsaHu3evkZPp5g5cZGsKOvAiXrqTKtSKcCDTvvqPfdZJvT B0zBtSCrAqQlnDSIcWBgeZMqje7c3ksHF3k88g+4EbRw/FmUvkVDmAClD07XncFVHfmw aVHg== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCWkJY2wTWE+SkANhw6E7jlhjkw1OrzpNZ5BSRzFj5KH1rKkm65aLZJ9BZAOACBQl3SXjCdNIxFYFhKeKPBorH3dqnqgwMg= X-Gm-Message-State: AOJu0YxxMzMy8Olgj6Lneh9QAIb5oA9YfIcYjhCkgK3WGCOzrMPhCrPR 08SzBLmBviZICtJsJjNPE950+meJIIZ76rZAIpT/6UWjvNwUDvSz X-Google-Smtp-Source: AGHT+IHg4garzhqol7AV8zxMorTs3UpF4JvgsYBSlSkm8eV7XVShOo4LsYumogUjF7dNL9Pxle5DMA== X-Received: by 2002:adf:a2d6:0:b0:33e:8fb4:f3e9 with SMTP id t22-20020adfa2d6000000b0033e8fb4f3e9mr543714wra.29.1710401455921; Thu, 14 Mar 2024 00:30:55 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:5010:0:b0:33d:d932:8013 with SMTP id e16-20020a5d5010000000b0033dd9328013ls193722wrt.2.-pod-prod-06-eu; Thu, 14 Mar 2024 00:30:54 -0700 (PDT) X-Received: by 2002:a05:6000:194f:b0:33d:24af:9153 with SMTP id e15-20020a056000194f00b0033d24af9153mr594801wry.20.1710401453860; Thu, 14 Mar 2024 00:30:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1710401453; cv=none; d=google.com; s=arc-20160816; b=unlBWwmFYsHKbV+NXOkhnG6dBQQgBImIXwVab72u/4DPsRrpySXOBZgddLZaHTtHUh jaj9fWyVjWU7vFaUXBJBgpGhBHfyV8DT0ChaOQHE2S7SfbTtkZlIX879nCoKEGAkxRKw BV/82/3LmklhRAGHN/9xrtJ9fKo33dJknJqYHnoQIykcwe+CkzP+UjgJvLXiyIuw/2RH MlJbN4A0g8OiZwtEupgcf9GeN+DCei0AtnQaxJlUYn5xMhH0xp89f8RRpzawe0yswC8y vVdEW8+r7qlRkiPleQPK/gUUHpbFHCcECK39gEdGXumFhybLwNYzhkTW19teV4uqRWqp alQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=WcMURk/hHZweb+xItjOlrMp56HyxacIQjJkxR3LKqgg=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=sHY2B4uMI3Lg0iz6w3zj57xb2FpZEOz/y5MSGRjDCBlwimfudiqtIblFHQmyv0e3rW qjjM2WT5a6G8rnOrBUNILiXmw4wN5qaj4bcQOzTMIfnrOBlrun5S15tqF2hhWWh7wciP +a32fZy8zYItOwXQN558J4hK/+zNQoMkzxt/FcrkdPUSl+ROXBg3DV0X4BfXqmdFx508 BSU2WwojuwE9b/xpkwOOtyYo3bp5nIvHh8D3PtkJpx4FcrPzAsTMng6aZZBaUQfboYQH Wrck2JBTFu6Vg8/09DRisikh/WU90/ebdiBiMxOrvjWyrXGFoPKo4cImdZYmqXPjqtU3 XJpQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id l4-20020a5d4804000000b0033da1913960si3126wrq.7.2024.03.14.00.30.53 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Mar 2024 00:30:53 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from hp15.localdomain (mm-209-185-213-37.mogilev.dynamic.pppoe.byfly.by [37.213.185.209]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 42E7Uo9x021482 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:30:53 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v6 04/11] meta: Use cached base-apt repo to debootstrap Date: Thu, 14 Mar 2024 10:27:21 +0300 Message-ID: <20240314073047.29465-5-ubely@ilbers.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240314073047.29465-1-ubely@ilbers.de> References: <20240314073047.29465-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch makes local base-apt repo to be created before debootstrap task. So, debootstrap is then done from it. The required packages are downloaded via python-apt and reprepro creates debian-like repository from .deb files. For debian targets host keyring is used while ubuntu/raspbian targets use keys specified by DISTRO_BOOTSTRAP_KEYS variable. The goal is have workable base-apt repo before first build completed. Signed-off-by: Uladzimir Bely --- .../isar-bootstrap/isar-bootstrap.inc | 94 +++++++++++++++++-- 1 file changed, 86 insertions(+), 8 deletions(-) diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 05c80308..e5d2523d 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -29,6 +29,13 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates" DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else ''}" BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}" BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}" +BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO_ARCH')}" + +# For newer distros "usr-is-merged" indirectly required by debootstrap +DISTRO_BOOTSTRAP_BASE_PACKAGES:append:bookworm = ",usr-is-merged" +DISTRO_BOOTSTRAP_BASE_PACKAGES:append:sid = ",usr-is-merged" +DISTRO_BOOTSTRAP_BASE_PACKAGES:append:sid-ports = ",usr-is-merged" + FILESEXTRAPATHS:append = ":${BBPATH}" inherit deb-dl-dir @@ -275,12 +282,63 @@ do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}" inherit compat +inherit debrepo + +debrepo_bootstrap_prepare() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return + + debrepo_args="" + if [ "${BASE_DISTRO}" != "debian" ]; then + if [ "${BASE_DISTRO}" != "raspbian" ] && [ "${BASE_DISTRO}" != "raspios" ] || [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then + debrepo_args="$debrepo_args --keydir=${WORKDIR}" + fi + else + if [ "${BASE_DISTRO_CODENAME}" = "sid" ]; then + debrepo_args="$debrepo_args --keydir=${WORKDIR}" + fi + fi + if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then + debrepo_args="$debrepo_args --compatarch=${COMPAT_DISTRO_ARCH}" + fi + + debrepo_workdir=${DEBREPO_TARGET_DIR} + if [ "${BOOTSTRAP_FOR_HOST}" = "1" ]; then + debrepo_args="$debrepo_args --crossarch=${DISTRO_ARCH}" + if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then + debrepo_workdir=${DEBREPO_HOST_DIR} + fi + fi + mkdir -p "${debrepo_workdir}" + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + fi + + flock -x "${debrepo_workdir}/repo.lock" -c " + ${SCRIPTSDIR}/debrepo --init \ + --workdir=\"${debrepo_workdir}\" \ + --aptsrcsfile=\"${APTSRCS_INIT}\" \ + --repodir=\"${REPO_BASE_DIR}\" \ + --repodbdir=\"${REPO_BASE_DB_DIR}\" \ + --mirror=\"${@get_distro_source(d)}\" \ + --arch=\"${BOOTSTRAP_DISTRO_ARCH}\" \ + --distro=\"${BOOTSTRAP_BASE_DISTRO}\" \ + --codename=\"${BASE_DISTRO_CODENAME}\" \ + ${debrepo_args} \ + ${DISTRO_BOOTSTRAP_BASE_PACKAGES} + " +} + do_bootstrap() { if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then if [ -z "${COMPAT_DISTRO_ARCH}" ]; then bbfatal "${DISTRO_ARCH} does not have a compat arch" fi fi + + debrepo_bootstrap_prepare + debootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}" if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}" @@ -302,8 +360,19 @@ do_bootstrap() { sudo -E -s <<'EOSUDO' set -e if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then - arch_param="--arch=${DISTRO_ARCH}" + arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH}" fi + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + ${DEBOOTSTRAP} $debootstrap_args \ + $arch_param \ + ${@get_distro_components_argument(d)} \ + ${@get_distro_suite(d)} \ + ${ROOTFSDIR} \ + file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO} \ + ${DISTRO_DEBOOTSTRAP_SCRIPT} + " + else ${DEBOOTSTRAP} $debootstrap_args \ $arch_param \ ${@get_distro_components_argument(d)} \ @@ -311,7 +380,7 @@ do_bootstrap() { "${ROOTFSDIR}" \ "${@get_distro_source(d)}" \ ${DISTRO_DEBOOTSTRAP_SCRIPT} - + fi # Install apt config mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d" install -v -m644 "${APTPREFS}" \ @@ -383,12 +452,21 @@ do_bootstrap() { if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH} fi - - chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \ - -o APT::Update::Error-Mode=any - chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f - chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ - -o Debug::pkgProblemResolver=yes + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \ + -o APT::Update::Error-Mode=any + chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f + chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \ + -o Debug::pkgProblemResolver=yes + " + else + chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \ + -o APT::Update::Error-Mode=any + chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f + chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \ + -o Debug::pkgProblemResolver=yes + fi umount -l "${ROOTFSDIR}/dev/shm" umount -l "${ROOTFSDIR}/dev/pts" From patchwork Thu Mar 14 07:27:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3405 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 14 Mar 2024 08:31:35 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f62.google.com (mail-wr1-f62.google.com [209.85.221.62]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 42E7V3dJ021579 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:31:04 +0100 Received: by mail-wr1-f62.google.com with SMTP id ffacd0b85a97d-33e7ef510aasf296906f8f.0 for ; Thu, 14 Mar 2024 00:31:04 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710401456; cv=pass; d=google.com; s=arc-20160816; b=bmvXauK5Fg608qGmPWmp/glfLH7B0uC5gjiRCMJz1SWy9jzAoYZanZRdJixj4/Skdd 84ht3Aurav/DIXk/Nuun+ef8RsLFpZV2Ibfo9qyjUg6vW8siwdQGASg3TKW1nXUJvF1z QwtNekmSIbPiqK71OvdZ6y/iGdrEBF6rMLaTkLoDvdTic8Z4XlhgAS5piq+xGUyGYSW8 pHXQl3mhrAx6ZN+1c/knkjUdQY8UVorByFFsUzIExBSlCAIwRhLCX20V+8tXLRChmXVr otktme7eYnqqQ33/mWXQp5qcGErfWPLUQVT/3DJHSoqy63XrwVBUpgrOOuqwKuUDlhc7 XVhA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=j3hf+dqUOOr6+JyxflK1UanAlbRAjriUIe9XRXLJA0M=; fh=epo2iNTgg1Sv/z5AErStHOZmGqrupst97YUxRwsk8QA=; b=tbno/LXAU2l9ghmev+ORwU5o6IxR5KYb16QFaLpiL3dzdI72lTQlj477s8w3X2SpAo WpRdWpewWjhaXBN7rWryrnTv/jZGmPAbQ3zwoUbmkWrQS96DRLkjCXkOsGNaO24p/Tq6 2bwEtco2nsP1+ygXJ2TfavUQdpvGZNrvUQ9jT3nnNJFpZY/owAkyb3PN4Kh/a3op4pLY a08dZ4z4l3WjZZDFmQpsMg09y8touIq8LVsDrE1zzo46lnMfrDHK+WodB6QpQiMj9yId btxqA7wpqbOQSuv44OXeAlNrNOVfJoiMDDOisAUCtDlgTd5DvTkQ6XlwTIzfo9/6K5/7 bFZw==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1710401456; x=1711006256; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=j3hf+dqUOOr6+JyxflK1UanAlbRAjriUIe9XRXLJA0M=; b=GIj4mWANze9gwVLogX3ZFetgGi3Fs3dzrYEOkpTfsoD3hSqz0FkJOLOVYmflF/009E 9UpZwdfYcQpjNPy7tEN5kChJ2LDQ5IiDGDOunjCpd/TeGIEQlqkM8+DRjbmezuukY5W1 0uKNmrIV2PMbnycTQNGD4l4c395GeceDO8z0uUwkYR3sOPH/1XXjKb6YzDvA0JwEmmT9 8vfsONAkMA7Jwh/b99aE4HuQYnT2SNiyNqC5Q0GFgmhEyfhogrEw/6wmAj1Myb2H4Awb DWhoqRDVc06umZl7kj4nNS1XzedoLmhtJHDMKK/YYBmDpMIxz+7JcMCNvsKjejM+hhwz W/uQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710401456; x=1711006256; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=j3hf+dqUOOr6+JyxflK1UanAlbRAjriUIe9XRXLJA0M=; b=m+46UhbDTm0vvEGMDqSGxmg3b/kCrwWMajyQZF8jEi7RmczqhL54wqJBGkJT5oqJgm zTXlj1B/t+C25ljdN8zsAhp2d87UIWhx4grfyxBaiTkOW2hGYbMQ3WXOIYmGrI+0hQvl 7lVADENUt2C3uwyaTnVF3F9vUi9WdBR1efV76GbqOI9z3aU50N0nuFrp0K1bfF5cEIqD qK08iDR/CVt8ws1f45LF1zutKt/3aaeMCWndNj+HFZvEkTElC5DJaPB3Y5HYFvWFkwFY YqHzVkZ7LWHBq7lxYNDQ3D5BkvIpNX2XR4J1tCufBrb67+Y8JYk5KsYDGiQdmmYL91vu 6e3g== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCXQUgXG6ath8LFVZvueD1H9IpT5FW/hr75OldMEUls7RDzy9Jlif5vDdfWHeDsj0rjomCvdPukLhodE75RARxigRT4/q4Q= X-Gm-Message-State: AOJu0YzV8o7VB05VY6Ox32fgYCSnPjEJUAUHW1ZRvrMkeDSdUkW18OlJ cR7+cXSb2f3YjNxScuo4iT/GIkL+bN0pdkXTKMjUqmHqK85xz6Nw X-Google-Smtp-Source: AGHT+IFcJJQdfOym4lv+FGPnP3uq6M0CJKSqjUeAiMrDxty2mQ/chLz5vMCcEJCsk8fpXOns5qM2+g== X-Received: by 2002:adf:a454:0:b0:33e:7cc8:bd7c with SMTP id e20-20020adfa454000000b0033e7cc8bd7cmr502202wra.19.1710401456297; Thu, 14 Mar 2024 00:30:56 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:66cd:0:b0:33e:77b4:c05d with SMTP id k13-20020a5d66cd000000b0033e77b4c05dls230098wrw.0.-pod-prod-05-eu; Thu, 14 Mar 2024 00:30:54 -0700 (PDT) X-Received: by 2002:adf:ff87:0:b0:33e:c4c1:b45a with SMTP id j7-20020adfff87000000b0033ec4c1b45amr702931wrr.8.1710401454092; Thu, 14 Mar 2024 00:30:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1710401454; cv=none; d=google.com; s=arc-20160816; b=UB3/0pgs0e+/VnJ4mcTNTOPGo9mr9cr5TMM+7m/92h2/tMsbFnr4WjdieouAfHNfps RnmuKv36EWfttSaCRp1hK6Gi6kBmfWeDm3MCGg0Zpr1o8x9fwGhTnkPFBqHdNOm+q5M3 AP4K+Tf85YaEEbAa/d415uubu9tAimuJsU7psNQ4UHskJnxJkUeQ+b2HV96YJFKtQTKC ehLRYeK3zzncjGgYQI9UqBTC1JOE/LmftpV/VVHVUldnySLg4NKf11/cpNEyZpe2TzmU ampw0dG9p3QTjdTyKsD2haetCRxurSVT4YDulOS8r/VjEJr7W+0iuXNVYzbyDwHQPDi9 jqyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=owX1JO1KAHmznsCtmChnOOBL/knoW/E14MzygLcANHc=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=QEbqMqiI/oMSobFmettt9prcAh7FR/2usUyPdN90lr514j5glFYVyjm0dz62UvBNA8 rbmvqDxhSU/hPyaAUXtH+QDNnFsmIMoOF6b+PSbs25K5liFBDMyaDDEUlwWH5IgorX2X YEy9uqoiKPn8oVVCZ0kE+GKYt8hoYTbgtTqp1mUchvY7PbwMsMAF+pkf3aHKVHyCXnnF lKdPTtDS5aMv+GGSkoYkFN9D8p9sg2T9pIs2GAeUc0tWYTLjHKDg7cNoV4L97nT16U+I Yu0iKy8m2WWK2HxRbGFS4iy6h9ovpUcFcBRP/jb472/+u+PuLbE4JkFyeVX0ySqlMb5g NRgA==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id h8-20020a5d4fc8000000b0033da656f9edsi4480wrw.0.2024.03.14.00.30.54 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Mar 2024 00:30:54 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from hp15.localdomain (mm-209-185-213-37.mogilev.dynamic.pppoe.byfly.by [37.213.185.209]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 42E7UoA0021482 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:30:53 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v6 05/11] meta: Consider global debrepo context Date: Thu, 14 Mar 2024 10:27:22 +0300 Message-ID: <20240314073047.29465-6-ubely@ilbers.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240314073047.29465-1-ubely@ilbers.de> References: <20240314073047.29465-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= For host/target and cross/native modes different debrepo context (e.g., working dir) should be used. Signed-off-by: Uladzimir Bely --- meta/classes/dpkg-base.bbclass | 11 +++++++++++ meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc | 11 +++++++++++ 2 files changed, 22 insertions(+) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 30caedf9..361a7e9b 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -12,6 +12,17 @@ inherit repository inherit deb-dl-dir inherit essential +python __anonymous() { + distro_arch = d.getVar('DISTRO_ARCH') + host_arch = d.getVar('HOST_ARCH') + cross = d.getVar('ISAR_CROSS_COMPILE', True) + + if cross == "0" or host_arch == distro_arch: + d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR', True)) + else: + d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR', True)) +} + DEPENDS ?= "" RPROVIDES ?= "${PROVIDES}" diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc index d794c6fd..2755eaa8 100644 --- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc +++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc @@ -8,6 +8,17 @@ LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260 PV = "1.0" +python __anonymous() { + distro_arch = d.getVar('DISTRO_ARCH') + host_arch = d.getVar('HOST_ARCH') + variant = d.getVar('SBUILD_VARIANT', True) + + if variant == "target" or host_arch == distro_arch: + d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR', True)) + else: + d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR', True)) +} + inherit rootfs inherit compat From patchwork Thu Mar 14 07:27:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3411 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 14 Mar 2024 08:32:03 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f60.google.com (mail-lf1-f60.google.com [209.85.167.60]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 42E7V6wZ021620 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:31:06 +0100 Received: by mail-lf1-f60.google.com with SMTP id 2adb3069b0e04-5133f2e0485sf472991e87.0 for ; Thu, 14 Mar 2024 00:31:06 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710401458; cv=pass; d=google.com; s=arc-20160816; b=BktBJQjq1Vs4N9MHsLKIQuFZhE1+C9o+FGA8N6b4u68iUa8AnUv7q6LFy8oj0C/kd5 BCMK/bao2ex38Rcz4xvhwQrkauTm+gswrS6NeNhH1OglZ/y+cv4rHsFk2Hyuv2f/ibh1 4ebbO6J3hH0luk60QGdB5HgcdrclxtTOBBXhpzBjxoUKlcEmyG52Jj3+XlbMqt4IY60O U78PUAX6isPAdGFQKRMT1+wwTnUFVqJa7rbkTd3zA1yIZbtzKfecCc+ogQP6iZdfUeDJ xzo+zhsKmL4Q52FR/EJ/eWkkQRoZq+tqetmVpHUswrKFois1uCB58wYXN5b5TP1o+1Pm IXJQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=/Kd32Sh/p1wBKuxeYTryl/dHVZXrZ+JbgCOUf0ggJLM=; fh=EJCLMzL4QrRA1ga/MeUP8dBrLHjR4MUnsCuVqN8Ww3U=; b=Rg2Q6oCwSlxqUMzEjZC9KCkKqp6iEGSWXhmdH93qA7zxhUJQW7hk8/ka9l+nu/g54C cs1fxBHd7/t5NmR5A86NkBAVKfsHEMg4hG7iiOu46hPLCcNAOXpWRLsWz2WaYB9jb4wn vonCHriSlF43/JahAT1CmHxx/7vXFTpTOSTWi/vJY8ZiH7/dNVwMgWXoeKQblWHh/cDl RmwOzvl/xaBCyzRQhjJKL3QUAj7+04+3vp/h+4FWJoUsxKX3XJMXWNcUF9+f8KmY27qH KUOdVdLmnspkFTI8O3IvK/jRQkw8MO1a5Qcr7c8Qc7pFPI989DJVQnJnJBUWAH3q2kQW VRqQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1710401458; x=1711006258; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=/Kd32Sh/p1wBKuxeYTryl/dHVZXrZ+JbgCOUf0ggJLM=; b=i73/dPnT9hZgtsZqKvIggLPGH+VIrNlnEvwEqr3moKPqTES9iCdL5os/vlpo5d94lm EvUILg1V4B4tru7xv6Lo4vJnKETvSIZi7OVupyOwxteI2QqK1RQpzc16Ug/ChURj+rrw xnZDwOGcGLpPk6SjJ9nkHy4YxTiSkjP4lRLPu0MS6aDWJhxcXFAijVi6TS6T4KquKl/r 9K7+ehEX+p+CgnYi8VSB42qyE8YNaQOqGzpuRNEXadmHSA5KfK9Uv70GC2Km8b3/xm0n RknNisk32cwqnNDJYnv8rCv5ERbHQozADswrtGAF4GAwJ8Jn8Cp58hAJYKeLM0YcOJUM 8e1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710401458; x=1711006258; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=/Kd32Sh/p1wBKuxeYTryl/dHVZXrZ+JbgCOUf0ggJLM=; b=FzPSJI22hQwWicwQwtiO1sz4vCTnSjUt18xD/qqbymbpZ0ru/eXFhyslfwh89se2z9 ab54J1fIRpj4bM5O1g3rA1DZJT07Y+NVHyA+Pz8/Wqk0YG9lsOyrbrAJaFbHWJIC35wn Vsum4qkKUKwrra/Bjb64l4HfDWuyF8P6lqs76eW9xLvh3yOlOeH38YTAWbvVFI6Y4A3I 6OBGWElRvYHHcc5fY/E0cEPJhFz/8vn7+fyvj/OR2fJlA0zpxqYA3rYtWzF7/lPdwGG5 E0Cgu6jXOh60xsdn/8QTaHp2n/2DAB79I0/HibU4JdW5+HyFSEcBlMQgd03q7ilT0ckz IiyQ== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCWKO0iJah4k7gxMOm4tMMgcssck7hU9nWQbbVrTAxvSPjRzw19g1LPt75nSLtUURWjlKkFC1qCC0v/nLTx/j6FK/xE6gYg= X-Gm-Message-State: AOJu0YzHljEsNCcA875ClrcHhVymHvzwO8Oa5HUP9hHEdvWx+K3Y+voI QbbvQ/W43MidfJZeY00kZ9VU57lo/Vl0DDR0NaFCBTQAC2AdjhYg X-Google-Smtp-Source: AGHT+IHdcvq37oh5UqSrywAsTRjN9l4Eiqp3hPXtUqTp1msZ5HyiSLLTVftPfK5yJ5b7X3Sub/1u5w== X-Received: by 2002:a05:6512:3084:b0:513:c97f:6ba6 with SMTP id z4-20020a056512308400b00513c97f6ba6mr2006615lfd.15.1710401457556; Thu, 14 Mar 2024 00:30:57 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:b041:0:b0:2d4:6c0a:be15 with SMTP id d1-20020a2eb041000000b002d46c0abe15ls171859ljl.0.-pod-prod-00-eu; Thu, 14 Mar 2024 00:30:55 -0700 (PDT) X-Received: by 2002:a2e:9295:0:b0:2d4:6d7b:4b68 with SMTP id d21-20020a2e9295000000b002d46d7b4b68mr419566ljh.6.1710401454920; Thu, 14 Mar 2024 00:30:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1710401454; cv=none; d=google.com; s=arc-20160816; b=cHpuwazE/hkdbsUdjvDQML4mJQkLZXsCGG0dSXreUPMKEZALptTFrejwtvXXqun6My RUxmEXVja0gJ+pB2y+d9ZocQjhRfeQUqnC3qDkZOmTrcGnEWp/7cPTqZ7sIG10zgVqeF x9WgPiiMZsZzTIIwo+pZvO8FoBTVveUU0RPx0lilkb63Wwg48HXyoVKC7ONPFuOCfVFi XDMdvt/NPBdbEWQHIOZCGx8xoJu5gk9xxfYwe1DOHPaiBbwU8jxRbcxe1LRnTZ7l4yFx 4sqGB1Ie9NSkVsUNV/aqzboaNLHavPY7HcjCbyxIXQi/QVEzo6OiGmELEVIjQnDEYIbk CoVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=EJJJARxNXYE8s9kG74Vkd+z9AVmvWaTcsF2BWxhXv0I=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=tf688zhmclKmpyGdxJqR4vTHyzW9091I+FAP6UOOe6iyGyPyLMM1KR/Bn+Wy+Y2W+H 2r8rohpmrMbd4pAtrH3TIa4sT9r8iay2w/h+o3Ecbzjh+DqWpH4QFJNxiNKS3rMpVV8v ORmHYiVWioTWU7iZoPTryoeVljk4rzFnTn/36NzS+5vrS+1SKFC8cxj1nCYyz1O/HYSd WyFPNxn8hl0sNFT01A/KWHZDnFoa5tHjMsPab/Sa437sY5J6YYxqV2DocsT8VZOgwLVk 9/PX5BwJzo0H0Fstg/MF6jmW/Hvy22LJNUkJ2eMd7D/1Jtye4S7Om1/NAyzbBlNa3CfO qZ9A==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id t10-20020a05651c204a00b002d471e919d7si6891ljo.7.2024.03.14.00.30.54 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Mar 2024 00:30:54 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from hp15.localdomain (mm-209-185-213-37.mogilev.dynamic.pppoe.byfly.by [37.213.185.209]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 42E7UoA1021482 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:30:53 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v6 06/11] base-apt: Predownload packages to base-apt before install Date: Thu, 14 Mar 2024 10:27:23 +0300 Message-ID: <20240314073047.29465-7-ubely@ilbers.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240314073047.29465-1-ubely@ilbers.de> References: <20240314073047.29465-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch uses debrepo script to predownload packages to base-apt repository before they are installed in buildchroot/rootfs. Signed-off-by: Uladzimir Bely --- meta/classes/dpkg-base.bbclass | 27 +++++++++++++++++++- meta/classes/dpkg.bbclass | 8 ++++++ meta/classes/image-locales-extension.bbclass | 5 ++++ meta/classes/image-tools-extension.bbclass | 9 +++++++ meta/classes/rootfs.bbclass | 4 +++ 5 files changed, 52 insertions(+), 1 deletion(-) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 361a7e9b..0df47529 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -11,6 +11,7 @@ inherit terminal inherit repository inherit deb-dl-dir inherit essential +inherit debrepo python __anonymous() { distro_arch = d.getVar('DISTRO_ARCH') @@ -125,6 +126,14 @@ do_apt_fetch() { trap 'exit 1' INT HUP QUIT TERM ALRM USR1 trap 'schroot_cleanup' EXIT + debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${SRC_APT}" + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + schroot -r -c ${session_id} -d / -u root -- \ + sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" ' + " + fi + schroot -r -c ${session_id} -d / -u root -- \ rm /etc/apt/sources.list.d/isar-apt.list /etc/apt/preferences.d/isar-apt schroot -r -c ${session_id} -d / -- \ @@ -147,18 +156,31 @@ do_apt_fetch[network] = "${TASK_USE_NETWORK_AND_SUDO}" # Add dependency from the correct schroot: host or target do_apt_fetch[depends] += "${SCHROOT_DEP}" +# Debrepo context is created by target bootstrap, need this dependency too +do_apt_fetch[depends] += "isar-bootstrap-target:do_bootstrap" do_apt_unpack() { rm -rf ${S} schroot_create_configs + session_id=$(schroot -b -c ${SBUILD_CHROOT}) + echo "Started session: ${session_id}" + schroot_cleanup() { + schroot -q -f -e -c ${session_id} > /dev/null 2>&1 schroot_delete_configs } trap 'exit 1' INT HUP QUIT TERM ALRM USR1 trap 'schroot_cleanup' EXIT - schroot -d / -c ${SBUILD_CHROOT} -- \ + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + schroot -r -c ${session_id} -d / -u root -- \ + sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" ' + " + fi + + schroot -r -c ${session_id} -d / -- \ sh -c ' set -e for uri in $2; do @@ -168,6 +190,9 @@ do_apt_unpack() { dpkg-source -x "${dscfile}" "${PPS}" done' \ my_script "${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" "${SRC_APT}" + + # End chroot session + schroot -e -c ${session_id} schroot_delete_configs } do_apt_unpack[network] = "${TASK_USE_SUDO}" diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index 3fa9f604..9aba205a 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -43,6 +43,7 @@ dpkg_runbuild() { export PARALLEL_MAKE="${PARALLEL_MAKE}" rm -f ${SBUILD_CONFIG} + debrepo_handle_controlfile "${WORKDIR}/${PPS}/debian/control" env | while read -r line; do # Filter the same lines @@ -111,6 +112,11 @@ dpkg_runbuild() { DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog) DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -print) + locked_update_cmd=":" + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + locked_update_cmd="flock -x /base-apt/repo.lock -c 'apt-get -y update'" + fi + sbuild -A -n -c ${SBUILD_CHROOT} \ --host=${PACKAGE_ARCH} --build=${BUILD_ARCH} ${profiles} \ --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \ @@ -121,9 +127,11 @@ dpkg_runbuild() { --chroot-setup-commands="rm -f /var/log/dpkg.log" \ --chroot-setup-commands="mkdir -p ${deb_dir}" \ --chroot-setup-commands="find ${ext_deb_dir} -maxdepth 1 -name '*.deb' -exec ln -t ${deb_dir}/ -sf {} +" \ + --chroot-setup-commands="${locked_update_cmd}" \ --chroot-setup-commands="apt-get update -o Dir::Etc::SourceList=\"sources.list.d/isar-apt.list\" -o Dir::Etc::SourceParts=\"-\" -o APT::Get::List-Cleanup=\"0\"" \ --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \ --finished-build-commands="find ${deb_dir} -maxdepth 1 -type f -name '*.deb' -print -exec cp ${CP_FLAGS} -t ${ext_deb_dir}/ {} +" \ + --finished-build-commands="mkdir -p ${ext_root}" \ --finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \ --debbuildopts="--source-option=-I" \ --build-dir=${WORKDIR} --dist="isar" ${DSC_FILE} diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass index 9149d643..4dd93f1f 100644 --- a/meta/classes/image-locales-extension.bbclass +++ b/meta/classes/image-locales-extension.bbclass @@ -6,6 +6,8 @@ # This class extends the image.bbclass for setting locales and purging unneeded # ones. +inherit debrepo + LOCALE_GEN ?= "en_US.UTF-8 UTF-8\n\ en_US ISO-8859-1\n" LOCALE_DEFAULT ?= "en_US.UTF-8" @@ -29,6 +31,9 @@ ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += "image_install_localepurge_download" image_install_localepurge_download[weight] = "40" image_install_localepurge_download[network] = "${TASK_USE_NETWORK_AND_SUDO}" image_install_localepurge_download() { + debrepo_add_packages "${DEBREPO_WORKDIR}" "localepurge" + debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt" + sudo -E chroot '${ROOTFSDIR}' \ /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only localepurge } diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass index 46bdf78b..4e5bd550 100644 --- a/meta/classes/image-tools-extension.bbclass +++ b/meta/classes/image-tools-extension.bbclass @@ -6,6 +6,7 @@ # This file extends the image.bbclass to supply tools for futher imager functions inherit sbuild +inherit debrepo IMAGER_INSTALL ??= "" IMAGER_BUILD_DEPS ??= "" @@ -40,12 +41,20 @@ imager_run() { echo "Installing imager deps: ${local_install}" distro="${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" + debrepo_workdir=${DEBREPO_TARGET_DIR} if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then distro="${HOST_BASE_DISTRO}-${BASE_DISTRO_CODENAME}" + if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then + debrepo_workdir=${DEBREPO_HOST_DIR} + fi fi E="${@ isar_export_proxies(d)}" deb_dl_dir_import ${schroot_dir} ${distro} + + debrepo_add_packages --isarapt "${debrepo_workdir}" "${IMAGER_INSTALL}" + debrepo_update_apt_source_list "${schroot_dir}" "base-apt" + schroot -r -c ${session_id} -d / -u root -- sh -c " \ apt-get update \ -o Dir::Etc::SourceList='sources.list.d/isar-apt.list' \ diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index ddc5d12c..13e06ae1 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -2,6 +2,7 @@ # Copyright (c) Siemens AG, 2020 inherit deb-dl-dir +inherit debrepo ROOTFS_ARCH ?= "${DISTRO_ARCH}" ROOTFS_DISTRO ?= "${DISTRO}" @@ -153,6 +154,9 @@ rootfs_install_pkgs_download[weight] = "600" rootfs_install_pkgs_download[isar-apt-lock] = "release-after" rootfs_install_pkgs_download[network] = "${TASK_USE_NETWORK_AND_SUDO}" rootfs_install_pkgs_download() { + debrepo_add_packages --isarapt "${DEBREPO_WORKDIR}" "${ROOTFS_PACKAGES}" + debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt" + sudo -E chroot '${ROOTFSDIR}' \ /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES} } From patchwork Thu Mar 14 07:27:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3410 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 14 Mar 2024 08:32:00 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f63.google.com (mail-lf1-f63.google.com [209.85.167.63]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 42E7V6uu021612 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:31:06 +0100 Received: by mail-lf1-f63.google.com with SMTP id 2adb3069b0e04-513b8b37446sf831767e87.2 for ; Thu, 14 Mar 2024 00:31:06 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710401458; cv=pass; d=google.com; s=arc-20160816; b=QAjFTcHzCa6bPVVbbe2eU3H0aVYkNxVW0ty+wwOv///pS3TdomNqSSgZTHwRdDwRfg dPLfaTXwmdMNOpzlQ4Eb06WAcQhJ8u+6T6lWS+TDSzc8RVqinWv/OQvmfI7v1DRKVXji xVwF+8N/tMWvGvzS7B71OkuMUN7lGC1IIM43Fw7213kkjcwU0y6GPsjRCsEM8BKptHPu f1yDd0XzWEsO4KFMiXDgtX/3pI1Ms/lqBMh4tRrYNLT3oacWfx8zFQuiu5XKm6a/nN/D FBeREu9jK0jD74XF2rPyitDHluqWKlwk8ebNtyMWLDf5kBETcc233lKw78+6Plsd0xjS b6PQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=iSUhful2V++t4xBXKZCe//tdC/z+0DWNGaytUgXZbEw=; fh=oZfzfZAOOK+A+FKt0KZb3TKc8AByK1H8ZXl5UhfzM7w=; b=Ogp+ak/0rLHA4EiefX0bPk5/z2F2hB54Mg3bULoq5F7mfUFCrLp/XbzVJvO7Q1reU+ D1s+6kydd0kW6F/uDkFLaOoxTkc/k4VPh8Pubw5oIGKgJsTY+9LI83M/UNDW8TYYMRZm zLOVqJDTFvMV7GvQDQ4MC06jgLIilIDRNnZnh9J/R1SpgkL7qKzV7VoYHRryAg3LHQLM QS3ZYgR2f6CnnWq+gASLKixUJDCsvN42uXktjQXq+ssUk9ZFAF2Ngeh9V336Z9+hCR6u nkBbet8aIV0EwrBuc2YODkseEi/tD3q9sx/z1gm5ZWPWYgubnrSWJ4EXXJXfM3PrV8L4 3ZWA==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1710401458; x=1711006258; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=iSUhful2V++t4xBXKZCe//tdC/z+0DWNGaytUgXZbEw=; b=rQHAFOsLwJRq2BKCzxYeLqj0yHSMQ9Lo0q+7rc9T0vV5utOZhXTS1HvKC6strODylb pfzgaYVzwQBaVo3LgNR6M297+NcDVJ6eWA8pZqbQY1P6iV7ytRVCsEoju37yTEHDixN3 f6y3qX3UETSfofWZ+hDMDIY8GWUfAT7HP3YzokBIZpTDIBaOlc9JKWtCPOJTtfFmbE67 ZGvfsXJimkFr50ERCaFKKBuFZerYokXdKCuUcl00QvGfkoJrt7BzNII1jNrwh1HB8jAh h3tP2ytzOcLcUdHRmORALrAmhx7NhlPtRG2pqOcZPdI9oOKo9Yl+uFDxZNYA0yP7ctp8 STWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710401458; x=1711006258; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=iSUhful2V++t4xBXKZCe//tdC/z+0DWNGaytUgXZbEw=; b=xCtVf85g6mzmSMAWas9dp8bI0kSKwLLyYCL2DGj8r6K9IhLEZdxKjQGUTGMPxT8djj wIxqSDhuj2DCJ0f/kbwMMuByl8TmdmNfnn9sWrnUXYCzfVlCJ1eEEwuu12iOLqjCprca 58H0gra6K+JUNLU8PWuM+OysSXc1+0Z7M+mWprLcopiPOqb/8PnNZ7G+Qac23Cc5aV31 T0eB5rzyjAh+agSKC6nt6FBVPMbevV9UZ1N1FPzIRAFm3Wn/EEergOuxmX8u4g+ZY5Vu fiD4eBjnRsjxR6388YcUaOHv20l5IglRQkTZ+SZ+tx/MVG8oKHtjtswudHueHbu7vm3O qPig== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCUeI0Q0aWVLiPnVLUw/C542Mf2etcxcgNDYEgMxLCS5IvBrQeYEqODcOzuC9Y7f+fG6FOeen+KSQQ3YTLrm+R3BxSwUaP4= X-Gm-Message-State: AOJu0YyojdiBxaTnZ6hdR5j1lJhq359bq7xmPJUL4xHNqLo4pbq1fnc+ 1Gwqv4Cst1bAPbi4ChbZLoOdWRLrKzntAiBHkn5jCx6Sg7KRaH2v X-Google-Smtp-Source: AGHT+IGtE6MDxEN2Eg+aFSWccNdg+A71Pru2fIEiXd7Kt/+z0W2MrdpO0lnoTBJQ4UZ2SetgyoLXqQ== X-Received: by 2002:a05:6512:32a6:b0:513:cfa6:3ea3 with SMTP id q6-20020a05651232a600b00513cfa63ea3mr626518lfe.6.1710401457651; Thu, 14 Mar 2024 00:30:57 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:489a:0:b0:513:caa6:6fe2 with SMTP id x26-20020ac2489a000000b00513caa66fe2ls224279lfc.1.-pod-prod-01-eu; Thu, 14 Mar 2024 00:30:55 -0700 (PDT) X-Received: by 2002:ac2:5b92:0:b0:513:2a84:c45c with SMTP id o18-20020ac25b92000000b005132a84c45cmr540522lfn.28.1710401455183; Thu, 14 Mar 2024 00:30:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1710401455; cv=none; d=google.com; s=arc-20160816; b=05/S6Z5QcCDwULpshcYKeaWCbOS3074TFktkCIOawUZkTd9H5crfamlZfS2cDz/+ty vKSwY2MxFKvM7hLwRtGPZgTtVumgTYQvWoc6oXMiimp3N7zKE7qVmUQ3S04M4kbM2WU0 ZCXe3oZYlQqOOApsPz0GaUW5CpcqWcU3D93XucehkXwFjtYP2jTeXL48BjqsIg3u/qdL 60UBGcwGaCsJ0+bkOw+QWbCR0+C8moUWwp0rIs/d5k0Q0P3B1wUHqA9Iobh/PwEu/6dq hTVIu4TlzwhiZrHNwimaj88mo0GyfXV/jpcu5+lAogCHwhTwTsUnNd5PNQja3Gh4gnXn Z9cA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=yYychC9Njwku80EDzk/5wi/Z0p9Sky7lipyhrqj2+NA=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=zTR9OqI10p+Y/OYdHg9Q9MrC6Npe9HKp5bM0rSuLWwTgP7G3wQheDbyBGMcIJgARiY 1Go0E2Z82JlcGga2v6M5rh0SPARbG64IqOyg4vTD7ujeMmTPbcF5bLUanb033zcYTwEp 1VukOKE0W/zSR3Qp9gIcCyMZyJRKKTwgwIRMePGL+oQvwJ9Grrfv8yvV+shuAsam+gDi 1jS80J4FsBvwAEUvQK9pzFC41PjBptwp0MqxP8aMe8t9ZMUYYiGyoNb7d0+FZe3AFd0A +2E5w/+mU0L+AH39INjdi7F6/WSP65LViulhQm2gRNYmSkE0KHl3QwWGkgNvPk6IzuZT XxVQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id q1-20020ac24a61000000b00513c1ff7958si55570lfp.1.2024.03.14.00.30.55 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Mar 2024 00:30:55 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from hp15.localdomain (mm-209-185-213-37.mogilev.dynamic.pppoe.byfly.by [37.213.185.209]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 42E7UoA2021482 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:30:54 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v6 07/11] meta: Add cache-deb-src functionality in base-apt mode Date: Thu, 14 Mar 2024 10:27:24 +0300 Message-ID: <20240314073047.29465-8-ubely@ilbers.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240314073047.29465-1-ubely@ilbers.de> References: <20240314073047.29465-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= Fill base-apt repo with source packages. Signed-off-by: Uladzimir Bely --- meta/classes/deb-dl-dir.bbclass | 17 +++++++++++++++++ meta/classes/rootfs.bbclass | 3 +++ 2 files changed, 20 insertions(+) diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass index d36b7190..2bb05b31 100644 --- a/meta/classes/deb-dl-dir.bbclass +++ b/meta/classes/deb-dl-dir.bbclass @@ -41,6 +41,23 @@ debsrc_undo_mounts() { EOSUDO } +debsrc_fill_base_apt() { + export rootfs="$1" + + local srcpkgs="" + for package in $(find "${REPO_BASE_DIR}" -maxdepth 6 -type f -iname '*\.deb'); do + is_not_part_of_current_build "${package}" && continue + local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )" + local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )" + local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')" + local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc") + [ -n "$dscfile" ] && continue + + srcpkgs="${srcpkgs} ${src}=${version}" + done + debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${srcpkgs}" +} + debsrc_download() { export rootfs="$1" export rootfs_distro="$2" diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 13e06ae1..ad4e728a 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -235,6 +235,9 @@ cache_deb_src() { sudo cp -Trpn --reflink=auto "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/" deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME} + + debsrc_fill_base_apt ${ROOTFSDIR} + debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt" debsrc_download ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME} sudo rm -f "${ROOTFSDIR}"/etc/resolv.conf From patchwork Thu Mar 14 07:27:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3412 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 14 Mar 2024 08:32:04 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f56.google.com (mail-lf1-f56.google.com [209.85.167.56]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 42E7V7JR021686 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:31:07 +0100 Received: by mail-lf1-f56.google.com with SMTP id 2adb3069b0e04-513c9a5fe33sf934113e87.0 for ; Thu, 14 Mar 2024 00:31:07 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710401458; cv=pass; d=google.com; s=arc-20160816; b=Jo+9ODzPXd4ntlMKg0u5WrvooM/6mnnTb8BSRnkOWsQ6f5JdgLbBzMAUc2tihUGVFy boUakFKr93aKTUDwnlBgfiyqdOfEOZj9ZGShfr0BnX7+YAR0vTrClxwqGQFz2SCkOkxt YNVJHiBu+J8CEpIEF4j+VOOGtufw8mYoIt8TXCoC+p/LFjj8gWzvUpBt5IvVmPB8qRQs 9J6QVE6WH5w/0Ep4LCKsYczjjif5ZY5i1Pz2w4YuDAq8XmcUJLfDYknkJMP1PymimN88 C1oA6Ey57K3s9j/3RSlcmtz/CHvwA236qZbJMPXeSbTLczRK6DQOqXwdLwUnIdQ84Puh VtlA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=Gu/KlcnkJaoICYvihH/1HE1qbMDebqk17uoGAB6oMd8=; fh=h2Z3E+A9mv2PgZJzpSc3WLYbQ5Vco8h4luuWOaGL6cE=; b=dMz1kS9Ekyn1k97SYlu8TvIfOP9Lo864IW4PoZ8we0/RS43w4ifzdbbPf9U7V9utlo 97S3yzdoukfQXH+Z1xS38lvcZs6bCVOilGrmI/wlrLim+qRmgGyGkcnoxTn9mtaGilkL I6Yj6ZLgB7OHbEikhsCu+l/uy43Vkl9FuTcGMHbgAcpFfV+vPPzjKX144s7ntCo62lj5 LwoNk+bTOWE9LLzuaqsl1tr2jpiB+jCw3g/l5LNrpxNfD3g6omPQ/MVsQ3y/vXVEmqYP QnOY3XbQajYGfyVVG/jmKCQhKO9+Nx0CROpcZgkujbGjDbtd/driQskkSd+CYHK2qfUP KYEA==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1710401458; x=1711006258; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=Gu/KlcnkJaoICYvihH/1HE1qbMDebqk17uoGAB6oMd8=; b=RMZ/6OuFW1WTslYOABm2ZH53+/Z+vNIPJ+ZfR11uqCt4qa+k+m0YZNAzG68liPenes yCGQodqNgh0fAqbWuJBG40LBf5Bb4xPyExlNT5oziVVVacQf9/KrvTCMpauMxf9tmtT0 k0H324lnkL6uZ05wrsY25Udvk6PzySfBTx9F12KUCwXfDM2YBSMil5U/TqGKW7I0kha2 AOQ+RRxN8GLa+wRQfaNft85oveGZEx0kEpNtpLoPgWiSKh4R7Jw8VtYUX43xphmbte+v 82P2oDnf29GQqeD78vpk/Fj/XAbUPCx+ZppdNAZpClP/K0926rEC11uSOAAX2El+E2xY e2+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710401458; x=1711006258; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=Gu/KlcnkJaoICYvihH/1HE1qbMDebqk17uoGAB6oMd8=; b=KirUMsZMzDDG1FfalX38iXOzznyjj8Er+2cpLivXATzEdGIOyF8AKd2ne4eMNATzxH YqxDJNOP3sORvWQYKAStntfhOnopgbOEgltmr3HdRpueO5bSGYxocjtQx5Qc+wrX04jL zXgsTp6E5IRuL4UGqnr7cjayNl5dsulJhhTrgQfeclafSxvKH9Mm6JVs3Ak6YO4tnnkk LGvrD6Nu6yD9wDglAq7h1xREveupTItiKfOfk4LOt/ZaKpbLTBSKC/3TZRvTNgWEjiJs gdqLQ/KIEyidORECSGatjcVEpmHNfVpFJpUkUaFGeIZN3/MAzvJP+K1Kne+TE2VwgBkg JnQg== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCV8wFyvmXy8vhjcqjSzxPVKYDggrZNpk0FGerOtdca166tWG3lmYTegVXr/iBtFpA80w4Pc/SZaN62YmlHRzjH0xV2uexQ= X-Gm-Message-State: AOJu0YwNXwZe2bOGXC1n6orkQXewE+jMFgiqmK8kwbA3hWlcMC0wlgU9 7puJGua0zx7DsjZa+gGZlDIb1gJM7Bkid4iSHlD5qfkAq7BqsYlV X-Google-Smtp-Source: AGHT+IGuoT128rwSWvdAS8QxOuaVn9Gz8iJKMMjnI9xCIOmhCs2ux9PuDuN9sk6s99EGtkwFxCGZLA== X-Received: by 2002:a05:6512:3288:b0:513:cf81:488b with SMTP id p8-20020a056512328800b00513cf81488bmr478083lfe.25.1710401458029; Thu, 14 Mar 2024 00:30:58 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:47f1:0:b0:513:4766:2710 with SMTP id b17-20020ac247f1000000b0051347662710ls160766lfp.1.-pod-prod-00-eu; Thu, 14 Mar 2024 00:30:56 -0700 (PDT) X-Received: by 2002:a05:6512:10d3:b0:513:cc74:bbfa with SMTP id k19-20020a05651210d300b00513cc74bbfamr1262841lfg.9.1710401455672; Thu, 14 Mar 2024 00:30:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1710401455; cv=none; d=google.com; s=arc-20160816; b=elYL8c8D1zHH825S0IlPMBijZZ7s4mUgP8Zc/KGFOkRG+E3Bo6W7pX9DTBNyHbV1cr s10ep+11/vp+jAu7TkYO+Mpk82G2cjO5gLfcjleDy4thaZ9+VkQ1fwfsf7CMZawJSPYp jEvNA1rmM1WS/ustAV3j9Kp88c67E/gnROZqRc+2fuWpH7GYjL7e+aWC54k8qCT26mIy BRlmFV17SLvTt3uErY2KTUYMVLBb28/DKfB/sQ6iWfPIm1nKXilHF7VI9t3R2Kb+k9v1 w/BJsnoPu4czhHXzNitucjNuGC1XRqKnJwk1qCihqExdjFjFSXuqFPsCWMZwSMZaBDQ2 Xz1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=iI+9Ab5nnVukWMWH2xIrqNtQNulEGFCu5dNluVXlSuI=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=EJQtX6Fyw7VOxYMrZ4qkh7fOHHhjkyURlSVaNIinwehdeJ543z08dqCyEhbPDue8cw IOURF1Nyhrf7ffI7Ta6JhpNzEib9zmA6JmQOuXROb6qv7tBKqfUfO/umcNAQFqUHyMZu pMmX1xZew4V3CyYLI072oId4oK+He2bR/jte//KE9mToyCbOVTeFeWOHzRqB2l57fjab fdU0r7H1Nu6cjFMFPjvoHQA48Szc2HZ73yOw5ogPgWFAYPFPv5e7nYdrbT6U9qGiOmyV zLJzHg/yj+4pN1IofHaInTU74jfFTqdohyk90RifMa+KsCe+GvF7ihlk+ik9JxKVoW7F U2ug==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id q1-20020ac24a61000000b00513c1ff7958si55572lfp.1.2024.03.14.00.30.55 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Mar 2024 00:30:55 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from hp15.localdomain (mm-209-185-213-37.mogilev.dynamic.pppoe.byfly.by [37.213.185.209]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 42E7UoA3021482 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:30:54 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v6 08/11] testsuite: Set ISAR_PREFETCH_BASE_APT by default Date: Thu, 14 Mar 2024 10:27:25 +0300 Message-ID: <20240314073047.29465-9-ubely@ilbers.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240314073047.29465-1-ubely@ilbers.de> References: <20240314073047.29465-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This makes Isar use `base-apt` repo in different way. Any package installation is done from `base-apt` repo which is prepopulated from external mirrors. This behaviour is disabled by default for downstreams. To enable it, set the variable to "1", like isar does in local.conf.sample. In order to be able to run CI in old mode, allow CI read the option from the environment. Also, adjust some tests (like repro one) to make them work with ISAR_PREFETCH_BASE_APT set. Signed-off-by: Uladzimir Bely --- meta-test/conf/local.conf.sample | 3 +++ testsuite/cibase.py | 4 ++++ testsuite/cibuilder.py | 8 +++++++- 3 files changed, 14 insertions(+), 1 deletion(-) diff --git a/meta-test/conf/local.conf.sample b/meta-test/conf/local.conf.sample index f692f533..941d7cce 100644 --- a/meta-test/conf/local.conf.sample +++ b/meta-test/conf/local.conf.sample @@ -34,6 +34,9 @@ IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsc IMAGE_INSTALL:remove:qemuamd64-sb = "example-module-${KERNEL_NAME}" IMAGE_INSTALL:append:qemuamd64-sb = " example-module-signed-${KERNEL_NAME}" +# Use new base-apt behaviour +ISAR_PREFETCH_BASE_APT ?= "1" + # Users and groups USERS += "root" USER_root[password] ??= "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/" diff --git a/testsuite/cibase.py b/testsuite/cibase.py index 90591f32..74f6c971 100755 --- a/testsuite/cibase.py +++ b/testsuite/cibase.py @@ -43,9 +43,13 @@ class CIBaseTest(CIBuilder): self.fail('GPG import failed') try: + self.move_in_build_dir('tmp', 'tmp_before_repro') self.bitbake(targets, **kwargs) self.move_in_build_dir('tmp', 'tmp_middle_repro_%s' % ('signed' if signed else 'unsigned')) + + os.makedirs(f"{self.build_dir}/tmp/deploy/") + self.move_in_build_dir('tmp_middle_repro_%s/deploy/base-apt' % ('signed' if signed else 'unsigned'), 'tmp/deploy/base-apt') self.configure(gpg_pub_key=gpg_pub_key if signed else None, offline=True, sstate_dir="", **kwargs) self.bitbake(targets, **kwargs) diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py index fa30c2f5..08f82295 100755 --- a/testsuite/cibuilder.py +++ b/testsuite/cibuilder.py @@ -104,6 +104,9 @@ class CIBuilder(Test): # get parameters from environment distro_apt_premir = os.getenv('DISTRO_APT_PREMIRRORS') + # get prefetch base apt mode from environment + prefetch_base_apt = os.getenv('ISAR_PREFETCH_BASE_APT') + self.log.info(f'===================================================\n' f'Configuring build_dir {self.build_dir}\n' f' compat_arch = {compat_arch}\n' @@ -120,6 +123,7 @@ class CIBuilder(Test): f' sstate_dir = {sstate_dir}\n' f' ccache_dir = {ccache_dir}\n' f' image_install = {image_install}\n' + f' prefetch_base_apt = {prefetch_base_apt}\n' f'===================================================') # determine bitbake_args @@ -165,7 +169,9 @@ class CIBuilder(Test): if sstate_dir: f.write('SSTATE_DIR = "%s"\n' % sstate_dir) if image_install is not None: - f.write('IMAGE_INSTALL = "%s"' % image_install) + f.write('IMAGE_INSTALL = "%s"\n' % image_install) + if prefetch_base_apt == "0": + f.write('ISAR_PREFETCH_BASE_APT = "0"\n') # include ci_build.conf in local.conf with open(self.build_dir + '/conf/local.conf', 'r+') as f: From patchwork Thu Mar 14 07:27:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3408 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 14 Mar 2024 08:31:51 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f64.google.com (mail-lf1-f64.google.com [209.85.167.64]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 42E7V5sO021594 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:31:05 +0100 Received: by mail-lf1-f64.google.com with SMTP id 2adb3069b0e04-513a1ed3ff8sf753390e87.1 for ; Thu, 14 Mar 2024 00:31:05 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710401459; cv=pass; d=google.com; s=arc-20160816; b=ueDcfaxBYvs5tW2wW0du4HLVFczco1l8R4q93PqtCaCW5Sp7CxKbpqRfHWmTVWdJ9r w2aQEKMWo2sX0P3M1vCGS4QQ+X6vrOk6RyNIzxCvQbyQBVSXITmT0TX/fKRzh4lJ2XXh 7KGCio2URPSFJyJWbuzXhBNHXKVSQZIfPYI11xcW6iVaRVMUMjWsLPosrSzjZUFDexFS 8oddzIH6YSCUFc1eUg4W7UxfSZGwQYxdbPLbEl09Mms7QtJiCN1L959lrR5dAHYrU+dx j8w0tg8clSlHiQrQ10bBwdkXxPWh7k4DjPCW1+6YC61lNwXCUFoLaWjejrcAnUZjjlO7 c20A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=hj89RmFAf3Ye/kHaFqtGwrFuM3+oUnM1Nv4QPOlBNTY=; fh=mE+nfrSUSNwjV+pDg3fDx3MXhBzbqtglDl3xbEKov2I=; b=HjzQYyivdOccfs7udgiV+arJejHtlw7y2DB7F6VtIj2g9eqGOP8CZdINhy2XkAWFf7 IJ6yMzP0+X9ooM9yGUtYiiPfkyFlNEYgYx/G8tVoh1HlxLpT3vxWX+W1ZqmWx5AR0XEs gT5VXXHsbSV4kQTweBU1CtSFUFy3S+FtVB0u26KLpgo8VwzDEDOeVlARe94/JwziH2DH perSDbryyfwHAnihwt+LEA0SpM959euWt79568PqVJTj3oN7376rF28fjg+m2q3DfZwU FBrozPbPK17dZj8EK2AFRIp5tL4i7KwF2EAq07dWKLqdFGC6CKRM074tTzdjYaAOiss7 Cw6g==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1710401459; x=1711006259; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=hj89RmFAf3Ye/kHaFqtGwrFuM3+oUnM1Nv4QPOlBNTY=; b=O5l3noDrnNaQfMyAdbF2S8642qiIvEjmcUC3bJf3/eam1CPOK81cuqcahpvb7R5/Eo kphQnalNmHhippIvTHVw1njHB1sCP1K+kyNJo4E+u9ZOJXnJ7XEMQo9Mfj8UXtP9T1Zf Vi6LXDs0vaW4ei6GEQ1QbSyDqEPXC9og/PYL0+yvDSEoY2o1EkeianeTpsjhTIvmuFuG G2AsX1/dWTDmU4VXKWu0VFPmWWYWPDE9jj8gvv8X/jtRMV4hiMQeJSKYTl1uzit+Mec5 KAAFBOA/j1C82WRd9QYI5/U7Ruef053RLUdXSbflpPKEt2l6zIu2FwtYZLreFy+U8HtG 02Bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710401459; x=1711006259; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=hj89RmFAf3Ye/kHaFqtGwrFuM3+oUnM1Nv4QPOlBNTY=; b=EMCTncqzboaBJ/0M32Aoq2yedWpQhBn9PuhbvbBCzFka9jUOX9rz44SO3jL1CSAEY4 3vpA9aBsiZ9IlLym2GzSkEAW1+gRmzFzIqFcL2K7u6L6XTrXruJw/3qtp2Z1Qxuw/vQV e4CsFtUueLZ3CaDNZWNb0Pa2ZD1LM/CPJH9YBHbxZ1XexiGouKYLgs0Nsgij3MinbUXj 3HNRPlANUUh5oyklXH/z+l0eNlFCnOg444JA5OvrouzcjZDDXf3Qo9yvltDxghBwRnHG lFEJU76WUELMQ9E89h0DZYpDNiuRQxePVSoqSv2dnm080nKmjyLBvtDwWc72vlU4RYEZ 1QNw== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCXpdeuwiny2HXwZ9nQjVZREFETZDvEGD7wKFSrpPSvv593nmruQSAZRwJkdsxTb4hgdd4TXXE3e2ephM8FVWXR21ESRQOw= X-Gm-Message-State: AOJu0YwDTyrizGOaUqSjKuQfeLxyGRG25mnQDpaXeaA2REM620xsQyUU upLnav7QEl+UZvByWGjhzpI1kvonE0Y3qP9Q1YcSnkN2QwpVTzDD X-Google-Smtp-Source: AGHT+IGgAks27dk0UKHU1HlgWAicPXdLYUIpQb6Zv3DRqkOPnFbf4VJNqP2RPpehBseVeuPw7XGrPQ== X-Received: by 2002:a19:8c47:0:b0:513:ca96:dbdf with SMTP id i7-20020a198c47000000b00513ca96dbdfmr533086lfj.12.1710401458450; Thu, 14 Mar 2024 00:30:58 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:5968:0:b0:513:cf4e:d3fd with SMTP id h8-20020ac25968000000b00513cf4ed3fdls253275lfp.1.-pod-prod-08-eu; Thu, 14 Mar 2024 00:30:56 -0700 (PDT) X-Received: by 2002:a19:e006:0:b0:513:80cd:e807 with SMTP id x6-20020a19e006000000b0051380cde807mr532388lfg.20.1710401456101; Thu, 14 Mar 2024 00:30:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1710401456; cv=none; d=google.com; s=arc-20160816; b=h6k4ZYGD85fklbqLsZtQNsnm5X2t93/kGIsm9AjXGHi7XgpG1HhthQbR/3rWtpdM01 PXd0qO8QdBnWwW8jaUZI7ZFpENSJZ1MMuLazPgY83oHAL2zKKrAXbbkg3MU/oF9qGgQw KMxOr/bOrq9QT7oEqKPQ+T1HxWLOaUq1ka0nV5G/hJfVfkZgnVK776DZgT9v0Fd6bK+K n0p4EEWUSF3be22/irKfDRe+qR8PRY1Oy3Dd514R9t2Vf5gL6zz6CvFzgYUVbPvcTNAF xP+lweJRRRA/bKrpPcU4z4za3FvtqCY1qI2jOUmzlpIrX+06c8DekKIZjpgniBk7u/BB XNjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=dEdink+xqe5rmCd4PrZC8xhZnkAjYjgNJhdy89VdapQ=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=U5gfK3tiz+P8iiU8DuW0BAo/jIEel7MXnZRvrccWqO/c6lTRaVTFJ3Opk1qeAdxdDA 045QGh6+HbfCIk2OL9DsLZ7/rFbjZ+wk3256r5LBACAo+FiMeAD9SnRK5MNmHvAN7sIz okrnDWIJkFJx1sLolEXkzktAaBjn91zmnyCSNuWJfgoZr3Zdahi/grii0VHIiPdcEEVp hxb88R3yb1tZhnVpWILULAFvqA24tu4rcQVqUhUpEZEaUoIGLJ8/FbiXdSpukvtuhflQ WI/eP4RSVJQGtEJ2ZGhCEirtIX6dCLxN8DQGFgNYXLvGXAeE+10ld0zGspA81du+IC9q hb7g==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id n11-20020a05651203eb00b00513d337f357si14701lfq.8.2024.03.14.00.30.55 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Mar 2024 00:30:56 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from hp15.localdomain (mm-209-185-213-37.mogilev.dynamic.pppoe.byfly.by [37.213.185.209]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 42E7UoA4021482 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:30:55 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v6 09/11] Disable deb-dl-dir in base-apt prefetch mode Date: Thu, 14 Mar 2024 10:27:26 +0300 Message-ID: <20240314073047.29465-10-ubely@ilbers.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240314073047.29465-1-ubely@ilbers.de> References: <20240314073047.29465-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= Since all packages and source packages are placed to base-apt repo during build, there is no need to have them in one more place. Signed-off-by: Uladzimir Bely --- meta/classes/deb-dl-dir.bbclass | 2 ++ 1 file changed, 2 insertions(+) diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass index 2bb05b31..4e4741be 100644 --- a/meta/classes/deb-dl-dir.bbclass +++ b/meta/classes/deb-dl-dir.bbclass @@ -89,6 +89,7 @@ deb_dl_dir_import() { export rootfs="${1}" sudo mkdir -p "${rootfs}"/var/cache/apt/archives/ [ ! -d "${pc}" ] && return 0 + [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] && return 0 flock -s "${pc}".lock sudo -Es << 'EOSUDO' set -e printenv | grep -q BB_VERBOSE_LOGS && set -x @@ -106,6 +107,7 @@ deb_dl_dir_export() { export rootfs="${1}" export owner=$(id -u):$(id -g) mkdir -p "${pc}" + [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] && return 0 flock "${pc}".lock sudo -Es << 'EOSUDO' set -e printenv | grep -q BB_VERBOSE_LOGS && set -x From patchwork Thu Mar 14 07:27:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3409 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 14 Mar 2024 08:31:59 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f61.google.com (mail-wr1-f61.google.com [209.85.221.61]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 42E7V6OW021616 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:31:06 +0100 Received: by mail-wr1-f61.google.com with SMTP id ffacd0b85a97d-33ec9bdcaa7sf26401f8f.0 for ; Thu, 14 Mar 2024 00:31:06 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710401459; cv=pass; d=google.com; s=arc-20160816; b=GPSw9CMMSkNOj3hFRk37+VtVGPF4zJayUauCoIAN1qB/36aoiSoA5t7xOK+lJMeUfB HW1yU27cxQPJzGU7MzXNOnQ14o0ovMpBpfVj0syqlSDzKS1Z1kB0V+TcTRLIUiGqYte8 UEM06tqTFFPlYTlpUYGdNW72Gfv9gEt5CV3TdlY9BWr8nYVefE6yEmtusiMS5iXDsy7k UmfpsLB3EJSi0tOYaLKKLUDxr4jO7qrbCYWMtLYJBGbP8jVFvxHfq8XLeNstil4LCYUq gGUn/0n/eAS0NiEI2iyH1QeJMTqKvCi2sLn548Kv3/pAb8BDaWqTFcM13S8LptCUx9NI WLzA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=nQR7a09l6MPEmtNCI+joTy9q5IY3ckOw1cxNP02qKy8=; fh=XOfUjuidmgpTw/BC2oG5euil8lvWDYfdWvbB3+CshhY=; b=hjrp5xrCd5y46KCM+01R7lPkQUBL7NYic6s4Y59CViZz7FxqhJlT1lbqGutbXM0ZHn Oo+hE+R8a3Z+JFOV3s7uHdDCLwYCjk6bT5sQiPYvMgjrlE366KbepfL959+b9b23otJY 49uXZtZDbP7QdfqAK9LfpQK8m8OFmlT/+5+Nh//WvS3+dO1iuRfSdHBCde8Z6ye+GX3L MiZBRb9FmLOYLs8Cbi/LSow2hjb4LQ3d2lUIWbDpEu8PK4pyxst4EP46x0DKWAY/Qtsp fXw9RAMBrHrdyMcncLNi7SlivJqVDwgsHma52WmRVPPq2GYABsVoIbD/ptfSWZRXh2bP NdAA==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1710401459; x=1711006259; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=nQR7a09l6MPEmtNCI+joTy9q5IY3ckOw1cxNP02qKy8=; b=boJ7sXiIgHqy19E+pPAeE5inrN2xzv8P6cC6HHnmJNfv7dEACm9QW9JNlHrKJMI2Uh fLs7hxObOuPEMEoru8/ri+KcFpYGZU7cuD6K70/umb7ZXS52ivZIE4DtHTuFDJjdXhLk zBEbQkISqWoWPWRVDhG+i1uSIhU8vVhabFBxfCIgK9hftF97C8MF5X8Vz+nqz6hnQpT2 BJpY2nHw72k9xI1/BVZP6y/egpuspJb2NvTmUAQuwYqzxHXfxWHShdp55lzFL3NUODTR hk/FwU07PAFnzIlfIhz/2F2j4rN910tH8f3GHI/oI/pFVWkgcUsxoqXlp6Jlop75ns+2 v81A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710401459; x=1711006259; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=nQR7a09l6MPEmtNCI+joTy9q5IY3ckOw1cxNP02qKy8=; b=YnabSF4bdzPnUmufpfN1/CAG/shVfhJtTJvF506KR1VoXxD8XuWLJpcwHXKV7Dhjzt o+HMom0y3nwiWWYSKVcbbe7YI3hQXuu/g4gYszlxQrJ4iJvOkuJGi8GmCw71TSpIZEHT m1Rx2WtlRuh7I8YdWxY97GB+/wBJjd6My2SSUr1AiS4S9DIdr8TQChQuFDCw7YIj5HxS FPzugOATMLeOUaFx4H9lDcwO0YCwF4u1n1O4QfH97Y8a6hewlqhliUgCC2JFtpO7tv18 VIC6NaHX068oV22Dn0vZY6TXAmSLDViEtpTMpCXm5nTnxjTE/sN8+LMp186oqhBl2SaP QFiQ== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCWrd5O4XbFrEc6Z2Y42fkAk8iZUykiXdy7e6SPVGUmfAwTaCNUBM8o5T31sNp0JWRLAQ3nRzN9ywvuXtjCuwBgXmGB1DIM= X-Gm-Message-State: AOJu0YxiNksZQaOyFsn9N8fqOTmXe7bv0Z4J7/AiswCJeYJ5pX7LKtu2 3cASOUUqbSenTALh3OAxbnrBArPcoCNJsaMeLmWM3DwZ0cgC6Q0C X-Google-Smtp-Source: AGHT+IEA/D52dJ/8c2x2asUuOB2p5FM2r53POzWfyonxY0FD/GCg4reID3iAii3vJHdb7fS180DvkA== X-Received: by 2002:a05:600c:b8d:b0:413:eb09:9bd8 with SMTP id fl13-20020a05600c0b8d00b00413eb099bd8mr804431wmb.0.1710401458180; Thu, 14 Mar 2024 00:30:58 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:450a:b0:412:e4f6:74a3 with SMTP id t10-20020a05600c450a00b00412e4f674a3ls309983wmo.1.-pod-prod-01-eu; Thu, 14 Mar 2024 00:30:56 -0700 (PDT) X-Received: by 2002:a05:600c:3b0e:b0:413:1651:6bb2 with SMTP id m14-20020a05600c3b0e00b0041316516bb2mr870515wms.8.1710401456181; Thu, 14 Mar 2024 00:30:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1710401456; cv=none; d=google.com; s=arc-20160816; b=F/omqfQTGsX5yXlImMSIEiJZClwBMKTt0Mv6KYEoUfybJjvvcfFFhu8/qhaNJ7pB8n bSyB8BDzTpp8dfngrFI4homQHmM0fz03qoY639n294YUHQh7XaAcTK/7oYDg0c/g4+ds SOd0PirhN+alQAJ3m0OIwQSIIZYfyn7nPLFc2pWAikpAbpkW/B/Lqa7ze4haLhey3ax2 13w4V0Ds8hOaGc8J3c9IZv9AKcxAtEo2lGjjS2Za7HHednPLRRbsz2xGkDbzNISsaDCQ MugToz3MummxM/j9XV4/suExp6DyTrE1JtOFM/bB9KmxwPWQUC309wYPPI5zbD0+1mgO TLHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=hDVqdMDX4qqmCWU+BONJeE21l6eGm38lACRcelAPfYI=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=d/LHKubI9n57loFzNPWYQdtL4XBuNvepZwZO8mIArQRPc7voWIbgH2nsR0+uh5gkd6 gzwYEZDE9fUUstYXgtqDd10JE3CRl5LOo9H8aI8RQq7wNsJhLG7etwTvL400oao2/au6 QrgVHbk4HF2fXhhmCYtjEbyvy27b2PXXXobFxs3Wdt06fPq2ijGKHSiUcOeFxQjgMhQm ejO8E2HNU5GqXYQRy8Q2iO/ePFe+1ycIPQ8GWEIjClzodTPX8WE5YfiLDe03F04MBStb NI/NGzL5QZ7xzY4YMqk9fJdr7ksfS8Hwv9o1fbTfHWaa5Jf+LcaXRPKvvJp6x5D7KewC EG2A==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id fl21-20020a05600c0b9500b00413f4cb2054si17945wmb.1.2024.03.14.00.30.56 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Mar 2024 00:30:56 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from hp15.localdomain (mm-209-185-213-37.mogilev.dynamic.pppoe.byfly.by [37.213.185.209]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 42E7UoA5021482 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:30:55 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v6 10/11] kas: Add PREFETCH_BASE_APT config entry Date: Thu, 14 Mar 2024 10:27:27 +0300 Message-ID: <20240314073047.29465-11-ubely@ilbers.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240314073047.29465-1-ubely@ilbers.de> References: <20240314073047.29465-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This option allows to set ISAR_PREFETCH_BASE_APT to "0" or "1" and choose between old and new base-apt behaviour. Signed-off-by: Uladzimir Bely --- kas/opt/Kconfig | 13 +++++++++++++ kas/opt/prefetch-base-apt.yaml | 9 +++++++++ 2 files changed, 22 insertions(+) create mode 100644 kas/opt/prefetch-base-apt.yaml diff --git a/kas/opt/Kconfig b/kas/opt/Kconfig index 1619a4cf..493f3ec6 100644 --- a/kas/opt/Kconfig +++ b/kas/opt/Kconfig @@ -146,3 +146,16 @@ config KAS_INCLUDE_CACHE_DEB_SRC string default "kas/opt/cache-deb-src.yaml" depends on CACHE_DEB_SRC + + +config PREFETCH_BASE_APT + bool "Prefetch base-apt repo" + default y + help + This makse Isar always take packages from base-apt repository where they + are prefetched by debrepo script before requested. + +config KAS_INCLUDE_PREFETCH_BASE_APT + string + default "kas/opt/prefetch-base-apt.yaml" + depends on PREFETCH_BASE_APT diff --git a/kas/opt/prefetch-base-apt.yaml b/kas/opt/prefetch-base-apt.yaml new file mode 100644 index 00000000..0cbdb93f --- /dev/null +++ b/kas/opt/prefetch-base-apt.yaml @@ -0,0 +1,9 @@ +# This software is a part of Isar. +# Copyright (C) 2024 ilbers GmbH + +header: + version: 14 + +local_conf_header: + prefetch-base-apt: | + ISAR_PREFETCH_BASE_APT = "1" From patchwork Thu Mar 14 07:27:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3407 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 14 Mar 2024 08:31:46 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f61.google.com (mail-lf1-f61.google.com [209.85.167.61]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 42E7V4aJ021591 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:31:05 +0100 Received: by mail-lf1-f61.google.com with SMTP id 2adb3069b0e04-513c9f60a60sf637726e87.3 for ; Thu, 14 Mar 2024 00:31:05 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710401459; cv=pass; d=google.com; s=arc-20160816; b=MSL/6eEEwDgW0L4mYdVCtv8Orq6zcYNIzMH8SF4qeIEX1I+PzjFmrZzYQHcZR01Qar K1ZvvOcWbKyyJ0FOfnbRJPot+MRrWDdNuO6PUKuJW9yP+FERaK6hBJJ+Wiv5MVzwH7tZ j3cKGZSNZCqxhqtlSuDxgx0ttQaq92nR+9mTVWvyFsLa9vqnTfJalpgcGcYfzRUVw8cr 5fxnck8mLbsieScTDn8jgsblVVs6vZJe8LbVCndgop4RzxXjSvEvXx4Pi4taT29Zm1HC wqQUwAr5P5b24Gda/+ze/CsKN87zMWP1EYbdeCN2ycnmrrv6bE/ioycL4ztcQRrVVfzm vnqg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=O1qvl3Jp8IbBdTngy0gkuHqoBTF9sSU90N+RAqy0+1c=; fh=TcdJoYWi01U0KAyuQ3bpPbggGBifVu4nBWWTx0tvwfk=; b=UZ64aHXylPgUGTok05HMovaYfyVGM16/F11hhzkhMh/0hHBO6+61NMBucW+dl7oaqJ 3H23WT8iWmQRmkuGFZclS3IL/cFkSNMaGxPYYMPTLe9C2JgHPxMteqi5iyJjuepzGs1w QDEr0H8UbhkDxoDuFRPjg47oUGOVkbRJ1qW+9/KukNfoylAhcCbHa+BGcKvppIwIirtM sSvZ0lTAAthSNqqg3+XmtT0POGRQZQmqbrcezvMouQnyeXV4SaI1MGHqAXGSb0XCltRk NNuAk8xh09mn8vtC6M6u6YgjWPV79fO9mANQO3g5ch6l9r+5MWXsvOr28ilGfBXOpTGu K0hQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1710401459; x=1711006259; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=O1qvl3Jp8IbBdTngy0gkuHqoBTF9sSU90N+RAqy0+1c=; b=FBEWgBozxWYxJ+O1NcM7F8+Cha2eNJiSjmHyXOiwyuRMEsLNteFSxjA9ze/415lc9D U/onLlRA38OXBmEZfszTUlOQgxUknT5X9b+yprM8wOtYlWwV6zo2717+g33rWT9VBlKd qKroEzjEXayIBIZVWI2iH6QcB5zxNPPAltiY2w9Gwn6++uib9g0fuliv1UWYLJCxzMeG q3DA0lv/Nc5gkhaWhavh032cXwgGevQxRGo7LyiJPfmXhe5ePZd1K4RbOwfBLVWTE6/1 sIq0JEbf4I/7AHB/+XoqRhJ+d9Qrun0wd9yvGhrKs65yEroYJKP018ThsKDoHUnOM9+c nMlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710401459; x=1711006259; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=O1qvl3Jp8IbBdTngy0gkuHqoBTF9sSU90N+RAqy0+1c=; b=wI5eLUushhv6jYOCDPDgwxqvMwctyHKnfLf8vur9KSDrrLa2J0cyueW15iZHiTM/uc uCRi82by5kQ+3f8wYfdqXM8sHDBV7j4wW5JtgRsB9WbbT/QKQ69hk3mOGuxb5Kp/ZSei TqTGv2EiU2m7KI2g1/FgkeAzd0WV0oBqfICEk7mbmeu2T6pl//V4OI02qhxnC6yPBXZF nLmVXc9gkJl+ViOTShu9dgEDIrXAt2cbvlpb3QcEqGnDz+/XcWHBnWaGhXMKbKYjPv7q 0TN0uQyGQjzZasbSbZu2v7tHinoECERgw/+S/kQy+YRNWCK3d+mrl3m83BTBXnusspFF pEkA== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCWlQXmi2KOSEOdVOPv0SHM+dh7Tjc+m5xEIySS2kPVshgAsqcdbIlPuB9i6fWAIuOVxO0C/5V1LSVVQGsKv023pCuwHptI= X-Gm-Message-State: AOJu0Yx8h1Ju230cQbsFl5vh/eWcdFrAWSyOKAmJkG+cWV0P15/1Wl2F h4S6M1HtZsaXFYoWTXp1ji3lt0/hN+iIlSi1Sw0IXXmkDQmJpyP9 X-Google-Smtp-Source: AGHT+IEo8Qsv/IcvNwlLmjq2JZ2ym12vVItNROuAy1gOGVWU4nA/FsFWb3xbIZaBS4GbR0gdcf1udg== X-Received: by 2002:ac2:4d8a:0:b0:513:177e:4254 with SMTP id g10-20020ac24d8a000000b00513177e4254mr495278lfe.14.1710401458468; Thu, 14 Mar 2024 00:30:58 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:5125:b0:413:c1d:cfa5 with SMTP id o37-20020a05600c512500b004130c1dcfa5ls316036wms.1.-pod-prod-09-eu; Thu, 14 Mar 2024 00:30:56 -0700 (PDT) X-Received: by 2002:a05:600c:5013:b0:413:f3b4:fd6b with SMTP id n19-20020a05600c501300b00413f3b4fd6bmr398858wmr.7.1710401456518; Thu, 14 Mar 2024 00:30:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1710401456; cv=none; d=google.com; s=arc-20160816; b=Y3le9dIlkha0A0+o+PbghAzOLfpNeVChA6xtTqgevbgyJlhxVnPViTOyjRRQlK4cFO zsAYG3jjN5xe7SlQEARHaXGkKjmdEHuj00udAjAH2ezLxp5XqZzT85zpbZFwj+wcCRZU p1Z1vpM9KXFtC688ycjG3Q1Pg8cZHH6q8kZsjq8QSkmpSCLOIOVUveJej7bGOMN2bmQu /RrwB7WtTYkHy3KJRvbyJxgGjUbp1t1sBSe/xCHtYedxAtCW/04/r6gY+fO4XtZ6sTfs Cc7vjNA0DPGT/w7cuMnI7nIrMO7oz66OaTMxCnT+oKFWlA7RnkOf9bEHWtPQxQp+xMro ihmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=5CWp3OmA9+Yw2s/raugYEsntGWgTd8rDDdctQUTHmIg=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=cWm/gxZ8ZSuBC4oGAgk5V/VE/bqjwu9VEHWC+R7QnmkQoSvfFIZ4rALu/aJjMJR6VA j0tXEHzX5paF2SyYioc7BW2Z36O0YIWDvQlSi6FdfuCcjwW3JSZT6ZBW855emG5EiSIC owmcNBHK2aYprkUyw/jca5bQUbeYF2LfsdpmswwpjKYW9w+z15KFWmAuKX6KvX+iwaRd iFHOqyxCqSP2zCesgMm3q6iVcZM0+e0MeI9gIyPfDLJYX8MN4Yf++qNtSuQM3sPwdsIE 9CdypaBl92Ut6Y7evdu+Zr9YzSrUFRLME5i92qvISMQM0JY96N9rCM75+kdC9lZ38g3l n33A==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id j13-20020a05600c190d00b00413ebf7640dsi156610wmq.1.2024.03.14.00.30.56 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Mar 2024 00:30:56 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from hp15.localdomain (mm-209-185-213-37.mogilev.dynamic.pppoe.byfly.by [37.213.185.209]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 42E7UoA6021482 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:30:55 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v6 11/11] ci_build.sh: Install python3-apt if not installed Date: Thu, 14 Mar 2024 10:27:28 +0300 Message-ID: <20240314073047.29465-12-ubely@ilbers.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240314073047.29465-1-ubely@ilbers.de> References: <20240314073047.29465-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This is mostly related to gitlab CI that migth use an image without preinstalled python3-apt. Also, make system python packages available in virtualenv. Signed-off-by: Uladzimir Bely --- scripts/ci_build.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh index 51ed3920..b2fa6b9c 100755 --- a/scripts/ci_build.sh +++ b/scripts/ci_build.sh @@ -19,7 +19,7 @@ if ! command -v avocado > /dev/null; then sudo apt-get update -qq sudo apt-get install -y virtualenv rm -rf /tmp/avocado_venv - virtualenv --python python3 /tmp/avocado_venv + virtualenv --python python3 /tmp/avocado_venv --system-site-packages # shellcheck source=/dev/null source /tmp/avocado_venv/bin/activate pip install avocado-framework==100.1 @@ -131,6 +131,12 @@ if echo "$TAGS" | grep -Fqive "-startvm"; then fi fi +# install python3-apt +if [ ! -f /usr/share/doc/python3-apt/copyright ]; then + sudo apt-get update -qq + sudo apt-get install -y python3-apt +fi + # Provide working path mkdir -p .config/avocado cat < .config/avocado/avocado.conf