From patchwork Thu Jul 25 15:07:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3723 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 25 Jul 2024 17:10:26 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f192.google.com (mail-lj1-f192.google.com [209.85.208.192]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46PFAPmw007151 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:25 +0200 Received: by mail-lj1-f192.google.com with SMTP id 38308e7fff4ca-2ef2e57fb7csf2927641fa.1 for ; Thu, 25 Jul 2024 08:10:25 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721920218; cv=pass; d=google.com; s=arc-20160816; b=zzQR1TplQ2WNklnQq7WAYb65CYLEzimkEIrQtmGPFgjJV2smLyFQihiA6tHgG4XRzI hxsc/kw6ruVTedPci7/RYD+6qD0ta98Ei70kh2NlBz1ca7PM5FCMQFAzviq0YNWcyhWO OPbMWdU+B4TeKpjpjNunDKg8CZsrbcbV8eE3hvTwh6VD+TDe4ATLrrPzQSjE/9dQIljb uXHI1Ygalsu9lJeXyZIQgq3b51/JHSlhF0rxFunKnE9HAmjBfx1J35Fk2SrtFOTJESTP ltQ7LDwAlF7DwowiAvBMQHv8V2VNT1+3VaE6obkR2viXeF0eBLKKrJHq6P36OIrREZbw sJBg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=tFdnmjJYAimkRtRyVB0aEXo1AHWmO1bsld4noT4l5DA=; fh=+hVEvCC+In4m/obfyGCIZ2DJWSnHPAKssFTJAqmnoAw=; b=IYTL4Ue/IPoQRgK88gHUOxtbYs4DuVccj3a7gZ1eUlj13kCrCHQLZz6xmT1PoeJa46 K174Hza6Yc86KQLAaj+0STnDJMbzh42C85cDW1xF8WCv/7PkWmuu7+2BsysiYneheda7 Gh5h3kushZV76cBJJGfeNx+U6pUb7sw7qxKXMT827av0l9EjxjMhrq4VmJlFWN/ws5vN XjkOk20XxYN8toj3isR+0vlu1efyVNyEsHGEBQZooT+9LjHdobbK3j1Ta5GgeDtpyZKt +Fvp5OJV3K0sTicC+DSv8dOipNLPbYfn+dDMsR8rG7ExVaEGPey/QzvIF3Cd1sJffdsB QsSw==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721920218; x=1722525018; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=tFdnmjJYAimkRtRyVB0aEXo1AHWmO1bsld4noT4l5DA=; b=HcojkmwQRQsdouX8BreqmdHem7Q0GKO3K6MSsTJA6vwtyEFA7yK+hVT8koXqAWcGHI Y1zGs/RyaS/wqF4I1hU62Kxzf4CZxeW/GxvYa27+JgGx4mFGtq7Egnk2wulGCV17LysI irPwStcZuoOqKK6AxLTB2krY6ONrS5kr2bJxU3bwy2+oBL77gzWJF6LnC8WqyTGcTkRp OV+RoAo7KQjUhooJKRIOO0+Lnoz0JC5lgWRGRzSsQDmiEuSqRVTS830uPqWr8s2j52Wo Cu1D0HQKCfYjTFy80Z0VoiAfbxSP6oE2iVVOv8zWVr0d9RRJS8+tD7tHb6qikn26x/HC BGEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721920218; x=1722525018; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=tFdnmjJYAimkRtRyVB0aEXo1AHWmO1bsld4noT4l5DA=; b=PWsr7GCwiZyLQnlIglQkmPDOd0VBqIGXAVSBowxmnu35RKVISWXLuMI79td7QhFdUb TfoodEB987TWQpeJRFf0lml2Id8Rd5RmIDBIRCuRrpWfmskEUMsxz9V0+Fyypavc2aUx 2X6vGtqNUDV/uUpRgRyfLql4UDUHWyazouZUVsbpyAzFltlWz86hVVXNpXPyA4qfiqXI 6NV0bTT9R/UHp8tiyzfbgT5mEqk/YRyQwisNg7/yTcvtAe1pgrD54ZOUQ/cii0YiX6rg h7aCUn8zw6Jg3hFT6JycEhUCV0KkuLFfSfoCIMAweZ/xFmXJDlJ9YgVc+A5qeGJ/pLML bfpg== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCVLl8J98CI+DezTX49l0BEBmWBQIXjyoRz0lmjwewHDv/v69Fnp8d5XfArkOtaRcBzuFe1VnIIVYyadaerlyrkycVMQTsI= X-Gm-Message-State: AOJu0YzuxuNp7DRQeZoDZiw3aHaq3JujPLWkrpMDgJ1Q/OkmPgbuTUUK 8d5Sg4g3AY1LNvww/tGnFg3nkmd16lfXOIrMcujOr2LzUaHFCh3i X-Google-Smtp-Source: AGHT+IGAd4GNSOjYn+mwaVLPoYIvZG/Rw1mQXCx0rqMLZPqRK4U9GlVFczAFwSZtna0oQHnnz5W5rQ== X-Received: by 2002:a05:651c:a07:b0:2f0:1e0a:4696 with SMTP id 38308e7fff4ca-2f039c4dda8mr26786391fa.7.1721920217905; Thu, 25 Jul 2024 08:10:17 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:a54e:0:b0:2ef:1eb3:4741 with SMTP id 38308e7fff4ca-2f03a2bb792ls5076091fa.0.-pod-prod-01-eu; Thu, 25 Jul 2024 08:10:15 -0700 (PDT) X-Received: by 2002:a2e:b610:0:b0:2ef:2893:796d with SMTP id 38308e7fff4ca-2f039df9347mr24393971fa.46.1721920215513; Thu, 25 Jul 2024 08:10:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721920215; cv=none; d=google.com; s=arc-20160816; b=Ykho8iNcAP5eo3faqfNOLAygsFzNGohKNPoDwE3AdwoIUfpSz+SLS6K5LUKXMb7qIu i5R3/v7iAQIf8ioFR3bDhoicUNENZepcKDqn1HhmMlYmjIGaoriKRrrqqZ/aJ2qPLhL3 P8MciFPe/Ui7q4wvZBCSBAStkTBXVKrCcD/72wGbasR6nP/aPId7hd9UjZzXwSJNZQ0e BqoYBeYJepQdtyvCqRqFFfvoWk/wpb37YyGrZteynPM073bpGzgvP8kQA0WwzDm7zXzK lBNINEVVZlqRypl9+3vNrz+wvv2sMuMjlbTHqhiPH20BbOmZgzNXkB436b1wetjcXhKd tMlg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=Cl4l/LeDcoGM0LKNP1fVcPZc8EYb+AY6CHYrNGGYQAc=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=UJohqnFWY52ioYDnbgsWUOZtfs8Yc0KqCuyF/1QT87/WBju5KtDfNuT2/XoTV93Tpr ZkxiTyDkAUnyxX+8LsXmQcs5Sy7KoZjcFyq+lD3msAptDgiXUpIijFYc5g/s8MMa95fu KcZRl8+gsI0tK206uVbhjk4dMkkKNyeFp8YXolaFJgHSjwBcWy7HWKAtUiEhKN0FLeVE x5Nqr5G2J//pItDeCRBCU7Yg2+qn/0RzOxNIWxyWCRzkTjr6yz8NxILMZpK9IaF5t7ex J3UKdzJGxkbTW7XloKXQfBEk4zkfb495CprFbLknKnWAt5/fg8fM11HYojKkb9OylR61 XFSA==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 38308e7fff4ca-2f03d04cf58si364571fa.8.2024.07.25.08.10.15 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 25 Jul 2024 08:10:15 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 46PFACKX007063 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:12 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v7 01/10] scripts: Add debrepo python script handling base-apt Date: Thu, 25 Jul 2024 18:07:33 +0300 Message-ID: <20240725151006.2129-2-ubely@ilbers.de> X-Mailer: git-send-email 2.44.2 In-Reply-To: <20240725151006.2129-1-ubely@ilbers.de> References: <20240725151006.2129-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This is the main utility responsible for prefetching packages into local `base-apt` repo from external Debian mirrors. It uses python-apt module and requires some kind of minimal `rootfs` to work (let's call it "debrepo context"). Once initialized with `--init --workdir=`, it stores the initial configuration in `repo.opts` file inside the context and uses it at futher calls. In future, the logic `debrepo` script implements could be directly implemented inside bitbake classes. Signed-off-by: Uladzimir Bely --- scripts/debrepo | 590 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 590 insertions(+) create mode 100755 scripts/debrepo diff --git a/scripts/debrepo b/scripts/debrepo new file mode 100755 index 00000000..b96fa58b --- /dev/null +++ b/scripts/debrepo @@ -0,0 +1,590 @@ +#!/usr/bin/env python3 + +""" +# This software is a part of Isar. +# Copyright (C) 2024 ilbers GmbH + +# debrepo: build Debian-like repo using "python3-apt" library. + +When building the image, Isar downloads required Debian packages from external +mirrors. After build completed, it can pick all downloaded packages from DL_DIR +and build local 'base-apt' Debian-like repo from them. + +This tool allows to download packages and create local repo in advance. So, +Isar just uses this local repository and does not interact with external +mirrors. Such approach makes deb-dl import/export functionality redundant. + +Script `debrepo` works in so-called "context" directory. It means some +rootfs-like directory with bare minimum of directories/files required for +python3-apt to work. + +Context directory path is passed with "--workdir " command-line option. +On context creating, all passed parameters are stored in the context directory +and picked every time the context is used again. + +1. Repo for building Debian system +``` +debrepo --workdir=d12 --init locales gnupg +``` +Initialize the context in "d12" directory and create a repository in +"d12/repo/apt" directory sufficient to debootstrap default system +(e.g., debian-bookworm-amd64). Additionally, packages "locales" and "gnupg" +with all their dependencies will be available in this repo. + +``` +debrepo --workdir=d12 docbook-to-man +``` +Adds "docbook-to-man" packages with its dependencies to earlier created repo. + +``` +debrepo --workdir=d12 --srcmode docbook-to-man +``` +Downloads source package for "docbook-to-man" and adds it to the repo + +2. Repo for building Ubuntu system +``` +debrepo --init --workdir=uf \ +--distro=ubuntu --codename=focal --arch=arm64 \ +--aptsrcsfile=/work/isar/meta-isar/conf/distro/ubuntu-focal-ports.list \ +--repodir=repo/apt --repodbdir=repo/db \ +--mirror=http://ports.ubuntu.com/ubuntu-ports \ +locales gnupg +``` +Initialize the context in "uf" directory and create a repository in "repo/apt" +directory sufficient deboostraup ubuntu-focal arm64 system. Mirror to use and +source list are specified by corresponding arguments. Packages "locales" and +" gnupg" with the dependencies will be also placed to the repo. + +``` +debrepo --workdir=uf gnupg,locales +``` +Add "gnupg" and "locales" packages with their dependencies to earlier created +ubuntu repo. Other parameters (distro, codename, arch) are ommited since they +are picked from the context. + +3. Repo for cross-building Debian system +``` +debrepo --init --workdir=d11 --codename=bullseye--arch=amd64 --crossarch=armhf +``` +Initialize the context in "d11" directory sufficient to deboostrap Debian +Bullseye (amd64) with foreign "armhf" architecture support + +``` +debrepo --workdir=d11 gcc +``` +Add "gcc" package (amd64 version) to earlier created repo. + +``` +debrepo --workdir=d11 --crossbuild gcc +``` +Add "gcc" package (armhf version) to earlier created repo. +""" + +import os +import sys +import fcntl + +import argparse +import shutil +import subprocess +import pickle +import urllib.parse + +import apt_pkg +import apt.progress.base + + +REPREPRO_TIMEOUT = 1200 + + +class DebRepo(object): + class DebRepoCtx(object): + def __init__(self, workdir): + self.distro = "debian" + self.codename = "bullseye" + self.arch = "amd64" + self.mirror = "http://deb.debian.org/debian" + + self.repodir = f"{workdir}/repo/apt" + self.repodbdir = f"{workdir}/repo/db" + + self.crossarch = self.arch + self.compatarch = None + self.keydir = "/etc/apt/trusted.gpg.d" + + def __init__(self, args): + self.workdir = os.path.abspath(args.workdir) + self.ctx = self.DebRepoCtx(self.workdir) + + self.cache = None + self.depcache = None + self.sr = None + self.extrarepo = None + + self.ctx_load() + self.ctx_update(args) + self.ctx_save() + + print( + f"ctx workdir: {self.workdir}\n" + f" distro: {self.ctx.distro}\n" + f" codename: {self.ctx.codename}\n" + f" arch: {self.ctx.arch}\n" + f" mirror: {self.ctx.mirror}\n" + f" repodir: {self.ctx.repodir}\n" + f" repodbdir: {self.ctx.repodbdir}\n" + f" crossarch: {self.ctx.crossarch}\n" + f" compatarch: {self.ctx.compatarch}\n" + f" keydir: {self.ctx.keydir}" + ) + + if args.extrarepo: + self.extrarepo = os.path.abspath(args.extrarepo) + + def ctx_load(self): + ctxfile = f"{self.workdir}/debrepo.ctx" + + if os.path.isfile(ctxfile): + with open(ctxfile, 'rb') as f: + self.ctx = pickle.load(f) + + def ctx_save(self): + ctxfile = f"{self.workdir}/debrepo.ctx" + + with open(ctxfile, 'wb') as f: + pickle.dump(self.ctx, f) + + def ctx_update(self, args): + if args.distro: + self.ctx.distro = args.distro + if args.codename: + self.ctx.codename = args.codename + if args.arch: + self.ctx.arch = args.arch + if args.mirror: + self.ctx.mirror = args.mirror + + if args.repodir: + self.ctx.repodir = os.path.abspath(args.repodir) + if args.repodbdir: + self.ctx.repodbdir = os.path.abspath(args.repodbdir) + + if args.crossarch: + self.ctx.crossarch = args.crossarch + if args.compatarch: + self.ctx.compatarch = args.compatarch + if args.keydir: + self.ctx.keydir = args.keydir + + def create_rootfs(self, aptsrcsfile): + os.makedirs(f"{self.workdir}/var/lib/dpkg", exist_ok=True) + with open(f"{self.workdir}/var/lib/dpkg/status", "w"): + pass + + os.makedirs(f"{self.workdir}/etc/apt/sources.list.d", exist_ok=True) + + srcfile = f"{self.workdir}/etc/apt/sources.list.d/bootstrap.list" + if aptsrcsfile and os.path.exists(aptsrcsfile): + shutil.copy(aptsrcsfile, srcfile) + else: + with open(srcfile, "w") as f: + repo = f"{self.ctx.mirror} {self.ctx.codename} main" + f.write(f"deb {repo}\n") + f.write(f"deb-src {repo}\n") + + dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}" + os.makedirs(f"{self.workdir}/{dir_cache}/archives/partial", + exist_ok=True) + + os.makedirs(f"{self.workdir}/tmp", exist_ok=True) + + def create_repo_dist(self): + conf_dir = f"{self.ctx.repodir}/{self.ctx.distro}/conf" + os.makedirs(conf_dir, exist_ok=True) + if not os.path.exists(f"{conf_dir}/distributions"): + with open(f"{conf_dir}/distributions", "w") as f: + f.write(f"Codename: {self.ctx.codename}\n") + f.write( + "Architectures: " + "i386 armhf arm64 amd64 mipsel riscv64 source\n") + f.write("Components: main\n") + + def apt_config(self, init, crossbuild): + # Configure apt to work with empty directory + if not init and self.ctx.arch != self.ctx.crossarch: + apt_pkg.config["APT::Architectures::"] = self.ctx.crossarch + apt_pkg.config["APT::Architectures::"] = self.ctx.arch + + if not init and self.ctx.compatarch: + apt_pkg.config["APT::Architectures::"] = self.ctx.compatarch + + apt_pkg.config.set("APT::Architecture", self.ctx.arch) + + apt_pkg.config.set("Dir", self.workdir) + + dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}" + apt_pkg.config.set("Dir::Cache", f"{self.workdir}/{dir_cache}") + apt_pkg.config.set("Dir::State::status", + f"{self.workdir}/var/lib/dpkg/status") + + apt_pkg.config.set("APT::Install-Recommends", "0") + apt_pkg.config.set("APT::Install-Suggests", "0") + + # Use host keys for authentification + apt_pkg.config.set("Dir::Etc::TrustedParts", self.ctx.keydir) + + # Allow using repositories without keys + apt_pkg.config.set("Acquire::AllowInsecureRepositories", "1") + + def mark_essential(self): + for pkg in self.cache.packages: + if pkg.architecture == self.ctx.arch: + if pkg.essential: + self.depcache.mark_install(pkg) + + def mark_by_prio(self, priority): + for pkg in self.cache.packages: + if pkg.architecture == self.ctx.arch: + ver = self.depcache.get_candidate_ver(pkg) + if ver and ver.priority <= priority: + self.depcache.mark_install(pkg) + + def mark_pkg(self, name, crossbuild): + pkgname = name + + if pkgname and (pkgname not in self.cache): + # Try for cross arch + if (pkgname, self.ctx.crossarch) in self.cache: + pkgname += f":{self.ctx.crossarch}" + + if pkgname not in self.cache: + print(f"Error: package '{name}' not found") + return False + + pkg = self.cache[pkgname] + + if (not crossbuild) or (':' in pkgname) or (not pkg.has_versions): + if (pkg.has_provides) and (not pkg.has_versions): + print("pkgname is virtual package, selecting best provide") + # Select first provide + pkg_provide = pkg.provides_list[0][2] + # Find better provide with higher version + for provide in pkg.provides_list: + if apt_pkg.version_compare(provide[2].ver_str, + pkg_provide.ver_str) > 0: + pkg_provide = provide[2] + self.depcache.mark_install(pkg_provide.parent_pkg) + else: + self.depcache.mark_install(pkg) + else: + version = pkg.version_list[0] + if version.arch == "all": + self.depcache.mark_install(pkg) + else: + if version.multi_arch == version.MULTI_ARCH_FOREIGN: + if (pkgname, self.ctx.arch) in self.cache: + nativepkg = self.cache[pkgname, self.ctx.arch] + self.depcache.mark_install(nativepkg) + else: + return False + else: + if (pkgname, self.ctx.crossarch) in self.cache: + crosspkg = self.cache[pkgname, self.ctx.crossarch] + self.depcache.mark_install(crosspkg) + else: + return False + + return True + + def mark_list(self, pkglist, crossbuild): + ret = True + if pkglist: + for pkgname in pkglist: + ret = ret and self.mark_pkg(pkgname, crossbuild) + + return ret + + def handle_deb(self, item): + fd = open(f"{self.ctx.repodir}/repo.lock", 'w') + fcntl.flock(fd, fcntl.LOCK_EX) + subprocess.run([ + "reprepro", + "--dbdir", f"{self.ctx.repodbdir}/{self.ctx.distro}", + "--outdir", f"{self.ctx.repodir}/{self.ctx.distro}", + "--confdir", f"{self.ctx.repodir}/{self.ctx.distro}/conf", + "-C", "main", + "includedeb", + self.ctx.codename, + item.destfile + ], timeout=REPREPRO_TIMEOUT) + fd.close() + + def handle_repo(self, fetcher): + dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}" + fd = open(f"{self.workdir}/{dir_cache}.lock", "w") + fcntl.flock(fd, fcntl.LOCK_EX) + fetcher.run() + fd.close() + for item in fetcher.items: + if item.status == item.STAT_ERROR: + print("Some error ocured: '%s'" % item.error_text) + pass + else: + self.handle_deb(item) + + def get_filename(self, uri): + path = urllib.parse.urlparse(uri).path + unquoted_path = urllib.parse.unquote(path) + basename = os.path.basename(unquoted_path) + return basename + + def fetch_file(self, uri): + filename = self.get_filename(uri) + subprocess.run([ + "wget", + "-H", + "--timeout=30", + "--tries=3", + "-nv", + uri, + "-O", + f"{self.workdir}/tmp/{filename}" + ], + stdout=subprocess.PIPE) + + def handle_dsc(self, uri): + filename = self.get_filename(uri) + fd = open(f"{self.ctx.repodir}/repo.lock", 'w') + fcntl.flock(fd, fcntl.LOCK_EX) + subprocess.run([ + "reprepro", + "--dbdir", f"{self.ctx.repodbdir}/{self.ctx.distro}", + "--outdir", f"{self.ctx.repodir}/{self.ctx.distro}", + "--confdir", f"{self.ctx.repodir}/{self.ctx.distro}/conf", + "-C", "main", + "-S", "-", "-P" "source", + "--delete", + "includedsc", + self.ctx.codename, + os.path.realpath(f"{self.workdir}/tmp/{filename}") + ], timeout=REPREPRO_TIMEOUT) + fd.close() + + def handle_src_list(self, pkgs): + if pkgs: + fetched_files = [] + for pkg in pkgs: + pkgname = pkg + pkgver = "" + if '=' in pkg: + pkgname = pkg.split("=")[0] + pkgver = pkg.split("=")[1] + + self.sr.restart() + while self.sr.lookup(pkgname): + if pkgver and pkgver != self.sr.version: + continue + + for sr_file in self.sr.files: + print(self.sr.index.archive_uri(sr_file[2])) + filename = os.path.basename(sr_file.path) + if filename not in fetched_files: + self.fetch_file(self.sr.index.archive_uri(sr_file[2])) + fetched_files.append(filename) + + dsc_uri = self.sr.index.archive_uri(self.sr.files[0][2]) + self.handle_dsc(dsc_uri) + break + + def apt_run(self, init, srcmode, pkgs, dscfile, crossbuild): + apt_pkg.init() + + extrarepo_list = f"{self.workdir}/etc/apt/sources.list.d/extrarepo.list" + if self.extrarepo: + extrarepo_list = f"{self.workdir}/etc/apt/sources.list.d/extrarepo.list" + with open(extrarepo_list, "w") as f: + distdir=os.path.join(self.extrarepo, "dists") + if os.path.isdir(distdir): + for dist in os.listdir(distdir): + repodir = os.path.join(distdir,dist) + if os.path.isdir(repodir): + for repo in os.listdir(repodir): + if os.path.isdir(os.path.join(repodir, repo)): + f.write(f"deb file://{self.extrarepo} " + f"{dist} {repo}\n") + + sources = apt_pkg.SourceList() + sources.read_main_list() + + progress = apt.progress.text.AcquireProgress() + + self.cache = apt_pkg.Cache() + if init: + self.cache.update(progress, sources) + self.cache = apt_pkg.Cache() + + if self.extrarepo: + apt_pkg.config.set("Dir::Etc::SourceList", extrarepo_list) + apt_pkg.config.set("APT::Get::List-Cleanup", "0") + self.cache.update(progress, sources) + self.cache = apt_pkg.Cache() + os.remove(extrarepo_list) + + self.depcache = apt_pkg.DepCache(self.cache) + self.sr = apt_pkg.SourceRecords() + + ret = True + + if init: + self.mark_essential() + # 1(required), 2(important), 3(standard), 4(optional), 5(extra) + self.mark_by_prio(1) + + pkgs = list(filter(None, ','.join(pkgs).split(','))) + if srcmode: + self.handle_src_list(set(pkgs)) + else: + ret = self.mark_list(pkgs, crossbuild) + + if dscfile: + fobj = open(dscfile, "r") + + try: + tagfile = apt_pkg.TagFile(fobj) + while tagfile.step() == 1: + deps = tagfile.section.get("Build-Depends", "") + # Remove extra commas and spaces - apt_pkg.parse_src_depends + # doesnt like lines like ", device-tree-compiler" + deps = ', '.join( + [s.strip() for s in deps.split(',') if s.strip()] + ) + print(f"parsed deps: {deps}") + for item in apt_pkg.parse_src_depends(deps, False): + pkgname = item[0][0] + self.mark_pkg(pkgname, crossbuild) + + finally: + fobj.close() + + if not ret: + sys.exit("Some of requested packages not found") + + if init or not srcmode: + fetcher = apt_pkg.Acquire(progress) + pm = apt_pkg.PackageManager(self.depcache) + + recs = apt_pkg.PackageRecords(self.cache) + pm.get_archives(fetcher, sources, recs) + + self.handle_repo(fetcher) + + +def parse_arguments(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--init", + default=False, action="store_true", + help="initialize context in WORKDIR") + parser.add_argument( + "--workdir", + type=str, required=True, + help="work directory storing debrepo context") + parser.add_argument( + "--aptsrcsfile", + type=str, metavar="PATH", + help="sources.list file to use when init") + parser.add_argument( + "--srcmode", + default=False, action="store_true", + help="add source packages instead of debs") + parser.add_argument( + "--repodir", + type=str, metavar="REPO", + help="repository directory") + parser.add_argument( + "--repodbdir", + type=str, metavar="REPODB", + help="repository database directory") + parser.add_argument( + "--extrarepo", + type=str, metavar="REPO", + help="extra repository to consider") + parser.add_argument( + "--mirror", + type=str, + help="use custom distro mirror") + parser.add_argument( + "--distro", + type=str, + help="select distro to use") + parser.add_argument( + "--codename", + type=str, + help="distro codename") + parser.add_argument( + "--arch", + type=str, + help="distro arch") + parser.add_argument( + "--compatarch", + type=str, metavar="ARCH", + help="compat arch to use") + parser.add_argument( + "--crossarch", + type=str, metavar="ARCH", + help="cross-build arch") + parser.add_argument( + "--keydir", + type=str, + help="directory with distro keys") + parser.add_argument( + "--no-check-gpg", + default=False, action="store_true", + help="allow insecure repositories") + parser.add_argument( + "--dscfile", + type=str, metavar="PATH", + help="Debian source file to parse") + parser.add_argument( + "--crossbuild", + default=False, action="store_true", + help="add packages with cross arch") + + parser.add_argument( + "packages", + nargs='*', type=str, + help="space- or comma-separated list of packages to add") + + args = parser.parse_args() + + return args + + +def main(): + args = parse_arguments() + + if not (args.init or args.packages or args.dscfile): + sys.exit("Nothing to do") + + workdir = os.path.abspath(args.workdir) + os.makedirs(workdir, exist_ok=True) + + with open(f"{workdir}/debrepo.lock", "a") as file: + fcntl.flock(file.fileno(), fcntl.LOCK_EX) + + debrepo = DebRepo(args) + + if args.init: + debrepo.create_rootfs(args.aptsrcsfile) + debrepo.create_repo_dist() + + debrepo.apt_config(args.init, args.crossbuild) + debrepo.apt_run(args.init, args.srcmode, args.packages, + args.dscfile, args.crossbuild) + + #Unlock debrepo context + fcntl.flock(file.fileno(), fcntl.LOCK_UN) + + +if __name__ == "__main__": + main() From patchwork Thu Jul 25 15:07:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3721 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 25 Jul 2024 17:10:24 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f187.google.com (mail-lj1-f187.google.com [209.85.208.187]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46PFAN2h007134 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:24 +0200 Received: by mail-lj1-f187.google.com with SMTP id 38308e7fff4ca-2ef23b417bcsf2311311fa.0 for ; Thu, 25 Jul 2024 08:10:23 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721920218; cv=pass; d=google.com; s=arc-20160816; b=gO42crGyhzOoMEasGgS6WRsQqCwK4JC80wUgVm8A/o2LRmoVngGV1Z5NMc1uoDDo/g YajPLksWYrqpfzsX3FHQTBme5XWRueb+rj4JmSBNXGZXoKjBGDK72yjqaQBvL6usBs4/ q0IGQJj3VVam5ZvpwW46paF3tQ51A9GrUpJyvXKLKXwC69ocL2uPSULbClPVRpg4da++ iYDlw1MIgeWv0ZJLg8AGhI0GLeuaPUeu+XQe92kZ50xiVz7Kwc2re3+m4PuOMA4rbkdG rYv5Zu9hj8JcR73PgMewJKIBmTLwtR1UkdQfsdhyLA8IVRiQW4hsK8pYZ7vCze834BBf zISg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=H312RUkhXbQ+C2T2MXZ1Y+e/a+b8dgyoR9x6njq3gmk=; fh=tfl7+F9LPtN9zPSJy3jBf34+mUnttezUyS4nbB7ipM4=; b=oD2ka+if6I33MVjsSi7RP0TWqb3zJgaK81a+SHhCuVkp3qX/4XX3rDzP/4W+2iE1Cg LZRJCo51Ut+pI/UduWmLkzcPiY/iPFNCnE4tujNGiqhP6h94llVNXby+HQrrG+l3xxjg /DgERa2bufgAzN5e6tXXu12Sg7Ty7ZGY9su7Kluem4KJi2rMHtYURatvhL63Gz+Csnrh V7mm01qvQjgdJb6zGxATzPXHCkD/PbK0OFd1R3eQwd1h+5jwN0Ov41J9BdIeJN7FifS6 AnsfYnU9Qx1JNNiU7hWtqd4plGpoOcsfvz03IHeAz4+x7pO3kBn6XIOEdBVQQIF3QVrY LLKg==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721920218; x=1722525018; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=H312RUkhXbQ+C2T2MXZ1Y+e/a+b8dgyoR9x6njq3gmk=; b=oetHX2rqnzJMWo3iNBHweaHg4zii5ItaZQ61Ysy+diHy5IpCbIbogyo2VLm6VWHo2+ poyMPH+ewZAmYjtPAjZPhiQRuf0ytHxa4dgtDpVcywByNemhIkBnSqeRgCrrAzF1DPkm 3hwTz/zToT2xanb9jB1Sriwtcabtnq6hQDF/laxa1BzaFhy0elId5JX3GuvJGZmM4bUJ Vnwfp4jqMYSoLAJugi1WU9loc3/MDP5YCvMM08bmKtXqr+88Qq3xDSjGpp3tOxuEFlXX 71fUU6oiwECs85DaBlMcaCTbvxbXZVgpq9m7UGih0HwPUpgtQ5XIJYlpNVCMGXZEQqqk JlXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721920218; x=1722525018; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=H312RUkhXbQ+C2T2MXZ1Y+e/a+b8dgyoR9x6njq3gmk=; b=axJ7jDofZ/5CR9t4pMINY/Hgu5jPYS64RgDv08q5EKsgBNG46neBaoMsf7qjcwz+r9 4LWHHmU7xfgUuUQg9K8MLVpcA/x8iLHjbdOu/BCB6oNfC50veLolIeWSy5aBIQSFgpEv 1lH5Nnu2qPRadb9dybcbaowxUly2QPXIJizIQmLfu4bLsN0GRrJdM9xMJbmH4ik+GxcW 7q0WT2xXnol1/vCJvMtAvnG5qKu6dTs4aJemorZIX2rjJ3DK7uPN6RrZkIyzQxXsmMgi 1RJZGz3Spmsdr0fRFKwMDx69DzVZZFmH24t+uY20HFRcS8/ODewTW+E32FZR39sfux7p 5LiA== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCV/8XCroo6rVlwCc3YZn4zpTqvrIlwJe20rESxuCmjOqNdJ8LfKz9D1tVlUPa44yoH5NYH+5FF0ZLTjcefobwkEQJvi1Vs= X-Gm-Message-State: AOJu0Yx7hZJShE1gygocq3QZjiBk3sbHiuWk/xXQWkCvKchFr8FDWzRW bwNknBOqkWqCH6ePKvkovrGy4JQRgfXnKKjJkGmgtKsthaTzohN9 X-Google-Smtp-Source: AGHT+IEc4aVQAJ3kESGdudhwKWKnpyaruU+5PC4xLCxVhlnZhMuqADBYRx1fkZBzW+RrnuHYrjtsJA== X-Received: by 2002:a2e:8094:0:b0:2f0:25dc:1886 with SMTP id 38308e7fff4ca-2f039c4e89dmr22880161fa.10.1721920217418; Thu, 25 Jul 2024 08:10:17 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:b12:b0:2ee:de96:7d96 with SMTP id 38308e7fff4ca-2f03aa5d97els4383221fa.1.-pod-prod-01-eu; Thu, 25 Jul 2024 08:10:15 -0700 (PDT) X-Received: by 2002:a05:651c:108:b0:2ef:2a60:c1c1 with SMTP id 38308e7fff4ca-2f039ca273bmr22591061fa.21.1721920214928; Thu, 25 Jul 2024 08:10:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721920214; cv=none; d=google.com; s=arc-20160816; b=H9lnopjNCz6GnFlKDgddbBqK6dIyrFyYwZmQVYYaCbjaXHoIiCavjXz6lEnxWrsiZ7 rSrvLTH4kJPP4YMXWQIorHUAzR+6CvkyiesmQSf7HtYuZcrQIyFhub+xjCUsmgJTWBnh c7U7CZZ1qcpxzLrGKLWMLJcGjfZvCVxcpI574ZsLeJwtV4TRjTTDT6HlQkc+ZsTV5bo9 tB7yamTcml2kBADAFTU6Fvelra7nr7zFnUKEAA7sRtjzkFnh9vbwmWYC+DnAGGGmvN/r LX+VlAViMsvxVnUpX7EX9FG3sG9W+VtutWnPDraFNvjFf0BI/02R46Xln9eEnMx2WgBT xS1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=okoNhLFL9jjP8xknS1E0x96pTNm+QJ3hxTPfjLrM95U=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=X8VGI1BjgOWHQAIiHU2CzAArtas+6QRJtDgXNe0SlOGe2LIAfxAvzLvgxEFQHTxmJ4 Ne9HfPf/vNbkDgyHItOhL9RcTlmgILOSqIF3k/QTkf97fxZf/GbvgYqePiVfAD8/UBUu WHkBg7tn/JpUtQ1n0aF8XGA91IxblfDv5XdBW5kcJE90tHgWL0/BnBWQqVOmAyhi7fAm N3+/os41VCizb3ShMedfV43yW+djQeuN5iouxBMTQPZHqtAOZ7QQSK8yTZbFjtxnn6Qj 0uaf1/yf1NSU7zH145E74Bny+aT5GDQ+Lx8xRyl746yWfva73WfEUEKMcrTGnNdSw2M9 kiYQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 4fb4d7f45d1cf-5ac6673bbcbsi45941a12.5.2024.07.25.08.10.14 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 25 Jul 2024 08:10:14 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 46PFACKY007063 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:13 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v7 02/10] meta: Add debrepo bbclass handling base-apt prefetching Date: Thu, 25 Jul 2024 18:07:34 +0300 Message-ID: <20240725151006.2129-3-ubely@ilbers.de> X-Mailer: git-send-email 2.44.2 In-Reply-To: <20240725151006.2129-1-ubely@ilbers.de> References: <20240725151006.2129-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This class uses 'scripts/debrepo' python script to prefetch given packages or sources to local base-apt repository. Signed-off-by: Uladzimir Bely --- RECIPE-API-CHANGELOG.md | 10 ++++ meta/classes/debrepo.bbclass | 90 ++++++++++++++++++++++++++++++++++++ meta/conf/bitbake.conf | 5 ++ 3 files changed, 105 insertions(+) create mode 100644 meta/classes/debrepo.bbclass diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index 12ea93ec..68204a96 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -629,3 +629,13 @@ into kernel kbuild package. Only the "host" specific package is built automatically at cross builds. * Support emulated module build with cross-compiled kernel for linux-module + +### "Prefetch" mode for base-apt + +Originally, `base-apt` repo is created only during second build when variable +ISAR_USE_CACHED_BASE_REPO is set. The repo is populated with every package that +took part in the first build and was cached in DL_DIR. + +New ISAR_PREFETCH_BASE_APT variable changes the way `base-apt` is populated. +Packages added to the repo before running any task that need them. Separate +`debrepo` script is used for populating base-apt repo. diff --git a/meta/classes/debrepo.bbclass b/meta/classes/debrepo.bbclass new file mode 100644 index 00000000..003c4e03 --- /dev/null +++ b/meta/classes/debrepo.bbclass @@ -0,0 +1,90 @@ +# This software is a part of Isar. +# Copyright (C) 2024 ilbers GmbH +# +# SPDX-License-Identifier: MIT + +# Prefetch to base-apt repo by default +ISAR_PREFETCH_BASE_APT ??= "1" + +DEBREPO_WORKDIR ??= "${DEBREPO_TARGET_DIR}" + +debrepo_update_apt_source_list() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + + chroot_dir=${1} + apt_list=${2} + + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + sudo -E chroot ${chroot_dir} /usr/bin/apt-get update \ + -o Dir::Etc::SourceList=\"sources.list.d/${apt_list}.list\" \ + -o Dir::Etc::SourceParts=\"-\" \ + -o APT::Get::List-Cleanup=\"0\" + " +} + +debrepo_add_packages() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return + + args="" + if [ "${1}" = "--srcmode" ]; then + args="${args} --srcmode" + shift + fi + + if [ "${1}" = "--isarapt" ]; then + args="${args} --extrarepo=${REPO_ISAR_DIR}/${DISTRO}" + shift + fi + + workdir="${1}" + args="${args} ${2}" + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + else + export GNUPGHOME="${WORKDIR}/gpghome" + fi + + ${SCRIPTSDIR}/debrepo \ + --workdir="${workdir}" \ + ${args} +} + +debrepo_parse_dscfile() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return + + dscfile="${1}" + args="" + + build_arch=${DISTRO_ARCH} + if [ "${ISAR_CROSS_COMPILE}" = "1" ]; then + build_arch=${HOST_ARCH} + fi + if [ "${PACKAGE_ARCH}" != "${build_arch}" ]; then + args="--crossbuild \ + crossbuild-essential-${PACKAGE_ARCH}:${build_arch} \ + dose-distcheck:${build_arch} \ + libc-dev:${PACKAGE_ARCH} \ + libstdc++-dev:${PACKAGE_ARCH} \ + " + fi + + args="${args} --extrarepo=${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO}" + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + else + export GNUPGHOME="${WORKDIR}/gpghome" + fi + + if [ -n "${DEB_BUILD_PROFILES}" ]; then + export DEB_BUILD_PROFILES="${DEB_BUILD_PROFILES}" + fi + + ${SCRIPTSDIR}/debrepo \ + --workdir="${DEBREPO_WORKDIR}" \ + --dscfile="${dscfile}" \ + ${args} +} diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 4cfa8b10..b0e33477 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -69,6 +69,11 @@ KERNEL_FILE:mipsel ?= "vmlinux" KERNEL_FILE:riscv64 ?= "vmlinux" KERNEL_FILE:arm64 ?= "vmlinux" +# debrepo config +DEBREPO_DIR = "${TOPDIR}/debrepo" +DEBREPO_HOST_DIR = "${DEBREPO_DIR}/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}" +DEBREPO_TARGET_DIR = "${DEBREPO_DIR}/${DISTRO}-${DISTRO_ARCH}" + MACHINEOVERRIDES ?= "${MACHINE}" DISTROOVERRIDES ?= "${DISTRO}" OVERRIDES = "${PACKAGE_ARCH}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${BASE_DISTRO_CODENAME}:forcevariable" From patchwork Thu Jul 25 15:07:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3720 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 25 Jul 2024 17:10:24 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-ej1-f55.google.com (mail-ej1-f55.google.com [209.85.218.55]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46PFAN5i007126 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:23 +0200 Received: by mail-ej1-f55.google.com with SMTP id a640c23a62f3a-a7a9447efcbsf30829666b.2 for ; Thu, 25 Jul 2024 08:10:23 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721920218; cv=pass; d=google.com; s=arc-20160816; b=yjvAOhkgOfQnAqJVteU4gzxhlE25qjcbYnebWDBiVzgSkZU/cYOU4HeqOgj6MeHjXw EY91ywco1HaL5a4PIBHpt0C/LHmevhqpXQGeskFrOarPLCrHn+i9776sV1r3I1JVwPmM VVkVr61jYognkcaBxspcmizc0Qhr6VZ3nHdeafIoPylGG1i/wyRoxQjF5kvNDaPNQcuc KoqUGJ7mFHRwMTjVS6UGE33Ro82qTU/1zo0gAjCDOIGlOqUS6YlTut0VqB6rCpXhFY9n 6dP9mr14NTCrtL4tmPs1hH/Hmz6E53dnuog+YgoFxk926i+6uwviDueaHfj2HeXX+9N2 HfWw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=0xvA1hW5Z9/ytIklwpf32WE8Ug1PqI+qp2j1envBGxs=; fh=IsXqzYkYZd+WIIlHjwvwjZloPEhXyTR45WzH4DKe7S0=; b=B4Y22rUNwTpX4PUGqlgjzCwH9J76BXDepVEDjoUnrbuY4/CRarqzWKJkiJKb11aE5V X+HoeFwgoEJpr5lxkW7Rv/BA6etWvnSwO14W0Fq3SL9bur1KCIb5b1peIsXXitv6Ok/n cqji/s9LU4r8bg/6O0ws7pHhrztkhyUPwcdem0o95cqRHFMQX3w+jEbSfuF9TYEYPXHF pmJhDM9/pjDpPDHRiIBhqpNxpwD5vJCIfeH6BOOxcpQO5WpcGctLV1CXbACUr60w01h1 JsLlLv4mEdqKHUPBSrYX7NFoR0uCWiyOe9EUcNnI974GOuUYGjcyTSPx7CDkvbH9kLJm AzLA==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721920218; x=1722525018; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=0xvA1hW5Z9/ytIklwpf32WE8Ug1PqI+qp2j1envBGxs=; b=YhyFZLMDdEodHpEglax8fgD5/O1vBAFMWIk3Ee0U39Ju1MRLoRJQeP1OjGHXUiZl6i 2KJHL2A1voUqU34aU6X6vNDMA1UEBlflNUMsoX/NKA9zour/GQ3nK0SP0aq3W2CV7K/I ZaRrmaDnE/CwQpb6LvspEwiakqVaSz5u3hr4S6IiNOMcDh5A77h6jF4cOyU1bk8gW1MF 8fmmZOBX+iB81nto6oVMOospREVArjrVP19v5t7c9Im/mGAbKmfy1tULnT7e2etJ2lrd o74B3mX3PyTHw9jkGwCHYCs1t2utjulzO7A98SycHiG1/0rUlOZqEc2oOMtWBjt87aVC cPTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721920218; x=1722525018; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=0xvA1hW5Z9/ytIklwpf32WE8Ug1PqI+qp2j1envBGxs=; b=b04N3Le4IQ3MqjE4IPJH+Ct7cTbD2B2DukNZchA+6WVKQ5fGWaL9p9P5Ubw8N+YoeD YOh3RsWOOk2Ps2dj3CxGsiUgzjR4UnSe5W7w1uRf6noEVi5Ie2/NJx0jtWXV1Ujhsqse gzsbicDcbRgsqrLl0Q66fWjUESrJ3k93WikSHLd5rG54sfMylBrNY0F9kOZmJcmcwGEn lEWCezncxuaD/mmZDTnaHPPI4hsLCU2G0slEp8CtxANpigsmFepzIa8UaVMzOMKxY0i7 Ne+ywP2G0dGa9wERWGz6K20fWgZS6VPDBvm9SwytF4c4307x0abk/dOqW8xYFWzavAgL coeQ== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCUOE3DUNMapo2HXJ/SZcj0vgJL9+h3QsLMLYxn4ASWYeDyAHNCDhDutMAAbn4ifT2ghpsh4qyHdubAOol3fLYkxGgJW9qc= X-Gm-Message-State: AOJu0YwsOVqe5sbOVetzrnHxWUWUn3SZgwg7CUPMGM8opGsXD8ZmuY9a iBaF7j2GQF5Ew8Reqzl256kM8wM6utV/kyneIGofLs+aA4DvzuPo X-Google-Smtp-Source: AGHT+IGpEpMtcBwhybEtmJnaAgNi7Wr8D0TwHz3zXA4VpZbRFkkOmg7gPQW49BRtR+QKpbkr6r3SYg== X-Received: by 2002:a50:cc98:0:b0:58b:f46f:ae45 with SMTP id 4fb4d7f45d1cf-5ac6396b221mr2302523a12.23.1721920217610; Thu, 25 Jul 2024 08:10:17 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6402:f11:b0:57c:a472:931c with SMTP id 4fb4d7f45d1cf-5ac08d42d84ls220855a12.0.-pod-prod-05-eu; Thu, 25 Jul 2024 08:10:15 -0700 (PDT) X-Received: by 2002:a50:9519:0:b0:5a7:464a:abf with SMTP id 4fb4d7f45d1cf-5ac6396b2a5mr2384555a12.24.1721920215341; Thu, 25 Jul 2024 08:10:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721920215; cv=none; d=google.com; s=arc-20160816; b=SdTFjTsxEWemJyjaPoAoUhugpVADpNjfGWJWZblt+omIPPEMCj85DwvI9MymuubRtI NAOk6qWxhthhneg9mAhcnhjGE3lmkj2Zup9tZ9MyJ7d24ynsX+EDm52VGbpGNvL6kOHx VVdBDYA8vkeDoQvnhVidSZqChPngL0Q8L12VUiook2R6OnMatJUXK0aFSPhbXgfIoc+G CqQTrJy5U2JQTIIZS5KPZyXspYp9RIZuYTRKtO9Z4Ax+cP/idozm+ODc13mLdbc7Uzri DNBpT9WIe/gGjN6RDfHGTsAGHDybfeBjkvjClEMOtk/ZCcFTwBUeJyFXXaqPOCcLOKKC V25Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=n60x1HTgMm/pWI8SeNW/SFcuiWorBb8UPgyLDUIkuAc=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=UZqXKgJ4TIB77119Pb4oGXeEcYhF5aQYIuiB5HPtkzqv7NSyBENgCBVT8EuY7sLYPo LQhlpyZn8ea1VPKwJgs8M7QjguKRXD+djWzBbQ66lbjdHZrZS6CRgR3MXDVEj1C5RAIC xpGDAIqvS/2NWD4NOLM+sHVbAweesGOrZMjiM0fhlnxLRYVGjndLo1Y7SwN9rdrzRo36 q8VK83xFNrPSpbC/RO/Rhf35T3PtH10S6IQqVZ9NQKKS9ftRCxVIVZuZ+bkAkC6O+4sd mj0LcfhvvBREkoepNJAbB4RmJCFzE3kNineNma/TwjBsPSgFM2+kSXgHCkKRhXFmNpyd aROw==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 4fb4d7f45d1cf-5ac60cf50bbsi44335a12.0.2024.07.25.08.10.15 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 25 Jul 2024 08:10:15 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 46PFACKZ007063 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:14 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v7 03/10] meta: Always use base-apt repo in local mode Date: Thu, 25 Jul 2024 18:07:35 +0300 Message-ID: <20240725151006.2129-4-ubely@ilbers.de> X-Mailer: git-send-email 2.44.2 In-Reply-To: <20240725151006.2129-1-ubely@ilbers.de> References: <20240725151006.2129-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This means only local URLs in apt sources.list* are present during the build. Any installation of packages is done from local base-apt. So, base-apt should be always mounted in *_do_mounts since now. Signed-off-by: Uladzimir Bely --- meta/classes/rootfs.bbclass | 5 +++-- .../isar-bootstrap/isar-bootstrap.inc | 12 +++++++---- meta/recipes-devtools/base-apt/base-apt.bb | 21 ++++++++++++------- 3 files changed, 25 insertions(+), 13 deletions(-) diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 2e091e0c..7d0bc0c8 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -54,8 +54,9 @@ rootfs_do_mounts() { mount -o bind,private '${REPO_ISAR_DIR}/${DISTRO}' '${ROOTFSDIR}/isar-apt' fi - # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set - if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ] + # Mount base-apt if 'ISAR_PREFETCH_BASE_APT' or 'ISAR_USE_CACHED_BASE_REPO' is set + if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')))}" = 'True' ] || \ + [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ] then mkdir -p '${ROOTFSDIR}/base-apt' mountpoint -q '${ROOTFSDIR}/base-apt' || \ diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 12f32ff0..9fd928ad 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -47,7 +47,8 @@ python () { # installation afterwards. However, debootstrap will include the key into # the rootfs automatically thus the right place is distro_bootstrap_keys. - if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): + if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \ + bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) : own_pub_key = d.getVar("BASE_REPO_KEY") if own_pub_key: distro_bootstrap_keys += own_pub_key.split() @@ -121,7 +122,8 @@ def get_apt_source_mirror(d, aptsources_entry_list): # this is executed during parsing. No error checking possible use_snapshot = bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) snapshot_mirror = d.getVar('DISTRO_APT_SNAPSHOT_PREMIRROR') - if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): + if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \ + bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) : premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" elif use_snapshot and snapshot_mirror: premirrors = snapshot_mirror @@ -299,7 +301,8 @@ do_bootstrap() { if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}" fi - if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" -a -z "${BASE_REPO_KEY}" ] || \ + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then debootstrap_args="$debootstrap_args --no-check-gpg" fi E="${@ isar_export_proxies(d)}" @@ -326,7 +329,8 @@ do_bootstrap() { install -v -m644 "${APTPREFS}" \ "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap" mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" - if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] || \ + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main" if [ -z "${BASE_REPO_KEY}" ]; then line="[trusted=yes] ${line}" diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb index 2766bc71..09a4509c 100644 --- a/meta/recipes-devtools/base-apt/base-apt.bb +++ b/meta/recipes-devtools/base-apt/base-apt.bb @@ -56,9 +56,12 @@ repo() { "${BASE_DISTRO_CODENAME}" \ "${WORKDIR}/distributions.in" \ "${KEYFILES}" - populate_base_apt "${BASE_DISTRO}" - repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" + if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \ + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then + populate_base_apt "${BASE_DISTRO}" + repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ + "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" + fi if [ '${BASE_DISTRO}' != '${HOST_BASE_DISTRO}' ]; then repo_create "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \ @@ -66,14 +69,18 @@ repo() { "${BASE_DISTRO_CODENAME}" \ "${WORKDIR}/distributions.in" \ "${KEYFILES}" - populate_base_apt "${HOST_BASE_DISTRO}" - repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \ - "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}" + if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \ + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then + populate_base_apt "${HOST_BASE_DISTRO}" + repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \ + "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}" + fi fi } python do_cache() { - if not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): + if not bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) and \ + not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): return 0 for key in d.getVar('BASE_REPO_KEY').split(): From patchwork Thu Jul 25 15:07:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3726 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 25 Jul 2024 17:10:27 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f57.google.com (mail-lf1-f57.google.com [209.85.167.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46PFAQmf007170 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:26 +0200 Received: by mail-lf1-f57.google.com with SMTP id 2adb3069b0e04-52ebdbf8a7csf205700e87.2 for ; Thu, 25 Jul 2024 08:10:26 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721920218; cv=pass; d=google.com; s=arc-20160816; b=Eo3EjXQwdC9zKrX2vXZXEBD4qgRu2WbCG2zwp54YU/O5lh7sITYMtdAbY7gnnzIY/M W1wqRMp9MWbJs1iDIB/FM165ftv4kMQUgvjYhxwieaWE86d5xSmotlalZvkcmNrce8e9 58lBhc/ZcxxFnlm1WKGaYRJJ0OvVKe7aqWNjQOGkbNXuTa9wbOjRd7zemw2+4WhzcSVt 1GKj04tPpyZ6+Bfmrr2FDsnxW4fhAtIAm6nA3miWIl5GDpTYzkjVeldIcflwELQw51Lr twF6EVoyjL+eZBMBFjPjbUiSNVrXEOAHPEn2QUFNrjCjBlmqGgPqDUwSq2g8PnRdfvOM Q1wQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=kjHb1SsEmaAkhbpS61GtNV9Z8+2/XkiOwwIvE8/peOE=; fh=i+pZP045BfjZjxy06VCz4ovKBxJHkPlJyRtbRU2muWo=; b=QpR0yrHuGVsqMXmhH53PEGShv7fgjKgJAfNpZUMH7GUtI0sCdLuahO4z3K7Jw5Dfy+ 7GlpXos/YkE6elAcV6dRfpUcl6XVBB4n60QtKXAL2fm+5g+YdtVdwxRLLn6UOYZLZCoY CeTC7Av7dIB+XYpNi97tI6ryl51+TWUX2qN1ESlH+MjZbmHyOjXyAF9F8njHM03gXoRP 0W9qnJFye5vCWD4jL/sdATMqlRf0RmrPBDMryICKI5FjPXRyLG1rb0SIvUfBMgnT8tIY wS3aU5m+lTB3M1L1OlQWBq56MbZmx9S3FWJ7A3SKt/Wt5mxwH5rkCwzS+7Ft28QkhD+w eIFg==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721920218; x=1722525018; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=kjHb1SsEmaAkhbpS61GtNV9Z8+2/XkiOwwIvE8/peOE=; b=nRuoIdOKa+UNNpTxKgbJTK23QngKGbbswWP4EwY5Ji8Bgd6XCyTljwF6bpkaZv9rdS T4OcexYFm2NBaZFlS+IpuvR9YJRdHkomLlTxxxqDND59sY0X6KP2MnPQQay8A8TTmoj2 2E8p7N4WVJgYLCKQwQ4epIcLSGQr2PK23nCtQ1Xd6eCQRWUyNzGlRtP73q/g4+1XCsn5 BMe7i9QfxH3We/e8qhaojAR2o7jFQ0VAPy8SkIUc/9zkrOUbYBiwOzHexztvuqVh71cK +JEwb/Y03dXCkw9T8MalyHxFGSf4ChCUUzNfhQYWnBiWDurtYd+eV1lWCCdpmIv4/4Qb s3eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721920218; x=1722525018; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=kjHb1SsEmaAkhbpS61GtNV9Z8+2/XkiOwwIvE8/peOE=; b=AQ1bJIs92P9rAAUJDGDhraVyQ4PyGWfcfz/G2RPF6uZvWXKdlQ9u323YyAZrW7swRh SZ1zCB45N1xFrW8mhM4paM0EgMqKyggWZuawAOXeCjzzRyE3vp1WBLjLOwe3axHdDbSf 8O3cD89XmjaMWZExapPNniXbHsmsReu9HRtRDwrT40Meph/5LABS7K+rUvdT0//WvS7p EhGtJaUP64I6kVgwNlh4paJZ6eLgbZ9VkuPAafMpXonHW5V+3GaHZTQUZg+paAqQPBOH OkRpIx66FspuF0G170cK+fDLfEIRaYluFz45MHD2h4qIFKxBng1z6Vmop1anrKZveaBI bYXg== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCVKaRJHXfvF+dBY1i0Fru4t6FyyPJ8jRsWDkT7640BYpaoK+3h6QzkLPk51YERm3P4LXvKN8UMCXXdDrSslpAiyMkKhVok= X-Gm-Message-State: AOJu0YxFSiMfLjrJo8tRSCil8AcGZEdRWfsZX4XfSNMhAjhLez4YBa4s GJOQTOKFxZLLH9Qv8yNAxsN6DGO2EYjXt1FqT+tBkpTxReo+fSta X-Google-Smtp-Source: AGHT+IFt0vahHzvqy11p890omF8yMvl80se2xu3D5s34krsLH0cJKLdquK1OazL4mS+h/guTsRn+4Q== X-Received: by 2002:ac2:568f:0:b0:52d:6663:5cbe with SMTP id 2adb3069b0e04-52fd60189b8mr1814488e87.12.1721920217809; Thu, 25 Jul 2024 08:10:17 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:4e03:0:b0:52e:9d2f:234 with SMTP id 2adb3069b0e04-52fd3f54566ls614854e87.0.-pod-prod-06-eu; Thu, 25 Jul 2024 08:10:15 -0700 (PDT) X-Received: by 2002:ac2:4f07:0:b0:52e:7448:e137 with SMTP id 2adb3069b0e04-52fd60189acmr1814385e87.6.1721920215460; Thu, 25 Jul 2024 08:10:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721920215; cv=none; d=google.com; s=arc-20160816; b=fofXg1/gQh3Qbg8jn3ViQ5A3z5ukR3bfMQiCd1y6CKDP8MxuCKZzJVncsyX+xzCQ5p 6I7X1Uy6K/oC1RV5pz7LoF6VwsiWK3LXpEAR5jSzCaBk8oDtNL+g6NEXhpkpU/8jeKZ4 mWmWSJ0zyV47xebddNEor5akYHl+ftiUVNTCHYsTBGfhTXvLL06fcYf6ZPFrLEpQNVZC ffwm4c/m1aqk8+yrlif+ku3XXOG6JvSxonO+x69+3bPVQ0EiKsdadKR7KTBL5+a4i7N9 FDIv/HQ0f8vMlujU2rar3I/7IrSTTFMiUQew1uyQ0oijEw41k1s9efzzpTCr5L1alNhy Iutw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=KKhUWQuFn3XJxtbskh8Qfp/nrLPLpvZsTNdE5cSLSWk=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=xlh2qZ3W0YU52VBoOqkttNqwXl3ACI2UjkhAPOk6Q1378j2kcW5SKowEzdnH7WZFHU I/KGS1tljj+XTf7vJ7FSGvE23vavcgxeqqFhOJYYYFqkeZ1ZOlUZrNg+BwvfOyLLEozk 0E9cNDH+VaczDa/eyC/Va7GXIODGbSBV+Pf8Z+9sFEfEXpWXq5ekoCeargt9rPk4xDWG K2LdZHAe4rX3OTLu38ePRLPUBznuPl/qKDxPiLk8TtVgbI/Gh2PkuMegQ9bYlUcLrBGD A77bfhc+JyFSL83FA22cam5gODEuAtlkAP35gtfGnTicnk+SyPu+4dUvYaUgPSwFQY/Q 9v6g==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 2adb3069b0e04-52fd5b8b998si31212e87.2.2024.07.25.08.10.15 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 25 Jul 2024 08:10:15 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 46PFACKa007063 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:14 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v7 04/10] meta: Use cached base-apt repo to debootstrap Date: Thu, 25 Jul 2024 18:07:36 +0300 Message-ID: <20240725151006.2129-5-ubely@ilbers.de> X-Mailer: git-send-email 2.44.2 In-Reply-To: <20240725151006.2129-1-ubely@ilbers.de> References: <20240725151006.2129-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch makes local base-apt repo to be created before debootstrap task. So, debootstrap is then done from it. The required packages are downloaded via python-apt and reprepro creates debian-like repository from .deb files. For debian targets host keyring is used while ubuntu/raspbian targets use keys specified by DISTRO_BOOTSTRAP_KEYS variable. The goal is have workable base-apt repo before first build completed. Signed-off-by: Uladzimir Bely --- .../isar-bootstrap/isar-bootstrap-host.bb | 2 + .../isar-bootstrap/isar-bootstrap.inc | 87 +++++++++++++++++-- 2 files changed, 81 insertions(+), 8 deletions(-) diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb index 4f90fd01..1ace818f 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb @@ -15,3 +15,5 @@ require isar-bootstrap.inc HOST_DISTRO_BOOTSTRAP_KEYS ?= "" DISTRO_BOOTSTRAP_KEYS = "${HOST_DISTRO_BOOTSTRAP_KEYS}" + +DEBREPO_WORKDIR = "${DEBREPO_HOST_DIR}" diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 9fd928ad..589fa2c1 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -33,6 +33,13 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates" DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else ''}" BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}" BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}" +BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO_ARCH')}" + +# For newer distros "usr-is-merged" indirectly required by debootstrap +DISTRO_BOOTSTRAP_BASE_PACKAGES:append:bookworm = ",usr-is-merged" +DISTRO_BOOTSTRAP_BASE_PACKAGES:append:sid = ",usr-is-merged" +DISTRO_BOOTSTRAP_BASE_PACKAGES:append:sid-ports = ",usr-is-merged" + FILESEXTRAPATHS:append = ":${BBPATH}" APT_SNAPSHOT_DATE = "${@ get_apt_snapshot_date(d)}" @@ -291,12 +298,56 @@ do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}" inherit compat +inherit debrepo + +debrepo_bootstrap_prepare() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return + + debrepo_args="" + if [ "${BASE_DISTRO}" != "debian" ]; then + if [ "${BASE_DISTRO}" != "raspbian" ] && [ "${BASE_DISTRO}" != "raspios" ] || [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then + debrepo_args="$debrepo_args --keydir=${WORKDIR}" + fi + else + if [ "${BASE_DISTRO_CODENAME}" = "sid" ]; then + debrepo_args="$debrepo_args --keydir=${WORKDIR}" + fi + fi + if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then + debrepo_args="$debrepo_args --compatarch=${COMPAT_DISTRO_ARCH}" + fi + + if [ "${BOOTSTRAP_FOR_HOST}" = "1" ]; then + debrepo_args="$debrepo_args --crossarch=${DISTRO_ARCH}" + fi + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + fi + + ${SCRIPTSDIR}/debrepo --init \ + --workdir="${DEBREPO_WORKDIR}" \ + --aptsrcsfile="${APTSRCS_INIT}" \ + --repodir="${REPO_BASE_DIR}" \ + --repodbdir="${REPO_BASE_DB_DIR}" \ + --mirror="${@get_distro_source(d)}" \ + --arch="${BOOTSTRAP_DISTRO_ARCH}" \ + --distro="${BOOTSTRAP_BASE_DISTRO}" \ + --codename="${BASE_DISTRO_CODENAME}" \ + ${debrepo_args} \ + ${DISTRO_BOOTSTRAP_BASE_PACKAGES} +} + do_bootstrap() { if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then if [ -z "${COMPAT_DISTRO_ARCH}" ]; then bbfatal "${DISTRO_ARCH} does not have a compat arch" fi fi + + debrepo_bootstrap_prepare + debootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}" if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}" @@ -314,8 +365,19 @@ do_bootstrap() { sudo -E -s <<'EOSUDO' set -e if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then - arch_param="--arch=${DISTRO_ARCH}" + arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH}" fi + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + ${DEBOOTSTRAP} $debootstrap_args \ + $arch_param \ + ${@get_distro_components_argument(d)} \ + ${@get_distro_suite(d)} \ + ${ROOTFSDIR} \ + file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO} \ + ${DISTRO_DEBOOTSTRAP_SCRIPT} + " + else ${DEBOOTSTRAP} $debootstrap_args \ $arch_param \ ${@get_distro_components_argument(d)} \ @@ -323,7 +385,7 @@ do_bootstrap() { "${ROOTFSDIR}" \ "${@get_distro_source(d)}" \ ${DISTRO_DEBOOTSTRAP_SCRIPT} - + fi # Install apt config mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d" install -v -m644 "${APTPREFS}" \ @@ -395,12 +457,21 @@ do_bootstrap() { if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH} fi - - chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \ - -o APT::Update::Error-Mode=any - chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f - chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ - -o Debug::pkgProblemResolver=yes + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \ + -o APT::Update::Error-Mode=any + chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f + chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \ + -o Debug::pkgProblemResolver=yes + " + else + chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \ + -o APT::Update::Error-Mode=any + chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f + chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \ + -o Debug::pkgProblemResolver=yes + fi umount "${ROOTFSDIR}/dev/shm" umount "${ROOTFSDIR}/dev/pts" From patchwork Thu Jul 25 15:07:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3722 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 25 Jul 2024 17:10:25 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f57.google.com (mail-wm1-f57.google.com [209.85.128.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46PFAOwH007138 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:24 +0200 Received: by mail-wm1-f57.google.com with SMTP id 5b1f17b1804b1-427df7c3a2asf7680635e9.1 for ; Thu, 25 Jul 2024 08:10:24 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721920218; cv=pass; d=google.com; s=arc-20160816; b=Ma2qh70vN8jSxcWsaKCktelPOyIaHT/TA7dsTrBhq0IKgXmoIyFyShEimBUYdkTUSV rKWm1gQQPBm1t4c36hy+qm3BjCQz8QCw8UXiiSCazBcXTp9BoS4Kd4g3r0x05jjpfhJg 2/QB6v6TYEm/FPJ1LCE4tagswlBhgLMG0G0knVGPzu4HFfs24O/uBsIob/MIOoobnx87 spHRIKE8VpPo7AVbAfpFfj10bkbN0jNRiN2khXsRCWVqWilD7zUyWomqRLMBK9w9rWEX RezepqSve/je0afT0HBvguJ/KXbXtQsQpq5i6NRYaL7bo0aT6QNhXa5/8e1WAUWr+X56 rvjw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=r/xxjf4FajxD4+sAk0cVvTezHgBf0oz62I8Eq5Q/9Ug=; fh=jpxsTVOPPlwlNSS8DWqlj9cO5dEp8dT98NgooChUZ5k=; b=TbadNYkUTDMSuQAVIMTaK5uZJkhxVHhze5X46RT2sidvOSBRzB4+AXZgv/jin7f4FL kayAlBnpWvravrEtx/obOWIpHoDIgN209LjFMuEhqEamFGJ33YTH1OHSm44Mdi1yKr+3 /HjFrH0cRDnRNXuU9WlJr6Ctb30Q3cbCCX0ZQc9y/dDYphuUkDsjKzayZ1WhE/EiHY7H uMRMnpKYQn78LsY0VHOy9N3zHerWSd6Bpdl8WK7FOoafxCCAqdSR6GF3a89DG/pROUxA oTmkLsFwkkmZT4oj9ErAUYO9FyMGDQOEA+pL89DLsgwifDZmVWosCGsF/utk/XX5DwfN No1w==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721920218; x=1722525018; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=r/xxjf4FajxD4+sAk0cVvTezHgBf0oz62I8Eq5Q/9Ug=; b=cjD3rrmqlkbMd/sztusIV+Ca6ractwaKxj4YoxI8JD6szgLYoCbkgAFZPA1I8Dcb5X 0/CUKGt3mEVTC3XDHT/XZ4oyPaO93C+r9E+/7DWf4j6kY//ysHsu/NNlpYmg9Hx8QwxP XgAd+VIRsmP0pYcMDAv5NGTX3McbiQ670581Alj6vNNtlPBUXmf3PVDn+I3y6hv/WZmr u+nttc7JB32ea1zECDnqkxlrpTCX0VlBESo8035bbQsMfV9+PM52z87iV4kkhuq0YCph 9wVa99XWZXIPLRNPpAq6cpLFF0+METELwWSr+RHMsrJChGFZBrqFY90LHh3i9QV6/H7q MLgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721920218; x=1722525018; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=r/xxjf4FajxD4+sAk0cVvTezHgBf0oz62I8Eq5Q/9Ug=; b=YD4kQocqnzd+c9dN4Ea//SLS7V6+U0USVztWXr1Cfpoza5j+UgCji9xSNUO0NHv1K5 RmdAa2Nhy7y0gi2SJm71hRe07w1LBSKKmPf8J7/6jVrmAMtL4Z/uHGU4vQ5cHjkrWqV1 K9EpD0jbMzf8LM+xn25ejLnInks2CS2ULFCaVxO07OGsXiUbYemEaNHk8K+UtKuZp3mo f0Sw4XbUgFmFEdMOa7d7wnUKFKa2se7Nzd4900xoSPKu6rRYXSCINac83UaNEKoihcOa bW2r6TZVHic4Wtn5arr+OhDS7P2mupPn5gIbIEHgZenZ8ajnXRPZfVUnJVJTnsik58r1 qysw== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCXjsDY2li26b4j54FhRhnar+rp6bYKnCMJc8Ig6UUUrbCyXdALHrObObTOCc4DoAIernFiS8zJ79jcNkGJna0NHvK67XpI= X-Gm-Message-State: AOJu0YwFHyt1TbR7udaqDNtC9s9PKXD+WK45XfDHiW4vUgduk0LUiBTI 95H7dsq5pJti/6IB8HKHtE8T1WInPMkUKNhL4aYq3jIwq2I8y+u0 X-Google-Smtp-Source: AGHT+IEc9e4YeleRRqRWo9kLvaVoSKK2GRJGF4uNbxe4BG931bDzHjxBF+TAk80qz0MbJIr7gypd5Q== X-Received: by 2002:adf:e84f:0:b0:368:4e98:6e9e with SMTP id ffacd0b85a97d-36b3644559emr1597219f8f.63.1721920218050; Thu, 25 Jul 2024 08:10:18 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:1d12:b0:426:5e93:d7ca with SMTP id 5b1f17b1804b1-42803b7b796ls4662975e9.2.-pod-prod-06-eu; Thu, 25 Jul 2024 08:10:16 -0700 (PDT) X-Received: by 2002:a5d:5491:0:b0:367:94a7:12c8 with SMTP id ffacd0b85a97d-36b3637efaemr1647987f8f.6.1721920216123; Thu, 25 Jul 2024 08:10:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721920216; cv=none; d=google.com; s=arc-20160816; b=jzxiLD+HNmuFACIRJ1qaM8Fhvi8xFUj/K2GCLpb8X+XVW1VcyoVwIQs1QzFrQhYGJF hWlIu/pJlcSqqEDTEbTnH2REJmuzLKJl+ewkDsT4IF7kwuuVa8+mHSHz6mQJL0zp/Nm6 iHwillqMsLtQayDhq2+8aWAXQHSzFk8jRouD1U60K6uE9EwDqmaeGujxyq6AhAZaW12q ToZX9juPhVBn6BOlroVSRosnHJqU0MEthoGQNUdbXGHErthyWvrGkh8NZqsKac1TUvNU Sb5R9SMxPsfWcNnAWrp3zwj2Gxma4kFzPNWWTw4hMmC3XhleWwGJH/gU4cxhViIRagJt b28A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=NFA2f6aW6K5nCB+gVMdmUGt6fT62XLvTj0WWYhI8rQw=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=UV6BJzOz6pp5a6RBTglUC38EtnoTZy8B4aW8k9qb32Yvcb4MiSaHzyCKBKwplX1o5+ smVpNTDEEflF8dxDucamEKYT0viMwmPGjGbQ56Fi6UGczo3QARuJRwM05f4Cb4G0150z g+pwjbti9/HYd0LXzqQKhs7UzHKAbe3YhoSa8Cly/s3UkiP2rlpyagXX4xtvKE0k62CQ ixaJC7fnAmxzixcudR/kj2k4M8uw4Gvw9UJ79uBCGGzA6Rdt1O+itlKwDNaOfRB6B0Yr sS+Hr7Hw7SrrJNYPLsLDlkaOtT2aIIVqXTkTywRcmf2otOg/eou+PdtV23DNwemV6MRB eBqw==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id ffacd0b85a97d-36b3685c516si37702f8f.8.2024.07.25.08.10.15 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 25 Jul 2024 08:10:16 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 46PFACKb007063 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:14 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v7 05/10] base-apt: Predownload packages to base-apt before install Date: Thu, 25 Jul 2024 18:07:37 +0300 Message-ID: <20240725151006.2129-6-ubely@ilbers.de> X-Mailer: git-send-email 2.44.2 In-Reply-To: <20240725151006.2129-1-ubely@ilbers.de> References: <20240725151006.2129-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch uses debrepo script to predownload packages to base-apt repository before they are installed in rootfs. Signed-off-by: Uladzimir Bely --- meta/classes/crossvars.bbclass | 1 + meta/classes/dpkg-base.bbclass | 27 ++++++++++++++++++- meta/classes/dpkg.bbclass | 8 ++++++ meta/classes/image-locales-extension.bbclass | 5 ++++ meta/classes/image-tools-extension.bbclass | 13 +++++++++ meta/classes/rootfs.bbclass | 4 +++ .../sbuild-chroot/sbuild-chroot-host.bb | 2 ++ 7 files changed, 59 insertions(+), 1 deletion(-) diff --git a/meta/classes/crossvars.bbclass b/meta/classes/crossvars.bbclass index 00326c9a..03a74ebe 100644 --- a/meta/classes/crossvars.bbclass +++ b/meta/classes/crossvars.bbclass @@ -27,6 +27,7 @@ python __anonymous() { schroot_dir = d.getVar('SCHROOT_HOST_DIR', False) sbuild_dep = "sbuild-chroot-host" + flavor_suffix + ":do_build" sdk_toolchain = "crossbuild-essential-" + distro_arch + d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR')) else: d.setVar('BUILD_ARCH', distro_arch) schroot_dir = d.getVar('SCHROOT_TARGET_DIR', False) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 789d6c74..367ea52d 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -11,6 +11,7 @@ inherit terminal inherit repository inherit deb-dl-dir inherit essential +inherit debrepo DEPENDS ?= "" RPROVIDES ?= "${PROVIDES}" @@ -114,6 +115,14 @@ do_apt_fetch() { trap 'exit 1' INT HUP QUIT TERM ALRM USR1 trap 'schroot_cleanup' EXIT + debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${SRC_APT}" + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + schroot -r -c ${session_id} -d / -u root -- \ + sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" ' + " + fi + schroot -r -c ${session_id} -d / -u root -- \ rm /etc/apt/sources.list.d/isar-apt.list /etc/apt/preferences.d/isar-apt schroot -r -c ${session_id} -d / -- \ @@ -136,18 +145,31 @@ do_apt_fetch[network] = "${TASK_USE_NETWORK_AND_SUDO}" # Add dependency from the correct schroot: host or target do_apt_fetch[depends] += "${SCHROOT_DEP}" +# Debrepo context is created by target bootstrap, need this dependency too +do_apt_fetch[depends] += "isar-bootstrap-target:do_bootstrap" do_apt_unpack() { rm -rf ${S} schroot_create_configs + session_id=$(schroot -b -c ${SBUILD_CHROOT}) + echo "Started session: ${session_id}" + schroot_cleanup() { + schroot -q -f -e -c ${session_id} > /dev/null 2>&1 schroot_delete_configs } trap 'exit 1' INT HUP QUIT TERM ALRM USR1 trap 'schroot_cleanup' EXIT - schroot -d / -c ${SBUILD_CHROOT} -- \ + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + schroot -r -c ${session_id} -d / -u root -- \ + sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" ' + " + fi + + schroot -r -c ${session_id} -d / -- \ sh -c ' set -e for uri in $2; do @@ -157,6 +179,9 @@ do_apt_unpack() { dpkg-source -x "${dscfile}" "${PPS}" done' \ my_script "${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" "${SRC_APT}" + + # End chroot session + schroot -e -c ${session_id} schroot_delete_configs } do_apt_unpack[network] = "${TASK_USE_SUDO}" diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index bcc3f828..bf3994a6 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -111,6 +111,12 @@ dpkg_runbuild() { echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG} DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print) + debrepo_parse_dscfile "${DSC_FILE}" + + locked_update_cmd=":" + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + locked_update_cmd="flock -x /base-apt/repo.lock -c 'apt-get -y update'" + fi sbuild -A -n -c ${SBUILD_CHROOT} \ --host=${PACKAGE_ARCH} --build=${BUILD_ARCH} ${profiles} \ @@ -122,9 +128,11 @@ dpkg_runbuild() { --chroot-setup-commands="rm -f /var/log/dpkg.log" \ --chroot-setup-commands="mkdir -p ${deb_dir}" \ --chroot-setup-commands="find ${ext_deb_dir} -maxdepth 1 -name '*.deb' -exec ln -t ${deb_dir}/ -sf {} +" \ + --chroot-setup-commands="${locked_update_cmd}" \ --chroot-setup-commands="apt-get update -o Dir::Etc::SourceList=\"sources.list.d/isar-apt.list\" -o Dir::Etc::SourceParts=\"-\" -o APT::Get::List-Cleanup=\"0\"" \ --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \ --finished-build-commands="find ${deb_dir} -maxdepth 1 -type f -name '*.deb' -print -exec cp ${CP_FLAGS} -t ${ext_deb_dir}/ {} +" \ + --finished-build-commands="mkdir -p ${ext_root}" \ --finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \ --build-dir=${WORKDIR} --dist="isar" ${DSC_FILE} diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass index 9149d643..4dd93f1f 100644 --- a/meta/classes/image-locales-extension.bbclass +++ b/meta/classes/image-locales-extension.bbclass @@ -6,6 +6,8 @@ # This class extends the image.bbclass for setting locales and purging unneeded # ones. +inherit debrepo + LOCALE_GEN ?= "en_US.UTF-8 UTF-8\n\ en_US ISO-8859-1\n" LOCALE_DEFAULT ?= "en_US.UTF-8" @@ -29,6 +31,9 @@ ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += "image_install_localepurge_download" image_install_localepurge_download[weight] = "40" image_install_localepurge_download[network] = "${TASK_USE_NETWORK_AND_SUDO}" image_install_localepurge_download() { + debrepo_add_packages "${DEBREPO_WORKDIR}" "localepurge" + debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt" + sudo -E chroot '${ROOTFSDIR}' \ /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only localepurge } diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass index e8ace8f5..ddb046a8 100644 --- a/meta/classes/image-tools-extension.bbclass +++ b/meta/classes/image-tools-extension.bbclass @@ -6,6 +6,11 @@ # This file extends the image.bbclass to supply tools for futher imager functions inherit sbuild +inherit debrepo + +python __anonymous() { + d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR')) +} IMAGER_INSTALL ??= "" IMAGER_BUILD_DEPS ??= "" @@ -40,12 +45,20 @@ imager_run() { echo "Installing imager deps: ${local_install}" distro="${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" + debrepo_workdir=${DEBREPO_TARGET_DIR} if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then distro="${HOST_BASE_DISTRO}-${BASE_DISTRO_CODENAME}" + if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then + debrepo_workdir=${DEBREPO_HOST_DIR} + fi fi E="${@ isar_export_proxies(d)}" deb_dl_dir_import ${schroot_dir} ${distro} + + debrepo_add_packages --isarapt "${debrepo_workdir}" "${local_install}" + debrepo_update_apt_source_list "${schroot_dir}" "base-apt" + ${SCRIPTSDIR}/lockrun.py -r -f "${REPO_ISAR_DIR}/isar.lock" -s < X-Patchwork-Id: 3729 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 25 Jul 2024 17:10:28 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f55.google.com (mail-lf1-f55.google.com [209.85.167.55]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46PFARsu007239 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:28 +0200 Received: by mail-lf1-f55.google.com with SMTP id 2adb3069b0e04-52eff760f37sf364452e87.0 for ; Thu, 25 Jul 2024 08:10:28 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721920219; cv=pass; d=google.com; s=arc-20160816; b=Y/gVyR1y3gDWHw6YCsM4NWS54ZY/6LPN4pOcu/30NSUaxnwrfRN3i1iaw2Ji8XAlVk UszsYr4IciCkLDbbWaGJ8cD4LkNWJQrPQRT3Ljq1rbnN0ydtiQqxFa/otglP53JdM0IS ujAs9rUEQ8n1Li82H/uBg1mV6sAJT/RlUbPetr3jWid6LZlYZ5uO1BWDF6cw1wHHGnho eLtkfTahPS2T06S607liFvufxGBZU/rOy+Twh/qrFgO1kBUb0uXaOHd9BHA1D4LWdnfs 2DwrcSYdUsUMmiRdw1W0drFRJ08iphZtK1bSGrRW+zi57op+e+VdStm6T5jAnKIxW9YR PUyg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=i60pxfaTbDxglRcJT9kulFvAOACLDGR1MPKp6d7JYcs=; fh=iRe0JhK6cRUcg5K/GSIsmsKKFsvRRa3787JT4zoIsEM=; b=wu+XspZrdvaIMDT49z5LrNHDv1WwafA3gcQkzlGsbIVMBNfZinFfAtkbK6/2uRAMht xRfUifG3bTi/qcnABuMg7Uqu2mfNflpeePy9JQpO5IsUeE+URmN3RdnmCjn71NDmZuBk GZAEOzwFM5X4KJCKeDiEB5DTPj7BdKJT4yA0vlAPw5dgTfhpAKybuE/rD6O2n7HoKoxP ReJ4B2JpuUav7zz27x79zLWbmuPuQ4ZRMeNrAM1fV1Qvf4WMOrIz+h/b2gzRPrqZTKdw qqnuy6LBSKd3SJ29hLWMbQbbejLqPsW4+nr/GPM/xRYZ8J4kjMnbWkOqAcfbdAgaZJD8 McYw==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721920219; x=1722525019; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=i60pxfaTbDxglRcJT9kulFvAOACLDGR1MPKp6d7JYcs=; b=NqTD66vn6lFFfIEz+B204cECf2p4QJ/HnNlakktXpjvLhuMWHKPTclV+VcdgE8eFSh BTKdQxJSk+9ujq5u9RIDu9jgprMxb5N557eQOHM2eWmP5FJ8lEFBg2belL+J/tBT+VkF jBB1obRXbD78EY+7Q9OupoO4Tm9NZoTAJ6wx/utwWTZc9Ep8EmAQ2ipqiJ58T+A2r4c7 7X4Uktg9y/rrDgociCAxUVDNRG1FyIfUk99Ml9yCMlQiCScA53nikir1u0WG5Ho/HZFp YMAVR9wC4hFKhdTDPl/oTDOg7o+hc/bjNtFVFxJr+wMubaY9/IYVUxC/KM5U5sQ3++BE Ky5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721920219; x=1722525019; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=i60pxfaTbDxglRcJT9kulFvAOACLDGR1MPKp6d7JYcs=; b=b/RJ+KSAR0ey90MqhlXmrVUGqTq9nYkVCYxujXmXeYAuFNfO5TKwZrXMRu2JlM47Rr tFzsUJE1v9JysLTsb9zbq8+GwY40Xp+UBW7AuL97ONf90jTQ4svfGrXVo8Qth4aGD7Qd z6MzTgZeZ4tj5VDdRrT4sZud/XyJU6fTo2s1bnmwo2I044yYqhqrNfAcSU2WUE0lBgbk uqLfRjtYqlL1ZlkK5ZSxaE1JPf3GpO7hB3n9DgoqnDXVG7tJGKsAvpifSsjzNNaVvzQL a2S24Rv1m+qaeYI5fpVFh+INA7Px0zUIA+72svstvnQZY+YFtFPSH+egrKAupyieGq2O IxsQ== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCWpowUDlhHMkezY9WgN0yX1dfozYIDS/CpTz2qaFMFM63fu2HPR/V5S8lg/JjwdjKgjqC5j8Wp8IvCY7sYRvQGFO81VagM= X-Gm-Message-State: AOJu0YwvS+rm+A9pQVsCF+5k3i6j861ldySi9ADKD+hFST2Q1TUJcreL muqiF07DcDdyASlbvTQ4IvDTJhwl4jqGyaKM+sorP6uOklSiXqmc X-Google-Smtp-Source: AGHT+IGyj8FwbD/OzMJvmMijCCuVetOEGbKi0XlbvhWvYIRCGXF0M4M+PU/BRfkTXOxDGdfM2UbkiA== X-Received: by 2002:a2e:8693:0:b0:2ef:392e:e4f with SMTP id 38308e7fff4ca-2f039dc898cmr20298741fa.47.1721920218293; Thu, 25 Jul 2024 08:10:18 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:2110:b0:2ef:1eb3:473b with SMTP id 38308e7fff4ca-2f03a2b4be5ls5890011fa.0.-pod-prod-02-eu; Thu, 25 Jul 2024 08:10:16 -0700 (PDT) X-Received: by 2002:a05:651c:305:b0:2ef:2c2d:a603 with SMTP id 38308e7fff4ca-2f039ce1930mr22036231fa.21.1721920215970; Thu, 25 Jul 2024 08:10:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721920215; cv=none; d=google.com; s=arc-20160816; b=Ji5HMXt1OBjSARb1+mLxWycjPC9h3Mwgjqzgdz5yECiaoCJq7ZdFRt4hQzAH4Ct8yP 0rqA3EeVdGgsiqLQufiZn2cHkMOD7MxjmX5dQNC9GkargOdNhENchT6F7ZT4wja3u6/m zfXdhj1P30qs8tGmzarEQkvHp46CAQUUT6LoqSuTJRA/o4upT8SbwJzTnaxP2zRhJwrY wqlhiKcAfy3LJ6+9a3IRQTjQPMuEK4j8ZKzq4uMPZXPLgvz6DH26LtVNbNgsnQtKBRMh 1kekhhVboWpIoF0/szJltWSsiSRet3Z3+mstqbFgwiWJ0W2z5Ictox4MY2H6PO5YMZW0 WUWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=oba5bPyQhf1AKGklDMzdOw+hiuOFo3LfNSG/bOxIBrg=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=WkwFzs9wBNg7k9DNvOaSu11DCTyI6xRjKicWEN2ZkCH7pmZm2+zEBu7NuvdEKtC+8u Tay6yh1/Nti8WBWbxKtJmvjgaSCb8ybLiJUsx3wS0jMPpX87JcpNajxN8jULXffB1UpB mtrsAeBfH94mTuq5lsKBifGpSd2vrbef/2puPZGLKQW0Us37IM1L8t2e67UOTYaClWZR E1bhrVkfFiK/p4VyKWx2xD3ur6VCDyXGPdjx0h1SgnkMcWHSw093vjQ4RA1xzIQNNLdW wTg8cNm4CpC4mDETx21fL0V4/8htDzx8LP/XVWumnhvLkg0+9Wc00NAgnJH8q8+JWZRY 72ZQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 38308e7fff4ca-2f03d02987dsi376891fa.6.2024.07.25.08.10.15 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 25 Jul 2024 08:10:15 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 46PFACKc007063 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:15 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v7 06/10] meta: Add cache-deb-src functionality in base-apt mode Date: Thu, 25 Jul 2024 18:07:38 +0300 Message-ID: <20240725151006.2129-7-ubely@ilbers.de> X-Mailer: git-send-email 2.44.2 In-Reply-To: <20240725151006.2129-1-ubely@ilbers.de> References: <20240725151006.2129-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= Fill base-apt repo with source packages. Signed-off-by: Uladzimir Bely --- meta/classes/deb-dl-dir.bbclass | 19 +++++++++++++++++++ meta/classes/rootfs.bbclass | 3 +++ 2 files changed, 22 insertions(+) diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass index 55e56c50..48233a62 100644 --- a/meta/classes/deb-dl-dir.bbclass +++ b/meta/classes/deb-dl-dir.bbclass @@ -41,6 +41,25 @@ debsrc_undo_mounts() { EOSUDO } +debsrc_fill_base_apt() { + export rootfs="$1" + + local srcpkgs="" + for package in $(find "${REPO_BASE_DIR}" -maxdepth 6 -type f -iname '*\.deb'); do + is_not_part_of_current_build "${package}" && continue + local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )" + local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )" + local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')" + local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc") + [ -n "$dscfile" ] && continue + + if [ `echo ${srcpkgs} | grep -c "${src}=${version}"` -le 0 ]; then + srcpkgs="${srcpkgs} ${src}=${version}" + fi + done + debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${srcpkgs}" +} + debsrc_download() { export rootfs="$1" export rootfs_distro="$2" diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index f41fb498..f9c37576 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -235,6 +235,9 @@ cache_deb_src() { sudo cp -Trpn --reflink=auto "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/" deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME} + + debsrc_fill_base_apt ${ROOTFSDIR} + debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt" debsrc_download ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME} sudo rm -f "${ROOTFSDIR}"/etc/resolv.conf From patchwork Thu Jul 25 15:07:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3727 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 25 Jul 2024 17:10:27 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f56.google.com (mail-wr1-f56.google.com [209.85.221.56]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46PFARHZ007210 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:27 +0200 Received: by mail-wr1-f56.google.com with SMTP id ffacd0b85a97d-36875698d0dsf660637f8f.3 for ; Thu, 25 Jul 2024 08:10:27 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721920219; cv=pass; d=google.com; s=arc-20160816; b=nvP/udQNVCLLbeaw+sqkVWsMNMTnR/Qp0VaZgLjQqf/mTNe+oo1+G82UahRx9RADTC tB+wdYjvgjtEaik5gUixpjFRezQPaxAspDJmTdOerlI1OXtcNpiI69hhZpKUolgttVkD HAc19VG3LLhrwY5JVOsMXEedcDLXsLxGHYMlwfWRCmqDn1WXUxBtA7qRPLIL0IQPUpmE xxrNHrtz/ps/U46yQtmZA7PPecD9QM2rMSwQ0PUUErMhjbLggMdf8GGbZ2NUceaiQSNE plvSctj4zL7PsV5JhpRjMBZL+sIzxWVV7wlpO0dkVh76vsLSl1iZTwlE67ISJ1BmimGq LGrQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=twuyw0ijfCGWzpyXMbP+5J1M9G+701CrpKmUh5dX0sc=; fh=bYPYIHM2mFO2JBHDywHrmPMznow7AnsMu/Lqw5A4GAo=; b=WL4SPAEDOIrDWnNZFJWjOx4g3nlh9ECoV/pZFc6r4OmutPXNaZxMjUw/M/0YtXxrXD 7J0j4q2Sp2yE/EkQorZFMiJDC790iZvIG53UU8zhcEqOoGt1WsbDxtcgs6YZGWkGbOn/ S7ZXXn1sneQPhg2a2ohMBtiOrNm0Y0XhFzJfLdWRe1nCCiezvnC2v7DHtpfaEPZVySqu r0StDAwBTQfDQQSSrXwEHylrSvOLjxuuSGsssrWt7rKnD7SANqj9p34cBCB/QRkhyNXw Ol9b5zTo0/6DnE4evs87WsiwROivj8fQhcNZIfm2agJJ5NsmWwLMfcWdaJt48XU/gy9i ujTA==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721920219; x=1722525019; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=twuyw0ijfCGWzpyXMbP+5J1M9G+701CrpKmUh5dX0sc=; b=rdrWCnZHqay+tOF05o/5onCn/zJ+XtynXxqV4l7kQ53caTxt8gKyfINyr4K6L6PXuN tNboQHcBUq/QCHJDgVtAxLG1bxKyl7dKWBOKhXbaKONTlnUCYON6UNuiUf34Js4HK6wC SSEX5zGwgOY3htj+GB6rbFQgE0X4kz2G+qQQFTdaSNPlw8S/G0gYhhleLizegyj+/MJ7 wmKToa6TwwsCa6aq4jue4ks+/kDnJub0FHgQKxkrkh9w2+DVW7C/5gflllsBoYSeBrNt G57Ct4xxgusO6NJfkQZCkX3MoF+r2UsdtE7O3nY7T4iyJGnNougM5idPWP8dTMezd/ZE YwPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721920219; x=1722525019; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=twuyw0ijfCGWzpyXMbP+5J1M9G+701CrpKmUh5dX0sc=; b=dfU5J59p+N8l2fpVA8zqgjofnZHluT9GbJbD/+p0AZUqTZroW7l+l6fiJ+e/VJBDVE alIJqvwBe17Kuk/aRlQ81H/BFoZ59Evgc1mH6UuZ8azIv1+RjBhbTUvfP12KFTDcOEzK 4PD2tPg9UW9A+duAjNWtUYRZhb4clNwU0Av5O1tskzQlaJfCyeOtE17Kb5Meew+nvhYo bbCk4Cj2jtgclwLu3jDyCKVJYMTIrTUYs9neiBIbI3Ksz3IwLOdleHnvrd7BAu87tmSg EiLzmyfK00eQdlLcDncvI8xXhZXO2V4XKbm6FetDH3r+0LKCvt7iSjI1ENw82cUNKJxa VZXw== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCXMXDnT4ESKdnlnd3W1tDv69cnwfZnukJb1uAGtrHZ3nzpPA5OH44O4qgx+mofBTTdOszp86BKTvS+r71sG1rQu8zdgQB0= X-Gm-Message-State: AOJu0YwNSFYa+QW29eTla2w9S9QBQ98pn+xLYQegM7bu9Ir5nD5vHZ+X 9zAcAoEDRDQtsLXndV4BgK2vhNV77afqy+KaGgsQG3RlmcJGmCGH X-Google-Smtp-Source: AGHT+IF8nyEG795JivoKpvnvpHoY0zjMwfIhXl/2SaJOpTozngn4sYf1atIHW3Ui0Jhe6AVYEHzkWA== X-Received: by 2002:adf:f185:0:b0:368:6633:c54d with SMTP id ffacd0b85a97d-36b319f306emr2458852f8f.35.1721920218725; Thu, 25 Jul 2024 08:10:18 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:3ba8:b0:426:64c1:8974 with SMTP id 5b1f17b1804b1-428038880c7ls5132245e9.0.-pod-prod-01-eu; Thu, 25 Jul 2024 08:10:17 -0700 (PDT) X-Received: by 2002:a05:600c:3591:b0:426:67ad:38e3 with SMTP id 5b1f17b1804b1-428055040abmr16807335e9.3.1721920216681; Thu, 25 Jul 2024 08:10:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721920216; cv=none; d=google.com; s=arc-20160816; b=YKDmbud5Gj6uzaK+A2L9rIy6J+41zWoJZI9ZM+t5oEj6d/sFmLgFdQc85pXcObf1pk 8xhCG8pckS6GSRsRnS8FqBGymYHqqew2x2Z4uC1Tpp2iXQSscl/iLQ5f9rApWZKa7tCs 3HCAXRUtnMTJvWYa4zUX1O16nvcomLEFJOWEIdElxt+c8fwK7DSSPqcMqp3xd7Jhup+T ZMZ5z9183fzjc86pCSd5Y/30TEwB2DTaNMOdMVKnJugDjPS2zga080HC/QRqEHyk34Sw D10Uu2Mwwz1jBCn8KkrEE2UzrTrG+FOZPXfuRCtATf30Qwaz1pUaLmOZaV4gSm3qyYAf 3OZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=5g5T4rlflGN0ik3uAI29bUKTlfYegoK+GsqfZVwVr2g=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=Zy+d6aDHYyQdMI2xjD3qAaN1JxxM6pYOxakf+CEd+MuZkzmSS4wZnnJxQVhly/1lf6 tJvDxFSR35ttrsqZAvpzQc6x1n59aupdO5svvCg3Urhzid4Ri3EUF4W7MclIGBUJ0su2 fLhNiC4EN24tVY8NnONEm/mhxe++EOJ49cgwUC44/E3X1RGEaj2Ku3/qXQre8vArUF+k J9SYdVJFibSdnE9F9sQvMKoMmBSRQGk7QwepREKM1geKPQKlSbYENCk+bVqu5V1gFkVv f5X9jqqDFk0mSpitqenPvQUm9iZC4KRRELOsVVwJa1+pgysx8pI6HohQ9YtiLQsY4eCA hpOg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-42805a57f47si364625e9.2.2024.07.25.08.10.16 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 25 Jul 2024 08:10:16 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 46PFACKd007063 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:15 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v7 07/10] testsuite: Set ISAR_PREFETCH_BASE_APT by default Date: Thu, 25 Jul 2024 18:07:39 +0300 Message-ID: <20240725151006.2129-8-ubely@ilbers.de> X-Mailer: git-send-email 2.44.2 In-Reply-To: <20240725151006.2129-1-ubely@ilbers.de> References: <20240725151006.2129-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This makes Isar use `base-apt` repo in different way. Any package installation is done from `base-apt` repo which is prepopulated from external mirrors. This behaviour is disabled by default for downstreams. To enable it, set the variable to "1", like isar does in local.conf.sample. In order to be able to run CI in old mode, allow CI read the option from the environment. Also, adjust some tests (like repro one) to make them work with ISAR_PREFETCH_BASE_APT set. Signed-off-by: Uladzimir Bely --- meta-test/conf/local.conf.sample | 3 +++ testsuite/cibase.py | 4 ++++ testsuite/cibuilder.py | 8 +++++++- 3 files changed, 14 insertions(+), 1 deletion(-) diff --git a/meta-test/conf/local.conf.sample b/meta-test/conf/local.conf.sample index f692f533..941d7cce 100644 --- a/meta-test/conf/local.conf.sample +++ b/meta-test/conf/local.conf.sample @@ -34,6 +34,9 @@ IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsc IMAGE_INSTALL:remove:qemuamd64-sb = "example-module-${KERNEL_NAME}" IMAGE_INSTALL:append:qemuamd64-sb = " example-module-signed-${KERNEL_NAME}" +# Use new base-apt behaviour +ISAR_PREFETCH_BASE_APT ?= "1" + # Users and groups USERS += "root" USER_root[password] ??= "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/" diff --git a/testsuite/cibase.py b/testsuite/cibase.py index b2a804b7..f5eec864 100755 --- a/testsuite/cibase.py +++ b/testsuite/cibase.py @@ -45,9 +45,13 @@ class CIBaseTest(CIBuilder): self.fail('GPG import failed') try: + self.move_in_build_dir('tmp', 'tmp_before_repro') self.bitbake(targets, **kwargs) self.move_in_build_dir('tmp', 'tmp_middle_repro_%s' % ('signed' if signed else 'unsigned')) + + os.makedirs(f"{self.build_dir}/tmp/deploy/") + self.move_in_build_dir('tmp_middle_repro_%s/deploy/base-apt' % ('signed' if signed else 'unsigned'), 'tmp/deploy/base-apt') self.configure(gpg_pub_key=gpg_pub_key if signed else None, offline=True, sstate_dir="", **kwargs) self.bitbake(targets, **kwargs) diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py index a20e88f9..3b3d8708 100755 --- a/testsuite/cibuilder.py +++ b/testsuite/cibuilder.py @@ -104,6 +104,9 @@ class CIBuilder(Test): # get parameters from environment distro_apt_premir = os.getenv('DISTRO_APT_PREMIRRORS') + # get prefetch base apt mode from environment + prefetch_base_apt = os.getenv('ISAR_PREFETCH_BASE_APT') + self.log.info(f'===================================================\n' f'Configuring build_dir {self.build_dir}\n' f' compat_arch = {compat_arch}\n' @@ -121,6 +124,7 @@ class CIBuilder(Test): f' sstate_dir = {sstate_dir}\n' f' ccache_dir = {ccache_dir}\n' f' image_install = {image_install}\n' + f' prefetch_base_apt = {prefetch_base_apt}\n' f'===================================================') # determine bitbake_args @@ -169,7 +173,9 @@ class CIBuilder(Test): if sstate_dir: f.write('SSTATE_DIR = "%s"\n' % sstate_dir) if image_install is not None: - f.write('IMAGE_INSTALL = "%s"' % image_install) + f.write('IMAGE_INSTALL = "%s"\n' % image_install) + if prefetch_base_apt == "0": + f.write('ISAR_PREFETCH_BASE_APT = "0"\n') # include ci_build.conf in local.conf with open(self.build_dir + '/conf/local.conf', 'r+') as f: From patchwork Thu Jul 25 15:07:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3728 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 25 Jul 2024 17:10:28 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f59.google.com (mail-wr1-f59.google.com [209.85.221.59]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46PFARJk007246 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:27 +0200 Received: by mail-wr1-f59.google.com with SMTP id ffacd0b85a97d-367962f0cb0sf615727f8f.1 for ; Thu, 25 Jul 2024 08:10:27 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721920219; cv=pass; d=google.com; s=arc-20160816; b=WQmnSsaO8olGjWKKusBv93Mp2vSh1SLo8br+cqDKOhpPKnUIQByqQXu1/3oQWPPpUZ vcu9XaXWSGKpDwF6PQbiN6lTS21UMREWy+CqYt5XG4GP5F8pUMxVa4ci7KghgI3XHygN 3CFOttAss3wYlW8YfaMR+Pnj410Ab5whGUkbnSIimLeqwM/3MBGwpkHr8BSeLbZs0UOK /vCAvr4T17sPHKqcuYg+0d/m/d6uUqz3L/7xFGlrNYDKwaXY5uMTmYK7jzftcg9evb2W 2XGjlHVR7WCeddpFE8eft2J09Pn1/+12QesifU3a9CEnHNmYfaUT4gAiHr8ngIQDGhdw loAA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=522GIk3qGvA7b3TtL8lJt42lSeY6QqBxCFh9T9HJ4po=; fh=I86OVS+rPzjBzd7ljAu2BqFFRJBB4tOz1XN137dG3KA=; b=qeD4UVYfbMbf8uNXWQ8g8TE400zy5pwlHP/+bqEdNPXeTJnqe49ijichjtn0phVdxp KoRTRcC9QIBrPyIuXG+vzkgZ1OFlzAowTNbHXZVPUInccZ61S3hkWvaY2DMWDdxfZT2L DTItle9jNhGpY2BSnAJW2UKaqB8HIxOrrcFMYsDEznGjGHuCrC4uexNE0q3GjIi19GlA yZAc4c5ceCU9vJWqCC+wTrSd/YrWv4dQfsysbsht7j2jlbqTsSJZROpm0RPSVxiDPAhD fNfaOwJiG9JZQWztqFwTEiv2bWo7+NRiw0RE+QJxFFtniTYayogB4KM7DN33oYL5FaGD 6BOQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721920219; x=1722525019; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=522GIk3qGvA7b3TtL8lJt42lSeY6QqBxCFh9T9HJ4po=; b=lWzZDNW3CDtZECQ2ReWG24sX6ktw2rno7qtO2I8P2TSMFNxWTGUeG7y+0elx5nGd9a D8r4jrAtZNRDjn5YOlW1IocVYDiQ/R88wM+Mca448nJghXy/4e+xEJ8DsbWOfLgNhAKP un4Ob0p8GA/PkmnPbMd8LftbJ9KpFWzzMaYgmozNwPnc6dUzt9JIFbAz2UPUkx7jCk2A a3eHKvQwr4qqyg6n6i4MrteS43a4ESdcFmch7jkAsJT0N8nQ5qxWAeDQAqW6ggJxsF8k B8cfZ5xxqSy50N3OOAiOp4kYKhek9D5bzVXZZujXAaI6GA6JWuLT9PPKFYi8L+maQOzu /fiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721920219; x=1722525019; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=522GIk3qGvA7b3TtL8lJt42lSeY6QqBxCFh9T9HJ4po=; b=O2uKNclK+8UCG/46zOBVcPSXe6o25ixQniBuFlWpocmVBcpREu05Y1MbAO5VdjdjZW E3J90JU8pJ6OJOECB1AzPQjPnrR+lkyJyEadsvv2uTnokaG7idvm6mp7fmNaqUttYlOq fyDH0jm/zyPWkwGIK2TxOi6+y8BwOq8XMqXDqzLJkKMWbjMOZHwDfK8qoh+iJDOLV0CC L/4FfnJ427nex5yQcZGz6XAtPLfe6DSWkm77UoNAKShc8MVpitf/P52ZX1Q5SFDIcF+n SSim8yFGz49Ivm5YGKhPv2nkfMhDeQ7hrIGkGdIBdVbSbdEeF93xY/zYr5iVFKm6QNfC gsKA== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCU/royecRSYIT2KppdiHT7sMAyTWWfGgWf7E6yRHa1p80IWObvrMUDtu94RJKNgvNjKZXYSOY8UKBGpXdulYXT9yHf2bdE= X-Gm-Message-State: AOJu0YxT/7zEj8IvatRQmwaszIRPw7fEMQIfl9gfJvhUo/ANrzZ5cIWz OAfvMCKEH+QGjzfKJemRvTnKPFaUJYDcYkOpyR2G7eIoqWKgjxpM X-Google-Smtp-Source: AGHT+IGt1/zPGkWgW/T3/gYaHZJiaMYv+8rzpBq2G4dEW1CwucJosNHad8ZD7Fof2j/mX3UjJQl3ug== X-Received: by 2002:adf:f212:0:b0:367:2945:2bfa with SMTP id ffacd0b85a97d-36b3642a421mr1752625f8f.52.1721920219014; Thu, 25 Jul 2024 08:10:19 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6000:ac4:b0:368:4489:3f40 with SMTP id ffacd0b85a97d-36b31acb718ls319722f8f.2.-pod-prod-07-eu; Thu, 25 Jul 2024 08:10:17 -0700 (PDT) X-Received: by 2002:a5d:5f83:0:b0:367:8ff5:5858 with SMTP id ffacd0b85a97d-36b3642fd61mr2078618f8f.56.1721920216956; Thu, 25 Jul 2024 08:10:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721920216; cv=none; d=google.com; s=arc-20160816; b=jbuuuhDWwgDtt3pvuOzmbuhDjJqphgQem+Hn0hdkwoWjx22e+DKCfIM7+x0Gm6rbG0 cHrRMBGAVCE6PV2d/rHKSztuH5pO60W/51G/zw13yMaWZSy7zv+GGtT7ZzYTKfpitWfQ OA62Xr1vD8l7pw3hZ/FUUgbzxlJplAO0spuPvRqiwVVh/Mvzrsj/+zxaKaO4GNLbc/ef Bea9IZW8MjdUk18bV6B49L/ElOGmKp+QEHBkuec75RdT9rjYWYZFlF2c6LGxIoMzWvzt O5hWWDDIK3tD/moDueHcu5fVI+/KUXhJb48Pl2jHz0VJE/Y9VF0FNKadUfTyhVYbfAOX FycA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=g6z0GeFNsqvuxKqEfExfwSjlOuAJvfRElHeWka/GJb4=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=zcirEQYstuiacWvKEK2xm45itgrROg/EYalbdSV70rxTFRC7a9yhoZ2RpW5nUMg6Bu UjkyvF3SIDxJOy++nixiuj2XWA/x18o+5tJam2uMrRo2bq83dJqH7E5rbstsXCwNTSo8 fFq9mbNHe0tWSr9X79Xrw4MkBppuJY4ceH84tmYczsmJvhO1DUYI0uuVfDLCjszzIfKq 7zaxyNgtId0XRJH+swyAUgOhKg5LRUynHFSmhabWYqJ4P7MS2ack1UFOT5quCT057KtB /uD2ai/1zhuFlcWnKQ9V9FQ2R29UJabnU0nLKRmYYNNdVTZUCjF24SwEnJ7tHZJTJlKR VvPg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id ffacd0b85a97d-36b36800e14si34072f8f.3.2024.07.25.08.10.16 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 25 Jul 2024 08:10:16 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 46PFACKe007063 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:15 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v7 08/10] Disable deb-dl-dir in base-apt prefetch mode Date: Thu, 25 Jul 2024 18:07:40 +0300 Message-ID: <20240725151006.2129-9-ubely@ilbers.de> X-Mailer: git-send-email 2.44.2 In-Reply-To: <20240725151006.2129-1-ubely@ilbers.de> References: <20240725151006.2129-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= Since all packages and source packages are placed to base-apt repo during build, there is no need to have them in one more place. Signed-off-by: Uladzimir Bely --- meta/classes/deb-dl-dir.bbclass | 2 ++ 1 file changed, 2 insertions(+) diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass index 48233a62..327e8be5 100644 --- a/meta/classes/deb-dl-dir.bbclass +++ b/meta/classes/deb-dl-dir.bbclass @@ -91,6 +91,7 @@ deb_dl_dir_import() { export rootfs="${1}" sudo mkdir -p "${rootfs}"/var/cache/apt/archives/ [ ! -d "${pc}" ] && return 0 + [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] && return 0 flock -s "${pc}".lock sudo -Es << 'EOSUDO' set -e printenv | grep -q BB_VERBOSE_LOGS && set -x @@ -108,6 +109,7 @@ deb_dl_dir_export() { export rootfs="${1}" export owner=$(id -u):$(id -g) mkdir -p "${pc}" + [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] && return 0 isar_debs="$(${SCRIPTSDIR}/lockrun.py -r -f '${REPO_ISAR_DIR}/isar.lock' -c \ "find '${REPO_ISAR_DIR}/${DISTRO}' -name '*.deb' -print")" From patchwork Thu Jul 25 15:07:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3725 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 25 Jul 2024 17:10:27 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f184.google.com (mail-lj1-f184.google.com [209.85.208.184]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46PFAQAG007174 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:26 +0200 Received: by mail-lj1-f184.google.com with SMTP id 38308e7fff4ca-2f01b9ae749sf2667431fa.1 for ; Thu, 25 Jul 2024 08:10:26 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721920220; cv=pass; d=google.com; s=arc-20160816; b=S5CxjFIIFGzXDOIkKayBvakWyoaMnm7RW5iqWbwYl3yAD6x7GM/5ligBUyyNI2Z0C3 4x7cTBzcNutLBp18kvMgCCwX4RDkjnvkOVc9/1PWO0n9m55pd5YwoSFB3t9KzPXA0NWP l7yEg3iGXU3RnkSzlOjbJMr07oba4S5YvutmF5pZdwkDIw8FkKuB4hxuRlPdOm/fBEFM VyNeAjbfOpbK8iaejtf+Als5jR4Tm6P3CGZrQfo9QOt6b20LPQEK37f/27QaNCsjB1DK BzzzKKTU1tqhAp7JmhQmpF0BtWsAYjeFj6f6DFzMGd35W1FuUGyOPZS1/viFS6P4L74G qVCA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=B3+CYQS88e9emfLfiquOOk0qImO/L7tYQ45bx+1K6GA=; fh=VCpuNv2J4yls+oik39n5QpsoV5+YnHNWAS+8buWkGs8=; b=Adwyaj0km7gdfuunnR9H5GaWB1j23keaDzcODx2ciUj0gJtWON999nYXlnmBfPAiUK G3RjLl3s2AP/eKsU8ymD1ioreNWrQ6SOPridKnbQhirsplBMpJVQmpcCnt4j5CjmDajM 9VnuNdqQ0oEWde7rb0VC3/Scg6wfmFUkoeKc8S+3dRyuoljArslZgIYjyn4yBLqW88i4 P8Hmp0fHjFCpOXGnGjhkRQ4NZUqZNj5ObO3+zRj0zN5eRbaHE98k/adqIIF//2f8iBuG vYwg8iedkV+fnvKqfFf6LUWL5fghdJ5JkyANDAlYUrNKWZIOVeJ07ySLpOLFTfSTQV9B 4Q0w==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721920220; x=1722525020; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=B3+CYQS88e9emfLfiquOOk0qImO/L7tYQ45bx+1K6GA=; b=h8oml3b23IjvfQ5kwi/j3G8DVlCbdWCaZ8DP+9P6vei2Ol8R2u8guXHWiJIsT2MBNt 8zcDMpUA5JpPeDPwfAg3TeNA3wVdkss1V4gJzSmISPnEvmAmrgCBI/sR83So0BXdttZ0 whFcUmt2/64mbmULab0TsUwSSgzbOo+Fq87HPWKveE4ah1FYWHgjds2Ddz/sodYPvii2 MfFPnpTgXeqFkjPYhJdjB+ZUbafHwMG7VVroOU2NzhFGYNvNKM7XLbx0+/EyMD+cW2/0 RLwp/AVxA73kwgQ2Nve2JCqEqg5FZOKgxB4BFiEqS9F2Lo+KOFMxKhAlXyVnJfmiSrER x2gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721920220; x=1722525020; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=B3+CYQS88e9emfLfiquOOk0qImO/L7tYQ45bx+1K6GA=; b=LvvncyhIMnqG8n54cmpDq4T0DvqmEidmrDoFre8ZITcwkqeQDjo6MPP9SnDKAg1cbD 3H7dii1QkUWKXbexmVRvJFKANaDRy9Eitlg+X1uu/3aaVOHMXPMaIYTMCU1Wwzc5xJW0 gYNP6TBbgumEBxrdthmP9J7Y7BisXaM3Gs3YfgFiWypYp7AAloE7MzusqWDt3qYh3CsQ x+s/rlMd0VX92zLbeNMj1MaLUS0W+8BKd1HQOatgs6gjImHzPm322UsYcuVLT7EEHRh3 iUsgisDOvi+pXfnB79sTB7oUASN9KDND9XojV9XKTgae5dlesXTp7IjR+jmL/dtNSMPo t16Q== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCWA23gEson6iMBx9z+H2UV3GyTQEpKAHkUfDiydDXqgFiIpBkhhZymNLdpmaD3nwff8J/U9f1e+yTMlgvvHh7OPTjbAEwY= X-Gm-Message-State: AOJu0YzIa8GYVdfRFJCbC0z7R/JtvDdeLq6cAdwZ1opsYeWVQR4QzA+E sAjDlBnBy0XDMdCc911/N7nJVl5UPfdzv0aKz/Qs4DfbYKfh0M2o X-Google-Smtp-Source: AGHT+IGYf0PGDVLuq0N6N5t4z4CEe7Imvokm3QT9VvjI9Tt4DaXAhOL9heqUY59ahdzgzauiJ/NBUQ== X-Received: by 2002:a2e:b893:0:b0:2ef:2a68:beca with SMTP id 38308e7fff4ca-2f03db8fde7mr17935161fa.24.1721920219470; Thu, 25 Jul 2024 08:10:19 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:a99d:0:b0:2ef:2eec:504f with SMTP id 38308e7fff4ca-2f03aa79f40ls157651fa.1.-pod-prod-07-eu; Thu, 25 Jul 2024 08:10:17 -0700 (PDT) X-Received: by 2002:a2e:9059:0:b0:2ef:2e6b:410e with SMTP id 38308e7fff4ca-2f03db70d32mr16679001fa.4.1721920217168; Thu, 25 Jul 2024 08:10:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721920217; cv=none; d=google.com; s=arc-20160816; b=x2JzZyLGuas+SMUz1enRhcnOczTbnhdq69Atx97hc7/aLXF76XVeAclfcduMvsidbd p3oOQ9fUrkx556IVqLxmfv8HWp4kIR5mSmKSuog9T4T83N/oDGLJM+8+Hsl1xHM99ML/ 30nmyI+cHNTnOrqd+9mtzNh7N57DLSl+hd2NI08WkjB9ekYlDpaMwt2F0Y1r6QgL/ZY4 aibdthFhRYPYC7/sByrZxkBf+OMCJ9PmxNVHdl5DopGcnTcxOsP8pEnoc82T9Rc4oc7R Fv5Nuk7pCH+TLXslgsUnnvyEfs8vPnT8Z8gtjET8Z04MEloFpdLKJOeV1zXmlkzBxB8S +qLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=+uEQGHyl2Wp1H+wYbP7lGgX8jFVsZCbM9wng1ShixYg=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=Z9ILxslDQEUwqxQ1QMbsGVHkVqIwoU9OLxHEzmWhbV08vtpBKg8a+jqZamEdNT6Rcn FcjBWeaxSZzL3hEINlMRAsSoq3xTRZ5eCL31lA89ya3fNFr2qOO8J+PU1wxPxUmVTJAi 6k+PLMUdufXC+H6+wogGWFuLHXmQmRV40HZZnN/7rSKCRj5VZ68qWXJHnWqHHH+8xqgR 2N6QBGIo8y8/YMoqVrJWMGCfXL6SGfEUSQrr79WD3n8bq5l4yhmrlykG3ebPWxS8+lm0 MBOD5MroCJbLX8i+Vln+cNHzGceTMZcqdWkdVKJR/xLX1SMb2rjas7heShCfsgRT67fG g0ww==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 4fb4d7f45d1cf-5ac6673bbcbsi45947a12.5.2024.07.25.08.10.16 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 25 Jul 2024 08:10:17 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 46PFACKf007063 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:16 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v7 09/10] kas: Add PREFETCH_BASE_APT config entry Date: Thu, 25 Jul 2024 18:07:41 +0300 Message-ID: <20240725151006.2129-10-ubely@ilbers.de> X-Mailer: git-send-email 2.44.2 In-Reply-To: <20240725151006.2129-1-ubely@ilbers.de> References: <20240725151006.2129-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This option allows to set ISAR_PREFETCH_BASE_APT to "0" or "1" and choose between old and new base-apt behaviour. Docker image kas uses should have "python3-apt" preinstalled in order to have new functionality working. Signed-off-by: Uladzimir Bely --- kas/opt/Kconfig | 13 +++++++++++++ kas/opt/prefetch-base-apt.yaml | 9 +++++++++ 2 files changed, 22 insertions(+) create mode 100644 kas/opt/prefetch-base-apt.yaml diff --git a/kas/opt/Kconfig b/kas/opt/Kconfig index 701bd263..e1476b8a 100644 --- a/kas/opt/Kconfig +++ b/kas/opt/Kconfig @@ -157,3 +157,16 @@ config KAS_INCLUDE_CACHE_DEB_SRC string default "kas/opt/cache-deb-src.yaml" depends on CACHE_DEB_SRC + + +config PREFETCH_BASE_APT + bool "Prefetch base-apt repo" + default y + help + This makse Isar always take packages from base-apt repository where they + are prefetched by debrepo script before requested. + +config KAS_INCLUDE_PREFETCH_BASE_APT + string + default "kas/opt/prefetch-base-apt.yaml" + depends on PREFETCH_BASE_APT diff --git a/kas/opt/prefetch-base-apt.yaml b/kas/opt/prefetch-base-apt.yaml new file mode 100644 index 00000000..0cbdb93f --- /dev/null +++ b/kas/opt/prefetch-base-apt.yaml @@ -0,0 +1,9 @@ +# This software is a part of Isar. +# Copyright (C) 2024 ilbers GmbH + +header: + version: 14 + +local_conf_header: + prefetch-base-apt: | + ISAR_PREFETCH_BASE_APT = "1" From patchwork Thu Jul 25 15:07:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3724 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 25 Jul 2024 17:10:26 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f60.google.com (mail-wm1-f60.google.com [209.85.128.60]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46PFAPkk007161 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:26 +0200 Received: by mail-wm1-f60.google.com with SMTP id 5b1f17b1804b1-426d316a96csf7779965e9.0 for ; Thu, 25 Jul 2024 08:10:26 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721920219; cv=pass; d=google.com; s=arc-20160816; b=qnfxrRZH5vWR1s0LWQAnTdGphWvaWx3lxiOm0XKXHcLl4oKRjFmO4RIVoJWZ/q+cL6 aBkTKX5XfvFOwR+ZXdQ1dfDqqI8Ls9vWb8wTo7bUG2NDChcAZ55Phlnyxfp5zsMRkDvX UWhAsydd+bmfDeFdPMQ3mmRkoaEJLPykqRaB25rBJzdQs9qVJx0AcMVAXv/GMBAqqn7o sK6ddsQQWUKoC1R0KHDhGg2ECgpEjxZUbkmOMDuaIxSaWXwFn+qg5kxNk6lvvTPM9+Yb rxNVPRphiH2+RbdVBcvKAGHSHTuRiT0hoBCOgAg/2XeOsYDuUfxulx7CqqfQcRy7uK5M 0znA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=dHA2YSm4U2nouSjekNyIZFmBW7Wpgkn7D+YOHmy/EFU=; fh=UlmsRIS07g834RD4il7msfpVTE5b16rIqeH+575oYkU=; b=PWHxgoPtu77V93Zh1DIGnKU2pFq0fza/c/2mheb+mjXTf7NwnGyoKYWS0BWP2+S4js WxitlI/mTxHQZVUMFgEXTZTaKRfWlWV3MmI1n/089MvkYIHJJgkZRWEm3Focb2BH6je0 qPW3WprOPlt1IFTaldedEFboBOES91z6uas7VZOF5Bpd3NoYGY+wVDGMHCADD4dYDaXW c3Vtz6y5TtgIQGrL2/CryK1CuI6fZe4dX9GSq7SdnidrcYN68pvM+VJBJbUcZboFEyf2 7bXJhmMKjAzFzpfrUnIjbN2xqN0RiE7Hvc4Fv3ONjraaMjh8j4130mt3WFhPnBxkWe1R 91rw==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721920219; x=1722525019; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=dHA2YSm4U2nouSjekNyIZFmBW7Wpgkn7D+YOHmy/EFU=; b=V6VMfEdAxT5OnTHo3BHWTxN1xCxXtKPnvtrF3NoErM7behbf6tFAjLQx0cO5vkJQVy v6gQwC2/7hKcNZ6PhTJNWMN2EInmFDovD6/95g9BUHgt5PkfWuGsiBYCNLH/3y4ElTP9 PRNntfcJ0fBYMaIjv46i5O+59YlBamwoeyf1OucQxNS3STydSTwhYa1Q5OO+PfcH4bxc aMHQCaVyH/WxbiNsY7VBi/v5D9K637GpvHcYHBXnJ6wcP6gIPR93BOYbAdvoBxI73Sn3 y8kopSHIA1DfcoDno3L0h15pNah5kL+xUXosb82khHxeoByxBpaa9DGfAbwI4eKHLSnn HDgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721920219; x=1722525019; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=dHA2YSm4U2nouSjekNyIZFmBW7Wpgkn7D+YOHmy/EFU=; b=BSF3jIms5AlZ+GW3LGFSXOCTfTT1tVDUkRnMmo0Lvk1FZGAlgwsTAmUklWsceZMKVN Knw9hLvKr+G88LXVh0Zbwch3b76M3hYcV1MBOuuN2UTq7LSQB3FjuBXq+S9xkKT/i6Ct QVmLcdRiewGJfoY5wy8KUlVN07vMTVKmlQ5SfHnFc7aFGX/G8bHqGFWFjLn3VOnIoVSO FgHYcWrHjHfP0eflO78OcnyKw6aFmpBGBXN2mro3xRoXgQNTAx1eN2MTG3vJBKWQ7UwM OQYDYSeIfgaJGMWcEzvFJbq22dPBvFwwfTk+jksEzSh/p8S4yrLdDCYtaBdcS/rf9irE hxQg== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCW6xRx1OodBVgEpx1nK0HrXiELvFerZC44zEZSNMxrDLH2ULGVO5rc9rPlbrS1Ut3p2jqwyurnG6K8rjd+6HVHTYOkpqEs= X-Gm-Message-State: AOJu0YycsYEPBmAX4F2g02BGJ+S89xEkoUh4VWz2WgKGbkPb3vdavAnj vOZTdW6E2atoxjkKSN2l7hRSoTzargmwhICPwOpSBACZHMyncYv5 X-Google-Smtp-Source: AGHT+IHo3PhvR+hfzvYTtqJSfR1Nx+QihpScpC03gAIXmmqVbVwCjd7D56x5gr5zKItopa6RFSNZMg== X-Received: by 2002:a05:600c:1f81:b0:426:6eae:6596 with SMTP id 5b1f17b1804b1-42806bf4b2amr15356005e9.25.1721920219363; Thu, 25 Jul 2024 08:10:19 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:3c86:b0:425:68a1:9da9 with SMTP id 5b1f17b1804b1-42803b6a203ls4664315e9.2.-pod-prod-09-eu; Thu, 25 Jul 2024 08:10:17 -0700 (PDT) X-Received: by 2002:adf:e349:0:b0:368:4226:4085 with SMTP id ffacd0b85a97d-36b363e4b83mr1537346f8f.8.1721920217558; Thu, 25 Jul 2024 08:10:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721920217; cv=none; d=google.com; s=arc-20160816; b=Ysexqo64pjl6ziWj6E/8cTRM/g6M7CAeFvmZR+9SytYnXsPSAjzoOmKzr8LAZ5JJHg Ta7ekVHM5baBp39h5TmM+wcV9Oq/KT2Fp4U6/aVqKekj5Pb0IVQaK87p/IaxUcRmn41u bUp+bEMAi1wYjbHIjaRulOOU1rKrqlUvz53SLgks0B0odMjbJRCogESjokIKNOTCJEiW 8SHQz9dAUqAV4e/86Yt8WDXEvSVm84NAJtyzN8AA2P/bKe6/Vr0HlsXwvTLDAlVslP4O no0OWj8Dhq0ldm6GK7kS8vY+LIcaQ2QRKYEAGUG0C22rBzA0JIs2fM33j5R7GoIysk2o Xqzg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=6TiQjxMZm0s0RwtlvOpYU0DF+GKRUHNvG6d3T4Gnqbo=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=mB3gMDt1U07eYB1jNLOc3JNsXLSuTn38l0WbZCVYa2fysN1OrWMQZSVuPcC6Eh61O+ 28u/2OUWHfCDTA0mRkjB9Z3o15TraFsM6VW928f2Rrmk5M3Kby2WncSrd5K9LLiwWLV+ 7DDrzxSahhWulgKl/0BHo+6HsgCgApEtHGLBUJFQQqg1f4Od2cMZFgsaAqIkLDfzgVfx CYJDx75X0NHu4/Q9CAM+kH8G3t0lLcTUsQaWZQGDOJFromwsV8XXFm/HRt3+sFpHMMbN XhAqdWhKW4hlbglB68QeRkMjR8Su8CWJDIXWVr74fWwK2Fm7HSgb8e5ggAsws5b4WSX8 6v2g==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-427f1f6e8f0si3076265e9.1.2024.07.25.08.10.17 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 25 Jul 2024 08:10:17 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 46PFACKg007063 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:16 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v7 10/10] ci_build.sh: Install python3-apt if not installed Date: Thu, 25 Jul 2024 18:07:42 +0300 Message-ID: <20240725151006.2129-11-ubely@ilbers.de> X-Mailer: git-send-email 2.44.2 In-Reply-To: <20240725151006.2129-1-ubely@ilbers.de> References: <20240725151006.2129-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This is mostly related to gitlab CI that migth use an image without preinstalled python3-apt. Also, make system python packages available in virtualenv. Signed-off-by: Uladzimir Bely --- scripts/ci_build.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh index 0aa2403d..b17d4622 100755 --- a/scripts/ci_build.sh +++ b/scripts/ci_build.sh @@ -19,7 +19,7 @@ if ! command -v avocado > /dev/null; then sudo apt-get update -qq sudo apt-get install -y virtualenv rm -rf /tmp/avocado_venv - virtualenv --python python3 /tmp/avocado_venv + virtualenv --python python3 /tmp/avocado_venv --system-site-packages # shellcheck source=/dev/null source /tmp/avocado_venv/bin/activate pip install avocado-framework==103.0 @@ -131,6 +131,12 @@ if echo "$TAGS" | grep -Fqive "-startvm"; then fi fi +# install python3-apt +if [ ! -f /usr/share/doc/python3-apt/copyright ]; then + sudo apt-get update -qq + sudo apt-get install -y python3-apt +fi + # Provide working path mkdir -p .config/avocado cat < .config/avocado/avocado.conf