From patchwork Mon Jul 27 11:26:20 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aliaksei Karpovich X-Patchwork-Id: 5286 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 27 Jul 2026 13:29:13 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f61.google.com (mail-wm1-f61.google.com [209.85.128.61]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 66RBTCB8001019 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 27 Jul 2026 13:29:12 +0200 Received: by mail-wm1-f61.google.com with SMTP id 5b1f17b1804b1-492488f8583sf24049495e9.2 for ; Mon, 27 Jul 2026 04:29:12 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1785151747; cv=pass; d=google.com; s=arc-20260327; b=b+VEIHtF+0ctvqN37J3gtdSTDmxTeQCyHYeslv8rya3wSAO/DBOyMRIZ7s0R0sqLrC mA2Gc+psVT6tGqq7m8s5b6VCMlBzq4rz4iy6XyuPZf9yfh0Z0wBwJkiBfBS2/dgUbb17 oofDPWQGZ+ZYAaIIaTG1uTXiJV9DaleIdpV2I/Y1M3rl1V3P21boR5RnvDE+VqaNl2OH GsjiRY7opiaVUbHPPLkumZLWJS4oKqQhAMvcrzDnGkUXWh8LUqEby5eN47nrD3etxq13 SGs2qbR3Ku91f/Afw39U0gPmpNOveO7N6kXm0QxEQ5eiMKoquidQddcDFNAoxj4gfeVc M6jw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=t16oWWIXdPGe+9fxIs8ePq7211igJcoFIvfdZF29mus=; fh=2+hZ8mL0Uhnf1zyIvdQBzJDAexcosLItaXMnugLjPWA=; b=O4G1KGnukRFz/PF4gNnH5r8KI0BhHJkXhkFpBSuECsg06QxUElCZ47+G5wAwuy6bqg uaiEwXyhcbR7KFB4QbmFn0llyn+zRpSc2cUSUxiocPnSzbDI+Itrv2IxzmUHhzmu38nN 7iPg7MUroy3WgJVhBoVTRhvi8DQUU3oS2BKSCCNmzA5Vu//kw0NtKw/E0tVymN+shak1 WQEcZMzJoTV6dk7MSgP8cobjjUbquoBb12n6lLZhOT/rQOO7Ti0/XQrtFfc4FDewIXuk lJ1oopbusA1n20/Bz9NmixnYpz6i8E7VRfxItKgRB7soRF90CpZuQoG/bjlhRXetGp24 bf1g==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20251104; t=1785151747; x=1785756547; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:content-type :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from:sender :from:to:cc:subject:date:message-id:reply-to:content-type; bh=t16oWWIXdPGe+9fxIs8ePq7211igJcoFIvfdZF29mus=; b=hKiNOoi+nkPRNU4rkgNpSkAdao8/SuB5SMPA36pRFzd6o29cGoyKvNdoQFhaXCLIin 8gPe9H4le11s/xn6qcx0c1UsC/JvJnk0pHVNsBPeiYsJq0oC/5Z12nBB8GEsSyW0LzOz jK3ynagaXg03hJ7/Xr8wHhou7l6FqnCINnSA6G8r5JUn77XQkwuJfmqa1MyCBL4W9GJY mGl2Qgok9x4ilfFH4DFh+J9ULSWXGS/1FdrXhcwHOJC1/UIkv4dUlqkzmw4iQ7dHzWB+ tutdWMN7NX1Rl9vXeil9kXPeQpSNs+rIuXWDjZFSaoDdkxLlxaZKjXsNZ+HxoGMlxr5N 894g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785151747; x=1785756547; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :content-type:x-original-authentication-results:x-original-sender :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:sender:from:to:cc:subject:date :message-id:reply-to:content-type; bh=t16oWWIXdPGe+9fxIs8ePq7211igJcoFIvfdZF29mus=; b=Zscp8WBgXvSk+gADZ60FsrCNr9ueBRIKlTf22yrt5hFkBaEPji/CwoElwDpU6WExwO zDBrLh3YZFSIwX7uVW8R7ql0+QVkpeJXQi3XdKCCdWeLtfAd/muAAkCyv0FgsiPAtVwx Iaol+eFFBlSi7XLDSWCjjMayYy6T/VE7v0OL3sss39G5s9TqSZf/l0J2gl8jcpvh/htt q3c6Me3yz+0afknsdpHped2DIhQ5g4Zvmj3kwd5lwxAtee8Sn1ZHurHiLgJBwBgHkuYT AEOw/54qVzAJPNCJfn3tx/+T6crtox61JbY41ccij9NQa1IV+dqEqee6azm1fpIbiBY1 Gbog== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AHgh+Ro43NXn1988XvaPL+85YdFxsAawH+q9L+wyUX3vnek1xxDtG4MS8oc/4zntls5GlFdSdQa651g=@isar-build.org X-Gm-Message-State: AOJu0Yzx8pUBQ4b9LwCI8/2nG0Dx8qM+ZTPoXd9fwK8gGH9VAvC0cC77 tF/5HIrWhsXqiNVAGeXCjMdzSYDhu37LN4ncaUNPWCQr3vDeU1vCHV6t X-Received: by 2002:a05:600c:a00b:b0:493:c182:6b08 with SMTP id 5b1f17b1804b1-496b570f64cmr106783245e9.36.1785151747002; Mon, 27 Jul 2026 04:29:07 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h="Aa7YSPTlZEjm7v9d2wg9R8liOdbayHhqNBWga82A0PNk3Mc03A==" Received: by 2002:a5d:4f8d:0:b0:451:2ff8:a40d with SMTP id ffacd0b85a97d-47f8c8bffc2ls2705173f8f.1.-pod-prod-02-eu; Mon, 27 Jul 2026 04:29:02 -0700 (PDT) X-Received: by 2002:a5d:5f92:0:b0:47f:9ac6:c9bc with SMTP id ffacd0b85a97d-47f9fc8a7b4mr10331689f8f.2.1785151742015; Mon, 27 Jul 2026 04:29:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1785151742; cv=none; d=google.com; s=arc-20260327; b=lCglv1lVDq1hUfTTpk4Q7bfqKrYYpxPIc0dWepg+dEI/CN+T4vrx7e7eFYiRAehsx0 aTDgfkUI0r8qZ0eHtKL1MVln5KO1hthSYqqZ4GrwWkc8yZryLx1/sqqMGot1On1xxuy0 fetLLKGOtUe6rqxze6ByeBKut1Fsqkc4N9iZNXoh1rT6BS46zV3mzvsVELPHwIU3a4bS raVW2rYHdw4adUCZ+FTfDQ+RNmp7Fxhq97qBGE3v/G+eMzED16l9OpEyJh2XuhTQDJX6 W5rnTpTvekdpry4p0c9XwXgCnFsfqwvdbIxJFGDajqZwW2ZMXAjvQpqRg2Q6Q6OLpQ7P xaSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=+oOkneHBsjhGCKih0dwrkhEX2lWNlz55rfkDGBuOYOQ=; fh=2ALxNboLzDt503PhX56nFQ4pQzOkkVjVTCOSpqCio6o=; b=s4sVGb7TvMWNjOq/ABql64QYpxoI0Gt9KIWfYKsdWmSyufgGpN0dXoqlOMvL2AbUFb 2Pm74UKRihmvp4H7im5bKgUuM7+BXmHxu7lMMsxD44U+rY0sWrsiENaVEqdZejMcZoev hzCfRUDAgOqoXfGasfotCuSZzhzi5EOTduulbe4GTx71nO4XmlHM7wlTWJPngt4y6z9n cPUJPKpv2FslxUoLidFUYkouEV/xvlXeSA7nX4jGBnOQsBs9bYbYm2rS28qDOiKiqG1k 1e9MY7a1QWOnARrXEFAW+F1TefQNuZcAHVppckbDd9MdXTBb/9JEAUzEH6CF1BCJ49VY dgTg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id ffacd0b85a97d-47f85c81b94si304718f8f.7.2026.07.27.04.29.01 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 27 Jul 2026 04:29:01 -0700 (PDT) Received-SPF: pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from akarpovich-Victus-by-HP-Laptop-16-e0xxx.promwad.corp (62.122.174.6.serverel.net [62.122.174.6] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 66RBSjdS000930 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 27 Jul 2026 13:29:01 +0200 From: Aliaksei Karpovich To: isar-users@googlegroups.com Cc: Uladzimir Bely Subject: [v8 02/14] meta: Add debrepo bbclass handling base-apt prefetching Date: Mon, 27 Jul 2026 13:26:20 +0200 Message-ID: <20260727112812.2255297-3-akarpovich@ilbers.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260727112812.2255297-1-akarpovich@ilbers.de> References: <20260727112812.2255297-1-akarpovich@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2, RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: akarpovich@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Uladzimir Bely This class uses 'scripts/debrepo' python script to prefetch given packages or sources to local base-apt repository. Signed-off-by: Uladzimir Bely --- RECIPE-API-CHANGELOG.md | 10 ++++ doc/user_manual.md | 1 + meta/classes/debrepo.bbclass | 90 ++++++++++++++++++++++++++++++++++++ meta/conf/bitbake.conf | 5 ++ 4 files changed, 106 insertions(+) create mode 100644 meta/classes/debrepo.bbclass diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index f5b84c84..7f89d121 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -1137,3 +1137,13 @@ targeting Microsoft Hyper-V virtual machines on amd64. The machine produces a `.vhdx` disk image using the WIC image type and GRUB as the bootloader. It supports Debian bullseye, bookworm and trixie. + +### "Prefetch" mode for base-apt + +Originally, `base-apt` repo is created only during second build when variable +ISAR_USE_CACHED_BASE_REPO is set. The repo is populated with every package that +took part in the first build and was cached in DL_DIR. + +New ISAR_PREFETCH_BASE_APT variable changes the way `base-apt` is populated. +Packages added to the repo before running any task that need them. Separate +`debrepo` script is used for populating base-apt repo. diff --git a/doc/user_manual.md b/doc/user_manual.md index dcc3f560..aa2b68b9 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -86,6 +86,7 @@ apt install \ gettext-base \ git \ python3 \ + python3-apt \ quilt \ qemu-user-static \ reprepro \ diff --git a/meta/classes/debrepo.bbclass b/meta/classes/debrepo.bbclass new file mode 100644 index 00000000..b4db9219 --- /dev/null +++ b/meta/classes/debrepo.bbclass @@ -0,0 +1,90 @@ +# This software is a part of Isar. +# Copyright (C) 2026 ilbers GmbH +# +# SPDX-License-Identifier: MIT + +# Prefetch to base-apt repo by default +ISAR_PREFETCH_BASE_APT ??= "1" + +DEBREPO_WORKDIR ??= "${DEBREPO_TARGET_DIR}" + +debrepo_update_apt_source_list() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + + chroot_dir=${1} + apt_list=${2} + + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + sudo -E chroot ${chroot_dir} /usr/bin/apt-get update \ + -o Dir::Etc::SourceList=\"sources.list.d/${apt_list}.list\" \ + -o Dir::Etc::SourceParts=\"-\" \ + -o APT::Get::List-Cleanup=\"0\" + " +} + +debrepo_add_packages() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return + + args="" + if [ "${1}" = "--srcmode" ]; then + args="${args} --srcmode" + shift + fi + + if [ "${1}" = "--isarapt" ]; then + args="${args} --extrarepo=${REPO_ISAR_DIR}/${DISTRO}" + shift + fi + + workdir="${1}" + args="${args} ${2}" + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + else + export GNUPGHOME="${WORKDIR}/gpghome" + fi + + ${SCRIPTSDIR}/debrepo \ + --workdir="${workdir}" \ + ${args} +} + +debrepo_parse_dscfile() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return + + dscfile="${1}" + args="" + + build_arch=${DISTRO_ARCH} + if [ "${ISAR_CROSS_COMPILE}" = "1" ]; then + build_arch=${HOST_ARCH} + fi + if [ "${PACKAGE_ARCH}" != "${build_arch}" ]; then + args="--crossbuild \ + crossbuild-essential-${PACKAGE_ARCH}:${build_arch} \ + dose-distcheck:${build_arch} \ + libc-dev:${PACKAGE_ARCH} \ + libstdc++-dev:${PACKAGE_ARCH} \ + " + fi + + args="${args} --extrarepo=${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO}" + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + else + export GNUPGHOME="${WORKDIR}/gpghome" + fi + + if [ -n "${DEB_BUILD_PROFILES}" ]; then + export DEB_BUILD_PROFILES="${DEB_BUILD_PROFILES}" + fi + + ${SCRIPTSDIR}/debrepo \ + --workdir="${DEBREPO_WORKDIR}" \ + --dscfile="${dscfile}" \ + ${args} +} diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 0f35a7fe..3f94392d 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -67,6 +67,11 @@ SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${BPN}-${DISTRO}-${MACHINE}" CACHE = "${TMPDIR}/cache" KERNEL_FILE ?= "${@ 'vmlinux' if d.getVar('DISTRO_ARCH') in ['mipsel', 'riscv64', 'arm64'] else 'vmlinuz'}" +# debrepo config +DEBREPO_DIR = "${TOPDIR}/debrepo" +DEBREPO_HOST_DIR = "${DEBREPO_DIR}/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}" +DEBREPO_TARGET_DIR = "${DEBREPO_DIR}/${DISTRO}-${DISTRO_ARCH}" + MACHINEOVERRIDES ?= "${MACHINE}" DISTROOVERRIDES ?= "${DISTRO}" OVERRIDES = "${PACKAGE_ARCH}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${BASE_DISTRO_CODENAME}::${BASE_DISTRO}:${ISAR_CHROOT_MODE}:forcevariable"