From patchwork Thu Mar 14 07:27:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3406 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 14 Mar 2024 08:31:39 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f189.google.com (mail-lj1-f189.google.com [209.85.208.189]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 42E7V33o021574 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:31:03 +0100 Received: by mail-lj1-f189.google.com with SMTP id 38308e7fff4ca-2d46a089baasf5013391fa.3 for ; Thu, 14 Mar 2024 00:31:03 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710401456; cv=pass; d=google.com; s=arc-20160816; b=JP0VhZMBk3Nx3qsOwH9Mv74wgAJKl7S/ErYh6/RtwrSNbpiljpTmUZ+K/YIEEGdocA UOzLjc77gq2sW2uADNDBP/qFQSfsQ+cPdDYX/1QBkQM1OUiCYW/pK2HYn/1oyAgz9Ndf fYKVcbdGROpRW6Y49ZyruE0mfREM2I3sR5BA5hQvdA+XPwzJQIyOzTGe1Yrys6TXdvkS +Vp0WAmsGfeQ8f77PKBGdgCd8Q5BIz3F/fluZXIbP+apBsXuJunAiRqy+J1CA5RrWBXp k31p9LczUyXIP9wl3HQPpOnHRmjX+GkXGM46ccZJR6l7nqx/YXP4v0pSvHkTV1UnCjbR 2rVw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=TAJz8r8DL/KOIr47RzIOfKvkByzzLOlRncw5CVSL3lY=; fh=Ft9TC1xmjPd+3EMSlRTjDHcwi91wluztK98bHOtSaX8=; b=GuzArFaR0RX/jLaAv2TA0kl+BMjTtvzlwmbbAW+kNhGtozxQbkSOy6Nv59HaqyjoY2 X0dbyhxZ25hsq0layJlOYGBwjRXK0H/63egduiaJdN0C/TU9OFe/J9wyPtDyxI9eTJKH JO0O9anAA+KRpFYHFA+INxjkFR3gBXLoPIucRmesjaaQWj537KMQnQyHhUgy4PoSNOIR gTJFaF+QaQ3ibpfghEQK4aEYM6w1MbmnIYlqBEfdj98+xBZH9BP35PppFk6c65EWWUdu O3xYO9BmfHbaFvSVabAMi43QQdG3jku0dAlUO4Kcncan9sckJNlm1kfcrr/rTnuao6tz eynQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1710401456; x=1711006256; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=TAJz8r8DL/KOIr47RzIOfKvkByzzLOlRncw5CVSL3lY=; b=gvAddUn6rBsjAfs4zfaqU0w5TuYrGrfKdnXKwOBu0KTO4qexfn9Un5EXqpsQybNENA LZbnHudZSOos8tPmyxpAD2lch37ZPsoSqDac5BvU/7UNeXROKiYRFi6PfNQKHxojMuVo q+GP2ix3xaxLhab6enF31XKtPimJ+X0XlFeWgWifb5NImKQ+GmOmTuDLR82uN9gcpZS2 FO81pu/+kqD1Zhsas3j5rthmhhHF18DHeIm8UrYMhIvzicKEPrYAXtamB92Yn5s37aiH WhqXDSs7fh8xnAb1CXQvoIA0w9h5HTln+lPL7Nptnl754jz9ThTFNeLZyPpShZvBNwfW GTuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710401456; x=1711006256; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=TAJz8r8DL/KOIr47RzIOfKvkByzzLOlRncw5CVSL3lY=; b=pXYczsnzDPNIFDfO6RxfdsU+W9QOd78W0ZWMhs+S1Bs1YzENZLqR31yOBCBZObUkAm pU6kn3U/F9pXFMZl2YuvZvQnXvQ81iVUJscyN3RgsPPOHWJwY8KErk5F/CHXysvD8u9r e/bX8cMZzCSz6pSOiHoOMxKX4TfDoR4Yv+pgHeNfvBal8119zML+GgZe3wnDSL8WuimR wLaUBl+g8EV55fjT0gsOvokXq2z59W9F19s5SRJ0Ua+4pv7Pjt3dZUthUtynZEr051Kh CQEIVrSS1XiioMWBHexbBrtDyuvK+Z2RPsLFuBX+LaDWZK6w9PXSCbZcBsyfI46GrsAG Ajow== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCVU5QnL2vlmJvP5qe9QFvLs7rfhiGdnIu2Nve9WO730Ce/OOdaDB0E0CDE1/kIW0HrmFaLTuQJ6xgcObS/c42lZs4zIJ1M= X-Gm-Message-State: AOJu0Ywsh/RMv9p2hi0QDjjYuVN+eD0V6LHrRm525yrBhBQNaE9aw5Y0 SgEfoI6qUpj9X+OGwqgLApfGbiCBzEzsIe2FuaRV7Rrrh0wB767v X-Google-Smtp-Source: AGHT+IEGT0Rp6DLawW91UNx437x0oycg0Pp6dzWY3zWsrYTwuX0gU4kzDnCjzj6vVQGRc+Y5V20O/A== X-Received: by 2002:a2e:9a8a:0:b0:2d4:6e5b:3601 with SMTP id p10-20020a2e9a8a000000b002d46e5b3601mr599167lji.14.1710401455806; Thu, 14 Mar 2024 00:30:55 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:19ab:b0:2d4:3db9:66b7 with SMTP id bx43-20020a05651c19ab00b002d43db966b7ls339429ljb.1.-pod-prod-04-eu; Thu, 14 Mar 2024 00:30:53 -0700 (PDT) X-Received: by 2002:a2e:a9a6:0:b0:2d3:2a95:6f0b with SMTP id x38-20020a2ea9a6000000b002d32a956f0bmr658943ljq.12.1710401453303; Thu, 14 Mar 2024 00:30:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1710401453; cv=none; d=google.com; s=arc-20160816; b=cSC/gRludF4MTRnjvX0YT7vDgVuld8SDHjk+YpMbS7UhYLzwgi665gQX8wbKqHbltt 184iOpmV3vB79e8fAxtF0p0Tty9q2P/uE2Qgsi9f9T2WMKrgzdxYffrCfGWYdQ9DcVyv aVz2s7Ex05SHvQVd374Hn9wpBkiVuF0juiMTco20XhPp23E6UkbYmwAXTeohpg3AMgeY uZYImllhsTGGjwbKFDCaS4fs18K1FX42oGpPsjVqcMJyLpfcqkjziXaZ1BwI5haemsLJ 0qGMwpNPUmFmGgLIel61KUiNWwewV1i5PGLEzjBpcmRAztjHBjpvEq869/loKmkXBBF2 GcLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=+jomylU6eahElKSbHJziRBkdNW0tot2Clr2cQQ/gLzI=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=p8pTY1fg6EGbVCfOBd0BpFLboNyoNRF+n7kIHupHJYJcdleNNExvFrpRjmNc0TyGvN YmXCHnzr7vWWnwzqzduUaBxp/Fk4X0oaiAPKR434hcDp6hesSzi38jvi/03VdMGeLda8 teYi0kSmd8MRq4FPyNvbYXFez2pP39u5agBDrPF0gSaguxGw2D4SZg6W2li37vIEYqKi vaDrdvRUEtrbOcOB8lJ1c7pTgypMLbStxdB9k+PmEb/26/qvHzL4vm7AGZDmxTDdr0ex gdui9KDEB/KqzApay7l3/dz3t50Z+dWYq2Ctbc81efRRa3eV7tM484spLeZnMWlECBHR 6Wsg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id z20-20020a2e9b94000000b002d46764b564si6140lji.6.2024.03.14.00.30.53 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Mar 2024 00:30:53 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from hp15.localdomain (mm-209-185-213-37.mogilev.dynamic.pppoe.byfly.by [37.213.185.209]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 42E7Uo9v021482 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:30:52 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v6 02/11] meta: Add debrepo bbclass handling base-apt prefetching Date: Thu, 14 Mar 2024 10:27:19 +0300 Message-ID: <20240314073047.29465-3-ubely@ilbers.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240314073047.29465-1-ubely@ilbers.de> References: <20240314073047.29465-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This class uses 'scripts/debrepo' python script to prefetch given packages or sources to local base-apt repository. Signed-off-by: Uladzimir Bely --- RECIPE-API-CHANGELOG.md | 9 ++++ meta/classes/debrepo.bbclass | 81 ++++++++++++++++++++++++++++++++++++ meta/conf/bitbake.conf | 5 +++ 3 files changed, 95 insertions(+) create mode 100644 meta/classes/debrepo.bbclass diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index 2da7968d..cb466eb7 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -572,3 +572,12 @@ the module build recipe. Remove all uses of the function deb_compat. The functionality was replaced with a dependency to the package debhelper-compat. +### "Prefetch" mode for base-apt + +Originally, `base-apt` repo is created only during second build when variable +ISAR_USE_CACHED_BASE_REPO is set. The repo is populated with every package that +took part in the first build and was cached in DL_DIR. + +New ISAR_PREFETCH_BASE_APT variable changes the way `base-apt` is populated. +Packages added to the repo before running any task that need them. Separate +`debrepo` script is used for populating base-apt repo. diff --git a/meta/classes/debrepo.bbclass b/meta/classes/debrepo.bbclass new file mode 100644 index 00000000..6c135ef2 --- /dev/null +++ b/meta/classes/debrepo.bbclass @@ -0,0 +1,81 @@ +# This software is a part of Isar. +# Copyright (C) 2024 ilbers GmbH +# +# SPDX-License-Identifier: MIT + +DEBREPO_WORKDIR ?= "${DEBREPO_TARGET_DIR}" + +debrepo_update_apt_source_list() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + + chroot_dir=${1} + apt_list=${2} + + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + sudo -E chroot ${chroot_dir} /usr/bin/apt-get update \ + -o Dir::Etc::SourceList=\"sources.list.d/${apt_list}.list\" \ + -o Dir::Etc::SourceParts=\"-\" \ + -o APT::Get::List-Cleanup=\"0\" + " +} + +debrepo_add_packages() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return + + args="" + if [ "${1}" = "--srcmode" ]; then + args="${args} --srcmode" + shift + fi + + if [ "${1}" = "--isarapt" ]; then + args="${args} --extrarepo=${REPO_ISAR_DIR}/${DISTRO}" + shift + fi + + workdir="${1}" + args="${args} ${2}" + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + fi + + flock -x "${workdir}/repo.lock" -c " + ${SCRIPTSDIR}/debrepo \ + --workdir=\"${workdir}\" \ + ${args} + " +} + +debrepo_handle_controlfile() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return + + control_file="${1}" + args="" + + build_arch=${DISTRO_ARCH} + if [ "${ISAR_CROSS_COMPILE}" = "1" ]; then + build_arch=${HOST_ARCH} + fi + if [ "${PACKAGE_ARCH}" != "${build_arch}" ]; then + args="--crossbuild \ + crossbuild-essential-${PACKAGE_ARCH}:${build_arch} \ + dose-distcheck:${build_arch} \ + libc-dev:${PACKAGE_ARCH} \ + libstdc++-dev:${PACKAGE_ARCH} \ + " + fi + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + fi + + flock -x "${DEBREPO_WORKDIR}/repo.lock" -c " + ${SCRIPTSDIR}/debrepo \ + --workdir=\"${DEBREPO_WORKDIR}\" \ + --controlfile=\"${control_file}\" \ + ${args} + " +} diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 91c5c815..1bff8c8e 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -69,6 +69,11 @@ KERNEL_FILE:mipsel ?= "vmlinux" KERNEL_FILE:riscv64 ?= "vmlinux" KERNEL_FILE:arm64 ?= "vmlinux" +# debrepo config +DEBREPO_DIR = "${TOPDIR}/debrepo" +DEBREPO_HOST_DIR = "${DEBREPO_DIR}/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}" +DEBREPO_TARGET_DIR = "${DEBREPO_DIR}/${DISTRO}-${DISTRO_ARCH}" + MACHINEOVERRIDES ?= "${MACHINE}" DISTROOVERRIDES ?= "${DISTRO}" OVERRIDES = "${PACKAGE_ARCH}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${BASE_DISTRO_CODENAME}:forcevariable"