From patchwork Thu Mar 14 07:27:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3404 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 14 Mar 2024 08:31:28 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f55.google.com (mail-wr1-f55.google.com [209.85.221.55]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 42E7V2ku021561 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:31:02 +0100 Received: by mail-wr1-f55.google.com with SMTP id ffacd0b85a97d-33eca5c305asf15928f8f.3 for ; Thu, 14 Mar 2024 00:31:02 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710401456; cv=pass; d=google.com; s=arc-20160816; b=KnVPfkldw3138s4cwhqmtZ4aX8Jvu+2rmf9vYQF72p8igs87M623hATDPqLyxA/oji AnDF0S55ol4KrowAQsHst73S5VtZcFtycRu+xeyXiwlNibFzaSwqjkwu8lQvePoSWPn2 RHIKybJBQXHdDYcw/L0adYQ9cy1kUiCLSycgR4J1pbekm2vgt6OK9GBLnnyElvSQT4Ui 1FyxBiQdv1yDSolXzgqopXtj3DkT3RHU4REeXOUUeIM1OYyy9HjnnUqaAeevlavaMCqP Ong1OdKOyyngQgWdXDiljBdJ9wL+KuFAFLSjyz4Q13Yp+OTyWI18hQUqOxXHNdu2wDdh /WQA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=P9QKlBLKzqFuvZd0HAs+OI0qtD7dAteP2Lx09jQwyu4=; fh=h78iQD1B22Ky3tmFpqSOP4nWJqssUS12lSS9bJbF2Rg=; b=swc6l74dHN1gdLDmCEJBt23zt/yw1eauXFb/Dq4wAWubS4xxIXfg0WDeXz7QFQK70i OKGKL4RnHRgpzTlRB/7E8SrqFvZ48wcjJLTyClqWZoqM13u2UrMKO5iHSC7v4GXd0P3x ytohP3DZ/gitzNm37qZWOmmOzq2tZ+MTPgD4VeXUpewP+ESqfhzd9Wd/8bG5fHvcyhBg 0YCIH3Utp4eNizzKa7WUcu2MOHLcOj9Lo7uYoF+SS2q1YuM1O7Ii51Uch1Bs1lwgAoTZ Pjs0m8rwBYH2EBmXBWQYA9jl83B3ebo3FRhPSTafLqWXW06yIwuMJgRCpf/aECkbBlip QNpw==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1710401456; x=1711006256; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=P9QKlBLKzqFuvZd0HAs+OI0qtD7dAteP2Lx09jQwyu4=; b=Eis5pHZHqgDP1h35bwILbRdUlktn/OrziWBX7u46WVu+7YCIY3kSUAA68f2KkU8V14 isluipIM1ASbiUyZKPQ+xyuff/shu8RSAUmlg/FT+pC9+vXPDuiwZvr5uQLLcAL7LQYy /Ghh2JbQpQwGO+0UFxvsk4ep7aOjJqN19odTcuUsNVcHXEkCX6/TUUqYRlsVPQmKFOaW njpS1kqTaJsPWb8f9sWDvulrHntwYkbjw8OuBb8JX523XqdZOFfvpR8dT1li5xediXVS 1x8uQHoTFxoc6epNcuxFkbCYpyisiwYsESKtPimJnmixXCur9FufWYUC7StK0JhUf9rN ZZGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710401456; x=1711006256; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=P9QKlBLKzqFuvZd0HAs+OI0qtD7dAteP2Lx09jQwyu4=; b=KC7jCEGf5et0xxuyKd/rzFhiS1UyZ76YdBBpYJcTsI8YcnGZuBXC1TrwXoJojrsxUr TETh+6FvUGOVR9vTSBwtl2k8etf5j2E1rPK+YdN0vLKS9vzNESUs+qFHBLlevN6T4rTz VLrHTdTPClbgP0LZtHtDHgaD8tqgKVlY7mziPKnuitOMCWgurwSJOXx1W/l6LXJw56tJ +wRKq7/O8KU0Q3dhbwDIEPEsaHu3evkZPp5g5cZGsKOvAiXrqTKtSKcCDTvvqPfdZJvT B0zBtSCrAqQlnDSIcWBgeZMqje7c3ksHF3k88g+4EbRw/FmUvkVDmAClD07XncFVHfmw aVHg== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCWkJY2wTWE+SkANhw6E7jlhjkw1OrzpNZ5BSRzFj5KH1rKkm65aLZJ9BZAOACBQl3SXjCdNIxFYFhKeKPBorH3dqnqgwMg= X-Gm-Message-State: AOJu0YxxMzMy8Olgj6Lneh9QAIb5oA9YfIcYjhCkgK3WGCOzrMPhCrPR 08SzBLmBviZICtJsJjNPE950+meJIIZ76rZAIpT/6UWjvNwUDvSz X-Google-Smtp-Source: AGHT+IHg4garzhqol7AV8zxMorTs3UpF4JvgsYBSlSkm8eV7XVShOo4LsYumogUjF7dNL9Pxle5DMA== X-Received: by 2002:adf:a2d6:0:b0:33e:8fb4:f3e9 with SMTP id t22-20020adfa2d6000000b0033e8fb4f3e9mr543714wra.29.1710401455921; Thu, 14 Mar 2024 00:30:55 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:5010:0:b0:33d:d932:8013 with SMTP id e16-20020a5d5010000000b0033dd9328013ls193722wrt.2.-pod-prod-06-eu; Thu, 14 Mar 2024 00:30:54 -0700 (PDT) X-Received: by 2002:a05:6000:194f:b0:33d:24af:9153 with SMTP id e15-20020a056000194f00b0033d24af9153mr594801wry.20.1710401453860; Thu, 14 Mar 2024 00:30:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1710401453; cv=none; d=google.com; s=arc-20160816; b=unlBWwmFYsHKbV+NXOkhnG6dBQQgBImIXwVab72u/4DPsRrpySXOBZgddLZaHTtHUh jaj9fWyVjWU7vFaUXBJBgpGhBHfyV8DT0ChaOQHE2S7SfbTtkZlIX879nCoKEGAkxRKw BV/82/3LmklhRAGHN/9xrtJ9fKo33dJknJqYHnoQIykcwe+CkzP+UjgJvLXiyIuw/2RH MlJbN4A0g8OiZwtEupgcf9GeN+DCei0AtnQaxJlUYn5xMhH0xp89f8RRpzawe0yswC8y vVdEW8+r7qlRkiPleQPK/gUUHpbFHCcECK39gEdGXumFhybLwNYzhkTW19teV4uqRWqp alQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=WcMURk/hHZweb+xItjOlrMp56HyxacIQjJkxR3LKqgg=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=sHY2B4uMI3Lg0iz6w3zj57xb2FpZEOz/y5MSGRjDCBlwimfudiqtIblFHQmyv0e3rW qjjM2WT5a6G8rnOrBUNILiXmw4wN5qaj4bcQOzTMIfnrOBlrun5S15tqF2hhWWh7wciP +a32fZy8zYItOwXQN558J4hK/+zNQoMkzxt/FcrkdPUSl+ROXBg3DV0X4BfXqmdFx508 BSU2WwojuwE9b/xpkwOOtyYo3bp5nIvHh8D3PtkJpx4FcrPzAsTMng6aZZBaUQfboYQH Wrck2JBTFu6Vg8/09DRisikh/WU90/ebdiBiMxOrvjWyrXGFoPKo4cImdZYmqXPjqtU3 XJpQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id l4-20020a5d4804000000b0033da1913960si3126wrq.7.2024.03.14.00.30.53 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Mar 2024 00:30:53 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from hp15.localdomain (mm-209-185-213-37.mogilev.dynamic.pppoe.byfly.by [37.213.185.209]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 42E7Uo9x021482 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:30:53 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v6 04/11] meta: Use cached base-apt repo to debootstrap Date: Thu, 14 Mar 2024 10:27:21 +0300 Message-ID: <20240314073047.29465-5-ubely@ilbers.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240314073047.29465-1-ubely@ilbers.de> References: <20240314073047.29465-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch makes local base-apt repo to be created before debootstrap task. So, debootstrap is then done from it. The required packages are downloaded via python-apt and reprepro creates debian-like repository from .deb files. For debian targets host keyring is used while ubuntu/raspbian targets use keys specified by DISTRO_BOOTSTRAP_KEYS variable. The goal is have workable base-apt repo before first build completed. Signed-off-by: Uladzimir Bely --- .../isar-bootstrap/isar-bootstrap.inc | 94 +++++++++++++++++-- 1 file changed, 86 insertions(+), 8 deletions(-) diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 05c80308..e5d2523d 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -29,6 +29,13 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates" DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else ''}" BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}" BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}" +BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO_ARCH')}" + +# For newer distros "usr-is-merged" indirectly required by debootstrap +DISTRO_BOOTSTRAP_BASE_PACKAGES:append:bookworm = ",usr-is-merged" +DISTRO_BOOTSTRAP_BASE_PACKAGES:append:sid = ",usr-is-merged" +DISTRO_BOOTSTRAP_BASE_PACKAGES:append:sid-ports = ",usr-is-merged" + FILESEXTRAPATHS:append = ":${BBPATH}" inherit deb-dl-dir @@ -275,12 +282,63 @@ do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}" inherit compat +inherit debrepo + +debrepo_bootstrap_prepare() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return + + debrepo_args="" + if [ "${BASE_DISTRO}" != "debian" ]; then + if [ "${BASE_DISTRO}" != "raspbian" ] && [ "${BASE_DISTRO}" != "raspios" ] || [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then + debrepo_args="$debrepo_args --keydir=${WORKDIR}" + fi + else + if [ "${BASE_DISTRO_CODENAME}" = "sid" ]; then + debrepo_args="$debrepo_args --keydir=${WORKDIR}" + fi + fi + if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then + debrepo_args="$debrepo_args --compatarch=${COMPAT_DISTRO_ARCH}" + fi + + debrepo_workdir=${DEBREPO_TARGET_DIR} + if [ "${BOOTSTRAP_FOR_HOST}" = "1" ]; then + debrepo_args="$debrepo_args --crossarch=${DISTRO_ARCH}" + if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then + debrepo_workdir=${DEBREPO_HOST_DIR} + fi + fi + mkdir -p "${debrepo_workdir}" + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + fi + + flock -x "${debrepo_workdir}/repo.lock" -c " + ${SCRIPTSDIR}/debrepo --init \ + --workdir=\"${debrepo_workdir}\" \ + --aptsrcsfile=\"${APTSRCS_INIT}\" \ + --repodir=\"${REPO_BASE_DIR}\" \ + --repodbdir=\"${REPO_BASE_DB_DIR}\" \ + --mirror=\"${@get_distro_source(d)}\" \ + --arch=\"${BOOTSTRAP_DISTRO_ARCH}\" \ + --distro=\"${BOOTSTRAP_BASE_DISTRO}\" \ + --codename=\"${BASE_DISTRO_CODENAME}\" \ + ${debrepo_args} \ + ${DISTRO_BOOTSTRAP_BASE_PACKAGES} + " +} + do_bootstrap() { if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then if [ -z "${COMPAT_DISTRO_ARCH}" ]; then bbfatal "${DISTRO_ARCH} does not have a compat arch" fi fi + + debrepo_bootstrap_prepare + debootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}" if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}" @@ -302,8 +360,19 @@ do_bootstrap() { sudo -E -s <<'EOSUDO' set -e if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then - arch_param="--arch=${DISTRO_ARCH}" + arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH}" fi + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + ${DEBOOTSTRAP} $debootstrap_args \ + $arch_param \ + ${@get_distro_components_argument(d)} \ + ${@get_distro_suite(d)} \ + ${ROOTFSDIR} \ + file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO} \ + ${DISTRO_DEBOOTSTRAP_SCRIPT} + " + else ${DEBOOTSTRAP} $debootstrap_args \ $arch_param \ ${@get_distro_components_argument(d)} \ @@ -311,7 +380,7 @@ do_bootstrap() { "${ROOTFSDIR}" \ "${@get_distro_source(d)}" \ ${DISTRO_DEBOOTSTRAP_SCRIPT} - + fi # Install apt config mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d" install -v -m644 "${APTPREFS}" \ @@ -383,12 +452,21 @@ do_bootstrap() { if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH} fi - - chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \ - -o APT::Update::Error-Mode=any - chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f - chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ - -o Debug::pkgProblemResolver=yes + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \ + -o APT::Update::Error-Mode=any + chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f + chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \ + -o Debug::pkgProblemResolver=yes + " + else + chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \ + -o APT::Update::Error-Mode=any + chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f + chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \ + -o Debug::pkgProblemResolver=yes + fi umount -l "${ROOTFSDIR}/dev/shm" umount -l "${ROOTFSDIR}/dev/pts"