From patchwork Thu Jul 25 15:07:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3726 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 25 Jul 2024 17:10:27 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f57.google.com (mail-lf1-f57.google.com [209.85.167.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46PFAQmf007170 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:26 +0200 Received: by mail-lf1-f57.google.com with SMTP id 2adb3069b0e04-52ebdbf8a7csf205700e87.2 for ; Thu, 25 Jul 2024 08:10:26 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721920218; cv=pass; d=google.com; s=arc-20160816; b=Eo3EjXQwdC9zKrX2vXZXEBD4qgRu2WbCG2zwp54YU/O5lh7sITYMtdAbY7gnnzIY/M W1wqRMp9MWbJs1iDIB/FM165ftv4kMQUgvjYhxwieaWE86d5xSmotlalZvkcmNrce8e9 58lBhc/ZcxxFnlm1WKGaYRJJ0OvVKe7aqWNjQOGkbNXuTa9wbOjRd7zemw2+4WhzcSVt 1GKj04tPpyZ6+Bfmrr2FDsnxW4fhAtIAm6nA3miWIl5GDpTYzkjVeldIcflwELQw51Lr twF6EVoyjL+eZBMBFjPjbUiSNVrXEOAHPEn2QUFNrjCjBlmqGgPqDUwSq2g8PnRdfvOM Q1wQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=kjHb1SsEmaAkhbpS61GtNV9Z8+2/XkiOwwIvE8/peOE=; fh=i+pZP045BfjZjxy06VCz4ovKBxJHkPlJyRtbRU2muWo=; b=QpR0yrHuGVsqMXmhH53PEGShv7fgjKgJAfNpZUMH7GUtI0sCdLuahO4z3K7Jw5Dfy+ 7GlpXos/YkE6elAcV6dRfpUcl6XVBB4n60QtKXAL2fm+5g+YdtVdwxRLLn6UOYZLZCoY CeTC7Av7dIB+XYpNi97tI6ryl51+TWUX2qN1ESlH+MjZbmHyOjXyAF9F8njHM03gXoRP 0W9qnJFye5vCWD4jL/sdATMqlRf0RmrPBDMryICKI5FjPXRyLG1rb0SIvUfBMgnT8tIY wS3aU5m+lTB3M1L1OlQWBq56MbZmx9S3FWJ7A3SKt/Wt5mxwH5rkCwzS+7Ft28QkhD+w eIFg==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721920218; x=1722525018; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=kjHb1SsEmaAkhbpS61GtNV9Z8+2/XkiOwwIvE8/peOE=; b=nRuoIdOKa+UNNpTxKgbJTK23QngKGbbswWP4EwY5Ji8Bgd6XCyTljwF6bpkaZv9rdS T4OcexYFm2NBaZFlS+IpuvR9YJRdHkomLlTxxxqDND59sY0X6KP2MnPQQay8A8TTmoj2 2E8p7N4WVJgYLCKQwQ4epIcLSGQr2PK23nCtQ1Xd6eCQRWUyNzGlRtP73q/g4+1XCsn5 BMe7i9QfxH3We/e8qhaojAR2o7jFQ0VAPy8SkIUc/9zkrOUbYBiwOzHexztvuqVh71cK +JEwb/Y03dXCkw9T8MalyHxFGSf4ChCUUzNfhQYWnBiWDurtYd+eV1lWCCdpmIv4/4Qb s3eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721920218; x=1722525018; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=kjHb1SsEmaAkhbpS61GtNV9Z8+2/XkiOwwIvE8/peOE=; b=AQ1bJIs92P9rAAUJDGDhraVyQ4PyGWfcfz/G2RPF6uZvWXKdlQ9u323YyAZrW7swRh SZ1zCB45N1xFrW8mhM4paM0EgMqKyggWZuawAOXeCjzzRyE3vp1WBLjLOwe3axHdDbSf 8O3cD89XmjaMWZExapPNniXbHsmsReu9HRtRDwrT40Meph/5LABS7K+rUvdT0//WvS7p EhGtJaUP64I6kVgwNlh4paJZ6eLgbZ9VkuPAafMpXonHW5V+3GaHZTQUZg+paAqQPBOH OkRpIx66FspuF0G170cK+fDLfEIRaYluFz45MHD2h4qIFKxBng1z6Vmop1anrKZveaBI bYXg== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCVKaRJHXfvF+dBY1i0Fru4t6FyyPJ8jRsWDkT7640BYpaoK+3h6QzkLPk51YERm3P4LXvKN8UMCXXdDrSslpAiyMkKhVok= X-Gm-Message-State: AOJu0YxFSiMfLjrJo8tRSCil8AcGZEdRWfsZX4XfSNMhAjhLez4YBa4s GJOQTOKFxZLLH9Qv8yNAxsN6DGO2EYjXt1FqT+tBkpTxReo+fSta X-Google-Smtp-Source: AGHT+IFt0vahHzvqy11p890omF8yMvl80se2xu3D5s34krsLH0cJKLdquK1OazL4mS+h/guTsRn+4Q== X-Received: by 2002:ac2:568f:0:b0:52d:6663:5cbe with SMTP id 2adb3069b0e04-52fd60189b8mr1814488e87.12.1721920217809; Thu, 25 Jul 2024 08:10:17 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:4e03:0:b0:52e:9d2f:234 with SMTP id 2adb3069b0e04-52fd3f54566ls614854e87.0.-pod-prod-06-eu; Thu, 25 Jul 2024 08:10:15 -0700 (PDT) X-Received: by 2002:ac2:4f07:0:b0:52e:7448:e137 with SMTP id 2adb3069b0e04-52fd60189acmr1814385e87.6.1721920215460; Thu, 25 Jul 2024 08:10:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721920215; cv=none; d=google.com; s=arc-20160816; b=fofXg1/gQh3Qbg8jn3ViQ5A3z5ukR3bfMQiCd1y6CKDP8MxuCKZzJVncsyX+xzCQ5p 6I7X1Uy6K/oC1RV5pz7LoF6VwsiWK3LXpEAR5jSzCaBk8oDtNL+g6NEXhpkpU/8jeKZ4 mWmWSJ0zyV47xebddNEor5akYHl+ftiUVNTCHYsTBGfhTXvLL06fcYf6ZPFrLEpQNVZC ffwm4c/m1aqk8+yrlif+ku3XXOG6JvSxonO+x69+3bPVQ0EiKsdadKR7KTBL5+a4i7N9 FDIv/HQ0f8vMlujU2rar3I/7IrSTTFMiUQew1uyQ0oijEw41k1s9efzzpTCr5L1alNhy Iutw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=KKhUWQuFn3XJxtbskh8Qfp/nrLPLpvZsTNdE5cSLSWk=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=xlh2qZ3W0YU52VBoOqkttNqwXl3ACI2UjkhAPOk6Q1378j2kcW5SKowEzdnH7WZFHU I/KGS1tljj+XTf7vJ7FSGvE23vavcgxeqqFhOJYYYFqkeZ1ZOlUZrNg+BwvfOyLLEozk 0E9cNDH+VaczDa/eyC/Va7GXIODGbSBV+Pf8Z+9sFEfEXpWXq5ekoCeargt9rPk4xDWG K2LdZHAe4rX3OTLu38ePRLPUBznuPl/qKDxPiLk8TtVgbI/Gh2PkuMegQ9bYlUcLrBGD A77bfhc+JyFSL83FA22cam5gODEuAtlkAP35gtfGnTicnk+SyPu+4dUvYaUgPSwFQY/Q 9v6g==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 2adb3069b0e04-52fd5b8b998si31212e87.2.2024.07.25.08.10.15 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 25 Jul 2024 08:10:15 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 46PFACKa007063 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:14 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v7 04/10] meta: Use cached base-apt repo to debootstrap Date: Thu, 25 Jul 2024 18:07:36 +0300 Message-ID: <20240725151006.2129-5-ubely@ilbers.de> X-Mailer: git-send-email 2.44.2 In-Reply-To: <20240725151006.2129-1-ubely@ilbers.de> References: <20240725151006.2129-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch makes local base-apt repo to be created before debootstrap task. So, debootstrap is then done from it. The required packages are downloaded via python-apt and reprepro creates debian-like repository from .deb files. For debian targets host keyring is used while ubuntu/raspbian targets use keys specified by DISTRO_BOOTSTRAP_KEYS variable. The goal is have workable base-apt repo before first build completed. Signed-off-by: Uladzimir Bely --- .../isar-bootstrap/isar-bootstrap-host.bb | 2 + .../isar-bootstrap/isar-bootstrap.inc | 87 +++++++++++++++++-- 2 files changed, 81 insertions(+), 8 deletions(-) diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb index 4f90fd01..1ace818f 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb @@ -15,3 +15,5 @@ require isar-bootstrap.inc HOST_DISTRO_BOOTSTRAP_KEYS ?= "" DISTRO_BOOTSTRAP_KEYS = "${HOST_DISTRO_BOOTSTRAP_KEYS}" + +DEBREPO_WORKDIR = "${DEBREPO_HOST_DIR}" diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 9fd928ad..589fa2c1 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -33,6 +33,13 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates" DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else ''}" BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}" BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}" +BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO_ARCH')}" + +# For newer distros "usr-is-merged" indirectly required by debootstrap +DISTRO_BOOTSTRAP_BASE_PACKAGES:append:bookworm = ",usr-is-merged" +DISTRO_BOOTSTRAP_BASE_PACKAGES:append:sid = ",usr-is-merged" +DISTRO_BOOTSTRAP_BASE_PACKAGES:append:sid-ports = ",usr-is-merged" + FILESEXTRAPATHS:append = ":${BBPATH}" APT_SNAPSHOT_DATE = "${@ get_apt_snapshot_date(d)}" @@ -291,12 +298,56 @@ do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}" inherit compat +inherit debrepo + +debrepo_bootstrap_prepare() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return + + debrepo_args="" + if [ "${BASE_DISTRO}" != "debian" ]; then + if [ "${BASE_DISTRO}" != "raspbian" ] && [ "${BASE_DISTRO}" != "raspios" ] || [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then + debrepo_args="$debrepo_args --keydir=${WORKDIR}" + fi + else + if [ "${BASE_DISTRO_CODENAME}" = "sid" ]; then + debrepo_args="$debrepo_args --keydir=${WORKDIR}" + fi + fi + if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then + debrepo_args="$debrepo_args --compatarch=${COMPAT_DISTRO_ARCH}" + fi + + if [ "${BOOTSTRAP_FOR_HOST}" = "1" ]; then + debrepo_args="$debrepo_args --crossarch=${DISTRO_ARCH}" + fi + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + fi + + ${SCRIPTSDIR}/debrepo --init \ + --workdir="${DEBREPO_WORKDIR}" \ + --aptsrcsfile="${APTSRCS_INIT}" \ + --repodir="${REPO_BASE_DIR}" \ + --repodbdir="${REPO_BASE_DB_DIR}" \ + --mirror="${@get_distro_source(d)}" \ + --arch="${BOOTSTRAP_DISTRO_ARCH}" \ + --distro="${BOOTSTRAP_BASE_DISTRO}" \ + --codename="${BASE_DISTRO_CODENAME}" \ + ${debrepo_args} \ + ${DISTRO_BOOTSTRAP_BASE_PACKAGES} +} + do_bootstrap() { if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then if [ -z "${COMPAT_DISTRO_ARCH}" ]; then bbfatal "${DISTRO_ARCH} does not have a compat arch" fi fi + + debrepo_bootstrap_prepare + debootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}" if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}" @@ -314,8 +365,19 @@ do_bootstrap() { sudo -E -s <<'EOSUDO' set -e if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then - arch_param="--arch=${DISTRO_ARCH}" + arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH}" fi + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + ${DEBOOTSTRAP} $debootstrap_args \ + $arch_param \ + ${@get_distro_components_argument(d)} \ + ${@get_distro_suite(d)} \ + ${ROOTFSDIR} \ + file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO} \ + ${DISTRO_DEBOOTSTRAP_SCRIPT} + " + else ${DEBOOTSTRAP} $debootstrap_args \ $arch_param \ ${@get_distro_components_argument(d)} \ @@ -323,7 +385,7 @@ do_bootstrap() { "${ROOTFSDIR}" \ "${@get_distro_source(d)}" \ ${DISTRO_DEBOOTSTRAP_SCRIPT} - + fi # Install apt config mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d" install -v -m644 "${APTPREFS}" \ @@ -395,12 +457,21 @@ do_bootstrap() { if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH} fi - - chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \ - -o APT::Update::Error-Mode=any - chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f - chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ - -o Debug::pkgProblemResolver=yes + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \ + -o APT::Update::Error-Mode=any + chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f + chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \ + -o Debug::pkgProblemResolver=yes + " + else + chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \ + -o APT::Update::Error-Mode=any + chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f + chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \ + -o Debug::pkgProblemResolver=yes + fi umount "${ROOTFSDIR}/dev/shm" umount "${ROOTFSDIR}/dev/pts"