From patchwork Fri May 26 07:00:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 2843 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 26 May 2023 09:00:45 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-oo1-f59.google.com (mail-oo1-f59.google.com [209.85.161.59]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 34Q70gpo013374 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 26 May 2023 09:00:42 +0200 Received: by mail-oo1-f59.google.com with SMTP id 006d021491bc7-54f6b5c7d1bsf311360eaf.2; Fri, 26 May 2023 00:00:42 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1685084436; cv=pass; d=google.com; s=arc-20160816; b=EkD427Z+eRca+qg6HQzgiMhLNmBmDPioWhPBEjOWg3sgLlazvVqoEOjU2QCv5Scb6t AsE1wMLO09f3x54NvFxSPqql7AWvVtX3r6KnBBfcc7RScIvAIWATDai/FonsJ30XdaPC Py++NlG74RZLp2BsqX6JfosE4/yuGo21RmwvjXk/U26P4XKyxsRv9Vfl3GFmM9OB5CHl 9gfXOnogqbXW1o8OYFy28nPPt+Biu8jyAcv45LCn8xpH8B8RuIyqpgoHvEN8kop5nYrP q2ciToQ6fKSSNKULiBco2knPheJ/4aARNnG0EcxC/LPifhF+DHFRBEkqLbkchbNatprU g6WQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=T8SurfF0iaXdGfjW2sVG4WMlP1SmIWBTqOGmzUgaaN4=; b=gu1FCUVXKzkZtUSK8bXpFYq77h6w2mGmM+TLg7bJ3cyhui+J3XyCAveXJ1wGkWkLFB RbT1T7ZcJ7tcMZoLyqQnu9ysDUdSakJgezGMrVQWed9bhFoFSEmhQYYRwTrAZOvV68YY vW0hLPgu7+arjXcY5Jj9ehB2d34K5CVxWXLovsvg5eOZ9YxRiLyeGJ18w7kzIA8hkeuE ClmQTDJKQF57NpbIn5IRg5JMRC1ghY6ItMTq6YYzxKKGl3MFwgbqay261b5+9R3JWBOM SqOWtunCbjqRAQhZGnEJwIiUdRkOdW2AeUnDYCeOfLZfc2ZubD0hxVm7PhpS/LhHfkRq gC8Q== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20221208; t=1685084436; x=1687676436; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=T8SurfF0iaXdGfjW2sVG4WMlP1SmIWBTqOGmzUgaaN4=; b=TC7be3bhh1AxCbK122Ze4pqrTbKWKCQmPtWPL5zLLZMvFObPJwPFc1f5me3ZMTgwLa ttqURgfKzrJY6iXedy1rkVrILDxe6319Hx4OWWQNsKl/wmeUENf3XtxRFOpy7geiIfaL cCfl3x1woOSdC2iZzVHLGMJgt2fYLChRcNcqZkIA5TueO4qklsEzqKU/icBOtVKW6lAx PVT0L78phlB6/ClWefiMzacedUPkXL4FoPStRNBwS5J5LVe3Qv9S7uNwek09TIMG8ZZr LAakrUiYSLchYnN1p6rOpuSqnC982LiOYgGTvQwwVmvvSi82oXf6BdbeYV5B+qPU8NgG Q2Rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685084436; x=1687676436; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=T8SurfF0iaXdGfjW2sVG4WMlP1SmIWBTqOGmzUgaaN4=; b=R1yeQURrcJaVC8qlFoDpbJwkOGDVgMrt2fB8tPzvN9RX8vRAJElb3ImzAAvJQuaJ9n gMEXdGGiB3uxxVAY+E66Kf2YdCRfk3jJccCXp//fX8z5BSc/j9bdtkbJoqCaszCyNXkt wR7jM+KSZ4HPGX1noo9whaR6+DSR0RTXVhm5nES7J1qKBY6x2Nb5QTdmuC7paeja3hgF FGsLqlonzAlyCJ1+C14KG6sFvEeeMb1jvvZqoOS4eeyCOR8oJ8UjtbiXqoqQGqA00sPT MwpggR317X63z+QBOtaDLtnzcwvqFH0Tdc/VzQZne5d6WvkqAMZ/jC08TDEXie+/HtLc 5gHg== Sender: isar-users@googlegroups.com X-Gm-Message-State: AC+VfDxdfohgZyuUqDbsjgySpC+GIsgG5cp7c5GdlaUuFGarF+kiv+/S 2LBrBvKcSqWLsSueW+Lz1Jc= X-Google-Smtp-Source: ACHHUZ6+ezgQ2jAmReYwVTbsy2aYBZeUkayshYyFStl0yBZyRGJSx5UwhljS/B4zALeZ9Iwm95YeiQ== X-Received: by 2002:a4a:5243:0:b0:54f:8345:1832 with SMTP id d64-20020a4a5243000000b0054f83451832mr375129oob.1.1685084436555; Fri, 26 May 2023 00:00:36 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6870:7711:b0:195:d294:2d3d with SMTP id dw17-20020a056870771100b00195d2942d3dls205547oab.0.-pod-prod-09-us; Fri, 26 May 2023 00:00:36 -0700 (PDT) X-Received: by 2002:a05:6870:b211:b0:199:ccad:7dcc with SMTP id a17-20020a056870b21100b00199ccad7dccmr275588oam.48.1685084436037; Fri, 26 May 2023 00:00:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685084436; cv=none; d=google.com; s=arc-20160816; b=HO09c+c0Tb3L9W+lubj1M1hfvcr0R/aQKrm0QDgTl6PN+NQiwNS3yqbSCFXDkAjorV s+/bEi4RGqYcmFyAy0ZcNEneeCxM24yvZr/HQ/t1aeqhvm3VvrmhYllB9I7l/zrHTdvR CKEWgy7ivTL0Wc8VrM8Am1tfbMPRrzO7cICt8qbJg32WRGzYeT+ekBJE8GYY1ChQxjmO FQYapX+zNA7CvRMp0+2NpFkfd1SMwzrBZhxfmUQ7yllPVZdpHSOQWCw/dfmfe7U1Q0eW cUCeTVTjcaVM29TO5Wtxq70ngzv85hGjZwLE4THzApgduG6Bf1zs9lsK3YjP1o8nntqX eoEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=wk/RA20AwaT5IYEiG7iqhNXKF2d4XIehMTXhYvEThPc=; b=xpspfZMt/i5JKmHuQyD6kjVHguZoAxd50XggauN0UaeWqKywiNckW424IwcsKNKB6X 1e6sHDZDt8M+7KE4SRgvKs66DO7r+5QJfBVp5lL9ryPDk9gb40ieElBuW2yrEWW4ENIL 1fNKaq57sHBajfaD3QoxtkB5xO5d9XqjHJAXkeUzF+7Jqlqeo+XT3nzb79PEeUj2BMdN olpiOCdTpYucAnAvpuEnR8HEwsxTI+3RC81X78EnICBEP6dLbWEVkSnKkQMNI0sLl+78 h0PvJC+2GZnL6cf/smfuozg8p64XwIk7KNVg6USgCg5OrYCurT0MqQ/VY9srLkXsKD3M CITw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id d15-20020a4ad34f000000b005552377a831si256498oos.1.2023.05.26.00.00.34 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 26 May 2023 00:00:36 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from baighyz.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 34Q70R3a013306 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 26 May 2023 09:00:30 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v5 06/12] meta: Use cached base-apt repo to debootstrap Date: Fri, 26 May 2023 09:00:21 +0200 Message-Id: <20230526070027.16890-7-ubely@ilbers.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20230526070027.16890-1-ubely@ilbers.de> References: <20230526070027.16890-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch makes local base-apt repo to be created before debootstrap task. So, debootstrap is then done from it. The required packages are downloaded via python-apt and reprepro creates debian-like repository from .deb files. For debian targets host keyring is used while ubuntu/raspbian targets use keys specified by DISTRO_BOOTSTRAP_KEYS variable. The goal is have workable base-apt repo before first build completed. Signed-off-by: Uladzimir Bely --- .../isar-bootstrap/isar-bootstrap.inc | 99 +++++++++++++++++-- 1 file changed, 91 insertions(+), 8 deletions(-) diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 0cfe82ab..e1ddac42 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -29,6 +29,8 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates" DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}" BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}" +BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO_ARCH')}" + FILESEXTRAPATHS:append = ":${BBPATH}" inherit deb-dl-dir @@ -278,12 +280,73 @@ do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}" inherit compat +inherit debrepo + +debrepo_bootstrap_prepare() { + if [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then + return + fi + + if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then + return + fi + + debrepo_args="" + if [ "${BASE_DISTRO}" != "debian" ]; then + if [ "${BASE_DISTRO}" != "raspbian" ] && [ "${BASE_DISTRO}" != "raspios" ] || [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then + debrepo_args="$debrepo_args --keydir=${WORKDIR}" + fi + else + if [ "${BASE_DISTRO_CODENAME}" = "sid" ]; then + debrepo_args="$debrepo_args --keydir=${WORKDIR}" + fi + fi + #if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then + debrepo_args="$debrepo_args --no-check-gpg" + #fi + if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then + debrepo_args="$debrepo_args --compatarch=${COMPAT_DISTRO_ARCH}" + fi + + debrepo_workdir=${DEBREPO_TARGET_DIR} + if [ "${BOOTSTRAP_FOR_HOST}" = "1" ]; then + debrepo_args="$debrepo_args --crossarch=${DISTRO_ARCH}" + if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then + debrepo_workdir=${DEBREPO_HOST_DIR} + fi + else + debrepo_args="$debrepo_args --isaraptdir=${REPO_ISAR_DIR}/${DISTRO}" + fi + mkdir -p "${DEBREPO_WORKDIR}" + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + fi + + flock -x "${DEBREPO_WORKDIR}/repo.lock" -c " + ${SCRIPTSDIR}/debrepo --init \ + --workdir=\"${debrepo_workdir}\" \ + --aptsrcsfile=\"${APTSRCS_INIT}\" \ + --repodir=\"${REPO_BASE_DIR}\" \ + --repodbdir=\"${REPO_BASE_DB_DIR}\" \ + --mirror=\"${@get_distro_source(d)}\" \ + --arch=\"${BOOTSTRAP_DISTRO_ARCH}\" \ + --distro=\"${BOOTSTRAP_BASE_DISTRO}\" \ + --codename=\"${BASE_DISTRO_CODENAME}\" \ + ${debrepo_args} \ + gnupg locales usr-is-merged + " +} + do_bootstrap() { if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then if [ -z "${COMPAT_DISTRO_ARCH}" ]; then bbfatal "${DISTRO_ARCH} does not have a compat arch" fi fi + + debrepo_bootstrap_prepare + debootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}" if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}" @@ -301,8 +364,19 @@ do_bootstrap() { sudo -E -s <<'EOSUDO' set -e if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then - arch_param="--arch=${DISTRO_ARCH}" + arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH}" fi + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + ${DEBOOTSTRAP} $debootstrap_args \ + $arch_param \ + ${@get_distro_components_argument(d)} \ + ${@get_distro_suite(d)} \ + ${ROOTFSDIR} \ + file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO} \ + ${DISTRO_DEBOOTSTRAP_SCRIPT} + " + else ${DEBOOTSTRAP} $debootstrap_args \ $arch_param \ ${@get_distro_components_argument(d)} \ @@ -310,7 +384,7 @@ do_bootstrap() { "${ROOTFSDIR}" \ "${@get_distro_source(d)}" \ ${DISTRO_DEBOOTSTRAP_SCRIPT} - + fi # Install apt config mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d" install -v -m644 "${APTPREFS}" \ @@ -382,12 +456,21 @@ do_bootstrap() { if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH} fi - - chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \ - -o APT::Update::Error-Mode=any - chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f - chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ - -o Debug::pkgProblemResolver=yes + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \ + -o APT::Update::Error-Mode=any + chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f + chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \ + -o Debug::pkgProblemResolver=yes + " + else + chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \ + -o APT::Update::Error-Mode=any + chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f + chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \ + -o Debug::pkgProblemResolver=yes + fi umount -l "${ROOTFSDIR}/dev/shm" umount -l "${ROOTFSDIR}/dev/pts"