From patchwork Thu Jul 25 15:07:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 203 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 25 Jul 2024 17:10:23 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f55.google.com (mail-wm1-f55.google.com [209.85.128.55]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46PFAMmx007116 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:22 +0200 Received: by mail-wm1-f55.google.com with SMTP id 5b1f17b1804b1-426d316a96csf7779425e9.0 for ; Thu, 25 Jul 2024 08:10:22 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721920216; cv=pass; d=google.com; s=arc-20160816; b=pg0XV0hoa3UCLdejLFkX7B7wwhT/F6bezjaxHjLGc3qAP5n+AEf3Z5Jy3eMqLsmv4w SBN8pIcfTqZz0e8dX22dqcbswDVt9BpWusJv1VHfwejWITYJ7dit3ajYZi+FMH6skJ2h p1cg5seGrMwzuy4Puyr2ELgFTRJ3lZQi4QOP6OO5y+v8ooYbHS0BCmuIFKs+ekdKfbha +mQQg6vqrD5vnThPW4MGcfZTDk9l5dzT7pY7iI2Zd/WxjSHK1Sq/EXAVUag2X10euA8L bCwnMNcjuonunHgXQihsdLSpG1zxVtEN2UsWGeCV265UnmS8oEQ3tFxyscHgs08X/V9c Pctg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:to:from:sender:dkim-signature; bh=AXs3NgCqgz2w6/XaW86ksufTXjE80tcT5JEt/SZn0Mk=; fh=FWmMtJIsB46y2XS67yzwWelr7ns1a+NJv6Vu1KQLeag=; b=pS4rXiQUak7ROD2fu0Xqz/SaFnzLOSjSF0L15eP0g/OuoXbKK3xE/D9GjoM120m4w0 MTspR25ztEuWXQxojMUQjp3M5vCnzB7QMIFQtDT5vwjAu5zl8obsB/HH+42ZtA0bLacC wdT4Hqcc+Zvpyz84ScpJFn2AtLZlKU2j2mIcTJEIjxDuvK8/lthgNh7K28cPKm/U7NkW CltzulYOzvB870lbSP9SJdqc7CEDqwlnfLECdcI7tiPDXGJiSTZLOyQH8lFZm75zV/Js 20GyYUmKVt6lAQAkfpiPYOgNExhDL1W3BpcmA4/SrxiJGlTH7Vq1I8JG5bj7G6YH50YQ NVUg==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721920216; x=1722525016; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:message-id:date:subject:to:from :sender:from:to:cc:subject:date:message-id:reply-to; bh=AXs3NgCqgz2w6/XaW86ksufTXjE80tcT5JEt/SZn0Mk=; b=DEuNyD9s6mbQqv54luIkiu+G5fbxM+6KleJuzeHD2EY0Fq8OWBzCyHfdm776LNZJGe D1kx1WM8pw+nIriJbHDoJAjvTOK/MzfL3wSAie/UCNwOMtK6nYeZTBCzW9DLIYK9upYk Rs+0x5qNFGm0s+povVvIcBVqmgOZKupvmVM229EvuaR+VrlaUuXn+aesS04avIUPMP0Z deZ7LZJgNfvf0WicAEpCFzOon8JEu/TAdPZIuKSjkPGH4EAb4A6oMx4cFvbPraCCrv+J k1p2YCbNYsIsF4kwhPLq9v8aO+Q0HfGT1cVK9tJKoW4vuQaRoF4RqsCBMy+Nvn9Q3qei 04YQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721920216; x=1722525016; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:to:from:x-beenthere:x-gm-message-state :sender:from:to:cc:subject:date:message-id:reply-to; bh=AXs3NgCqgz2w6/XaW86ksufTXjE80tcT5JEt/SZn0Mk=; b=IrA7CY5eqVkPcuyW5Y+7BPAykRPz2D1RvdvI8Ny4b2ldYq9EGgA7btbX2m8CbIcnh/ 22GzMBVdBE10oWvDP/TQQAraTAZLOTv+0rjiIA7Es92fi+aW+oPdQqaZUm7y+PAMGcPQ sf0ZKBkZ6OZuSCSAOZ5AqZvK3D6hxFWrf1cJkyO5YyADH81pbgYlDXE5yHYD+WOnyz6g rDpQtg/IpvPtuZS9h407HAF/eSC9kDQru7BLEjcsEy/xSk2Xrm0wUCBpSOcnNRgR6GhA nWUorbXLyIWpP1YopHDy4Hxw/Sw3EcKU2pdId1yji/fFQpyvFgW6U1gyWgodF8GUdS9q x6oA== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCXPWXajVNMvUZc/+xo1aileEEK/Ja3mDEoxxPhEuEDrmN2ZMPlTAow3GWCnwvRMfquOZXYT802Kh/v0c69oYiHajUodRgI= X-Gm-Message-State: AOJu0YwsYR4ZIxDCEwVdf3RMvtPyCihfdIRwc9zpNdPxXueJkMWJoCej L7oIMOOVU9Q7h6pDzioBwhA0zKxCAs9KSkJA1fzpInJ5FtPfbsdI X-Google-Smtp-Source: AGHT+IGXiqh38JdIeFyA5bYVoQkWJeWZm6eS1nlI+R7JFYe6qZItGIJBBxp2I2kP4htOJvO8GvRKKQ== X-Received: by 2002:a05:600c:4f16:b0:426:6ed5:fd5 with SMTP id 5b1f17b1804b1-42806b6be12mr16917455e9.6.1721920215823; Thu, 25 Jul 2024 08:10:15 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:3c86:b0:425:68a1:9da9 with SMTP id 5b1f17b1804b1-42803b6a203ls4663875e9.2.-pod-prod-09-eu; Thu, 25 Jul 2024 08:10:14 -0700 (PDT) X-Received: by 2002:a05:600c:4f16:b0:426:6ed5:fd5 with SMTP id 5b1f17b1804b1-42806b6be12mr16916665e9.6.1721920213754; Thu, 25 Jul 2024 08:10:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721920213; cv=none; d=google.com; s=arc-20160816; b=Ed1loDP0GlSMpoB7uFHcojND9/n7yqaBJJKBOdl4mkiYNyaMRYvwQPsXkHR+j0bzr7 CNJYrPO60HLwj0MHXVOT9moMIHFWLbyi29llaNnHz4jf9MMJEGACcaKA0nSAEtbuR5RZ jgpkbu/TbqDDMdbu8ec/PYH/AlzpOsORSVbLhQgp7H09jDHY/5lmCNetCVo24w6t2LSn a2YBteYp4RnreaEWcivmfdPbp0FzTqmui8gp0lLyHF7CHkkudB5o1PbzNRP1zkVMvPpC Oonn64RdUpzcZir7ilspJvFjPYdS6Hezs/iOSkwBC9ppOBQeizIAUm5Zsix3cGfCztDv SztQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from; bh=1PlBqO2ii0s9YxUVnymyOl1EN7IWbB14UWV4LBMpLHk=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=FNzhbQnfr1SIA7pDycooywzAcsduafQHZqA9VnV8KC0B5oURD4Z3Yl4XzkCHoExPXS j7CWZzFKMqQ0uMf0VNxhi8CrH1CCifVhfyNMTM1PW2TFG4xjXAU0HUliZR3UHUjaoRLJ gfpZltCrTstidQanh+wFrRXos7Cl9VNE7Z5Z1sbwBaP3LJ8Ivps8+j9PHanFt5QgTO8O MrBZ3vonmW4cX+NJL6H8kzw85Es4uVQu4+py05V3H8x2bImKI334y/uShJSQ/cHR6N7v jvgYNIobEQpkD7sxLa+MtbOrMUDR0Ha7RCmsA/tQVBOFHSZjUOtHZh3I3Fa3cPcyAEFs KYxA==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-427ff751fc4si1666845e9.1.2024.07.25.08.10.13 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 25 Jul 2024 08:10:13 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 46PFACKW007063 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:12 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v7 00/10] Improving base-apt usage Date: Thu, 25 Jul 2024 18:07:32 +0300 Message-ID: <20240725151006.2129-1-ubely@ilbers.de> X-Mailer: git-send-email 2.44.2 MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= `base-apt` is a local apt repository containing all upstream (Debian, Raspberry Pi OS, Ubuntu...) packages needed for a particular build. This series implements upfront repository downloading. This is the first step towards local partial mirror management. The current approach in `next`: - On the first build, debootstrap and sbuild are used for building Isar artifacts. The packages downloaded from the Internet are cached in local directories. - On the next build: - Analyze the logs from the previous build, save packages downloaded by the bootstraps, sbuilds and imagers into `base-apt`. - Use `base-apt` for bootstrapping, building and image creation. Some issues with the current approach: 1. Different policies must be followed for the first and the subsequent builds. 2. As we have multiple versions of the same package from the main and security repositories and rely on build logs and `find` for populating `base-apt`, extra care must be taken to ensure that the right package version lands in `base-apt`. 3. We rely on internal implementation of `debootstrap` and `sbuild` for saving and reusing the packages. Changing to e.g. `mmdebstrap` breaks the unrelated `base-apt` functionality. 4. Source packages are stored in a different flat directory, `apt-get source` for upstream packages is not possible. 5. At the moment of `base-apt` creation all we have is the package name. The knowledge about the upstream repositories is lost and no local repository policy implementation is possible (e.g., for the "multiple products, multiple distros" use case). 6. For implementing further use cases like "fetch all sources necessary for bootstrapping the base system itself", additional logic is necessary. The new approach: - On the first build: - All packages necessary for bootstrapping and building are identified and downloaded upfront. - `base-apt` is used for bootstrapping, building and image creation. - On the next build: - `base-apt` is used for bootstrapping, building and image creation. This series addresses issues 1-5 and provides the architecture for implementing further use cases. The new approach is enabled by default. Setting `ISAR_PREFETCH_BASE_APT` to zero falls back to the old approach. The implementation uses `debrepo` script which can also be called manually for pre-fetching packages to the local 'base-apt' repository. It requires `python3-apt` to be installed on the build host. Some examples of its usage in standalone mode: ``` # Create local `/build/ba` repository sufficient to debootstrap Debian system with `armhf` architecture: debrepo --init --workdir=/build/dr --repodir=/build/ba --arch=armhf # Add some packages to this repo (e.g., build deps for some recipe): debrepo --workdir=/build/dr locales gnupg # Add srcpackages for some package to the repo: debrepo --workdir=/build/dr --srcmode tzdata ``` Changes since v6: - Rebased on latest next. - Fixed possible stuck when `cache-deb-src` feature was enabled. - Fixed support of debian build profiles. - Moved locking of `debrepo` context from the bbclass to the script itself. This makes usage from Isar simpler. Changes since v5: - Rebased on latest next. - Changed order of the patches. - Fixes in `debrepo` script that allow to use it outside Isar in standalone mode. Changes since v4: - Rebased on latest next. - Rearranged patches since some of them are already in next. - Added possibility to select between new "prefetch" base-apt mode and old behaviour when it's populated on 2nd build with packages downloaded during 1st build. New behaviour is disabled by default, but enabled in local.conf.example for testing purposes. - Code passes both full and fast CI in both "old" and "new" modes. Changes since v3: - Rebased on latest next. - Cross-building for raspberry supported. - Code passes both full and fast CI. Changes since v2: - Populate base-apt before using at all steps of native build. Changes since v1: - Rebased on latest next. - Updated patchset description. Uladzimir Bely (10): scripts: Add debrepo python script handling base-apt meta: Add debrepo bbclass handling base-apt prefetching meta: Always use base-apt repo in local mode meta: Use cached base-apt repo to debootstrap base-apt: Predownload packages to base-apt before install meta: Add cache-deb-src functionality in base-apt mode testsuite: Set ISAR_PREFETCH_BASE_APT by default Disable deb-dl-dir in base-apt prefetch mode kas: Add PREFETCH_BASE_APT config entry ci_build.sh: Install python3-apt if not installed RECIPE-API-CHANGELOG.md | 10 + kas/opt/Kconfig | 13 + kas/opt/prefetch-base-apt.yaml | 9 + meta-test/conf/local.conf.sample | 3 + meta/classes/crossvars.bbclass | 1 + meta/classes/deb-dl-dir.bbclass | 21 + meta/classes/debrepo.bbclass | 90 +++ meta/classes/dpkg-base.bbclass | 27 +- meta/classes/dpkg.bbclass | 8 + meta/classes/image-locales-extension.bbclass | 5 + meta/classes/image-tools-extension.bbclass | 13 + meta/classes/rootfs.bbclass | 12 +- meta/conf/bitbake.conf | 5 + .../isar-bootstrap/isar-bootstrap-host.bb | 2 + .../isar-bootstrap/isar-bootstrap.inc | 99 ++- meta/recipes-devtools/base-apt/base-apt.bb | 21 +- .../sbuild-chroot/sbuild-chroot-host.bb | 2 + scripts/ci_build.sh | 8 +- scripts/debrepo | 590 ++++++++++++++++++ testsuite/cibase.py | 4 + testsuite/cibuilder.py | 8 +- 21 files changed, 927 insertions(+), 24 deletions(-) create mode 100644 kas/opt/prefetch-base-apt.yaml create mode 100644 meta/classes/debrepo.bbclass create mode 100755 scripts/debrepo