From patchwork Fri Mar 25 00:32:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 1638 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3) with LMTPA; Fri, 25 Mar 2022 11:32:34 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f63.google.com (mail-wr1-f63.google.com [209.85.221.63]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPS id 22PAWYeK030626 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 25 Mar 2022 11:32:34 +0100 Received: by mail-wr1-f63.google.com with SMTP id j11-20020adfb30b000000b00205a3f19489sf547577wrd.16 for ; Fri, 25 Mar 2022 03:32:34 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1648204349; cv=pass; d=google.com; s=arc-20160816; b=fQ9VkJhhy9cWnEiDvbQdBnOCdkLrw1xsyfS4xT8omHlnylwPR7NaFUSHt/isLO9FD9 S3VMfFCFqMS9Q2Ipn1gfqJamB3aagCfwIXOHE7fYB4a3j7upWhCkHRACSSq5DXyo+OKf pXTDOLPm5Zt2zTor90m84rczoUhU3G8h217FPrdfGL0xFrXw+r6CClm3n/mgZnRoeoIi dWTL0+NQ5wTkFBk03nKcfI50hwRzjrDcYPf7Xk4+QGdj7UmfSf5SFyfZ7eShs61H6a+h hdSN6lkK9+0fyYZzALrKE9u+ajDjDuI5ky79WH37nrR3a0VOmQyTmP4zCUvOOmN0C2Cq 7Zpg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:to:from:sender:dkim-signature; bh=ZHULFZ8VZLuWkd9vIh+aHW5h52feex4AyJ06xJvGeKk=; b=L6tY7CXNWjUw6bC1jN55SOzJvIKuUqHqMNfK1EQGZ9a88ZOtZ1a3mVo9vmXyj34tLN R8FPIBdToiUYqEwPRfeMhgZbw8Xzc+4zL1Pm0wKLXFEQHdmjfsosXxhFQVMoY4aAzoje +4rW2GCid1IGFyRnxz/DOWTPrw1+A0Vy/OUEbeLcg7zVDo8oSRY4x3QcJLExdUhsA0+0 3s/4eHf9rKt0SsdbAOht+qV5CXycjo9kVgvISxu2I0NIoa8nzgcDMpQ8k0tFyMyAWvHj SKW5PGoEESHUtkkDLxrmHcPo4lQWmpzAx+EHqcZesG6o3NwaOblG98DTyZoM00wM15n1 8qUQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:subject:date:message-id:mime-version :x-original-sender:x-original-authentication-results:precedence :mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=ZHULFZ8VZLuWkd9vIh+aHW5h52feex4AyJ06xJvGeKk=; b=V9YtK4pPHH1WgeZHAm7GaOzq2lNQpNYiMEtXLT6PUEzxYD/kCm/ZLLKrw8Td89uBqv gmQSIn9BBcr7cCV1blv2sx66AOzQtWnoyHGiCdHtqHP4Wbny+3TauyETZc360awZenuj Jel3n+E/9riBnIVlqcaT7b5LMv9rVy5CQVftbl3Z8dh3BY7rkcB5ZjVIC5BkImEfnyAr FmNEuY6sj4iWlpL7D8OlUpvwuEERfmpKVjsRylSepIXvqe7UI52tpJ49+2s/9ePrCnDm CzJ7mJ8q+RJwDnWD44MNkbhbLIw/xmBYtTJNWvFYTMKxyJmW9jVTmaj9tDkMWUckfygJ IuBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:subject:date:message-id :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:x-spam-checked-in-group:list-post :list-help:list-archive:list-subscribe:list-unsubscribe; bh=ZHULFZ8VZLuWkd9vIh+aHW5h52feex4AyJ06xJvGeKk=; b=g5oc4PEObKK0D0XcbYJ5nxXZgagwfxrGUJR/XH+4V0UzMUZmxKJnDTeRLb8uj/03aV isWG+wj6hOs8UGxLxfskaanPp3xTJBmafNpqJqtA7J8nKNUIxpOO/GNd7bK5kNb3/0FZ Kx8WV6G97u5wHyIj5X8CSG8jmKGxWRDC2NT9mz4B+GHYlJZfP5USXs8t0QExb3sAwtoX X4RCopIUiLfqmIBGqdNTVcFu36qgLN4clt0jhY0OLm7Z7leyrcfIWC9VXb35iPWLobvl flsZ89xKX8m35OOS6OUpm/FN4aqGZOJAgS4w886S4kkMhJfh3pa47zgQcP3qm0WpgRJG 5uXA== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM5330SKlaPDpw6VhUFbDPJxkTDPVCAZaBO3+o9VnhfpVTFRkyARPb 0SXZq1lZjeES9f7kjqWo8/Y= X-Google-Smtp-Source: ABdhPJwKvnIRnTbTKfOIOb5K+cZLXwZaYX2N6Mmsh+bvEqqKWh4iS/1PBmJiuGWfLxONjuJDggxeOw== X-Received: by 2002:a5d:5846:0:b0:204:1a79:f1ab with SMTP id i6-20020a5d5846000000b002041a79f1abmr8330771wrf.520.1648204348970; Fri, 25 Mar 2022 03:32:28 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:4e01:b0:38c:9f99:b05 with SMTP id b1-20020a05600c4e0100b0038c9f990b05ls5817624wmq.1.canary-gmail; Fri, 25 Mar 2022 03:32:28 -0700 (PDT) X-Received: by 2002:a7b:cd13:0:b0:38c:9a08:5c62 with SMTP id f19-20020a7bcd13000000b0038c9a085c62mr18551244wmj.154.1648204348012; Fri, 25 Mar 2022 03:32:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1648204348; cv=none; d=google.com; s=arc-20160816; b=rGVsSiJHnK2LXt+JQ+27F9zeKduUesZsvac1cK/OSJGL4rNlR6WWItP/dWw7lX9xLQ BuwY964S/Cj/b9T3VfatSFwjENt+kamz82ckSx718NIMOiPf8YALSpCf9s7q2ikP2Bn8 IWXlhsDFznCjZ3IqZDGQ6T2FbuS1BayLvR5zjCE/bmErjFx80u25DSXO3VRXda3BwpTw 34DIz4bvME0gcVjA2UEsBPiUFWo2Ms2j0bFJqzK2ROGU7Yo1nSpYw3xFXjMw+3ZCSz50 0yL06TAiELiM/9wEUsUgfjep8Xs+s0/XJfuHqPwyMsMgd9qJ6nMn20yfRpFIFS8QYvjx f1Lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from; bh=F3Ft31EMUoi6x0uZqzrohcVHVdjPtT/OSuG9HQyfWqs=; b=vwDxTa1mThuK1BcXKc7jvqk3/w4v89KMBvh3DDgSn16UDeknDKe2OuPinmiHhexqS4 dJDuOif/oUPrsi+5z5OqwWhzTGk4+Vk23zHxSQUQawAUriKFEVuy33wKwHhjQp1a17CB Z4MqOckxp/3d9eh5kr1/tKD2/+ojGoTrgw25iG/xDSsqxr9h/o1U4LwU6d8GeFtH7COP 8XTIZYI4BNDegHPsQzLiZgvmGQCLyilwtyyptLOFRTM9sndmUuMr23+kri7p45cNUFFr tPvkwkzU/tnjBYfEjyNuN0XOP+Zrg92i3pTeM4JrWVkcSEWRWmaCQPYjTOFbhKTtmjza EmYw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id m128-20020a1ca386000000b0038ca3a3b9a9si223172wme.2.2022.03.25.03.32.27 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 25 Mar 2022 03:32:27 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from baighyz.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 22PAWQZW030611 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 25 Mar 2022 11:32:27 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v3 0/5] Improving base-apt usage PoC Date: Fri, 25 Mar 2022 11:32:21 +0100 Message-Id: <20220325103226.27033-1-ubely@ilbers.de> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: INBOX Currently, base-apt is used in the following way: * At first build every deb file that took part in installing is cached in directory $DL_DIR/deb/. Debootstrap is done from remote source. * At second build, `base-apt` repo is created from previously downloaded debs (if ISAR_USE_CACHED_BASE_REPO is set). Debootstrap and installing packages later is done from this local repo The idea of this patchset is to precreate `base-apt` at first build and use it later for all steps that require any package downloads (build dependencies. rootfs populate, etc). So, no second build is required to get ready-to-use base-apt. Currently, only native builds are supported. Architecture related dependencies predownloading is not yet ready, so cross-builds are not working at the moment. In the future, there are plans to switch from debootstrap to mmdebstrap that is able to work without sudo and uses more clean (apt/python-apt) approach in comparison with debootstrap. Changes since v2: - populate base-apt before using at all steps of native build. Changes since v1: - rebased on latest next; - updated patchset description. Uladzimir Bely (5): Add debrepo python script handling base-apt meta: Use cached base-apt repo to debootstrap meta: always use base-apt repo in local mode base-apt: Predownload packages to base-apt before install. draft: make isar-apt repo visible for base-apt meta-isar/conf/distro/ubuntu-focal.conf | 4 + meta-isar/conf/distro/ubuntu.public.key | 53 +++ meta/classes/buildchroot.bbclass | 10 +- meta/classes/dpkg-base.bbclass | 8 + meta/classes/dpkg-gbp.bbclass | 7 + meta/classes/dpkg.bbclass | 7 + meta/classes/image-locales-extension.bbclass | 7 + meta/classes/image-tools-extension.bbclass | 7 + meta/classes/rootfs.bbclass | 30 +- .../isar-bootstrap/isar-bootstrap.inc | 59 ++- scripts/debrepo | 363 ++++++++++++++++++ 11 files changed, 526 insertions(+), 29 deletions(-) create mode 100644 meta-isar/conf/distro/ubuntu.public.key create mode 100755 scripts/debrepo