From patchwork Thu Nov 26 08:28:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vijai Kumar K X-Patchwork-Id: 252 Return-Path: Delivered-To: ilbers.mnt@gmail.com Received: by 2002:a4a:df02:0:0:0:0:0 with SMTP id i2csp1571946oou; Thu, 26 Nov 2020 10:32:46 -0800 (PST) X-Received: by 2002:a1c:98cd:: with SMTP id a196mr4895301wme.42.1606415325817; Thu, 26 Nov 2020 10:28:45 -0800 (PST) X-Received: by 2002:a1c:98cd:: with SMTP id a196mr4895197wme.42.1606415324401; Thu, 26 Nov 2020 10:28:44 -0800 (PST) ARC-Seal: i=3; a=rsa-sha256; t=1606415324; cv=pass; d=google.com; s=arc-20160816; b=bP/rkwlpc8B6RhslTUvZ8szSusSWJw776hHoQiDWtJhQxYIyyisG0nBVww3P9cCHDw 32XGVCLyHvU7srcAVDvbRkjpnyCYG089OlstY1LmH4DHMG5bwJEpqyB+PRztg9ZkAyc5 UyklzDYqggRSxkjzzeyNlu8uXFd+k2fUYvytznhI+d28Vcv4HkfZnuMXGxCXgGg0mRLi ShIsFD4Ze+Wy27phgohGROPbcDixTqZUKKRUxVWChyglg8DG7lUmPAtwivVSm1yBt7v2 roh3lGGl9PtY9y93quNpQb2Z+iM3DDVVkCRueaBw2QqhxqV6jXwMpAjyU3tQV4xckD9G 8EJA== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:ironport-sdr:ironport-sdr:sender:dkim-signature; bh=U9GE++vFeph/JH6xyqHQZTjr4UYtrF40oj+c4y5IuFY=; b=QUP37blkG0OaQ0rOMnipE6onH3DA84wW+ALWlz0JD635UPxC3s2SkTBMcYuHokBK1L DAGqKTJhs6Z6EGsLVPqlWDUYj4SuwyzPnYQzXxF/eOs/87i/VQ7A9fTx7q3R9hX6jqJO IroS0pq1v+ur6HWC6iudAdY7NosBespbysbyFajNUGKC79RqT0MyjOPp4AnOoYHdPkuy lF4PeOWAb2w76Th8YhwZWAhWfbwi5JHaSFcW0tyZNVx9O577B2ehoacqOIe1D0bUmllB bGKiVSPVC9hkpUFV6zgoZl/mY1iUGIiQ39E19pgx9rOGyWtNZ/per/UkpdHYuqumPWVN YNiw== ARC-Authentication-Results: i=3; mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=KNSbNDQc; arc=pass (i=2 spf=pass spfdomain=mentor.com); spf=pass (google.com: domain of isar-users+bncbcwp7lf37ukrbw7h776qkgqet7rgeaq@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCWP7LF37UKRBW7H776QKGQET7RGEAQ@googlegroups.com Received: from mail-sor-f55.google.com (mail-sor-f55.google.com. [209.85.220.55]) by mx.google.com with SMTPS id c75sor3340662wme.25.2020.11.26.10.28.43 (Google Transport Security); Thu, 26 Nov 2020 10:28:44 -0800 (PST) Received-SPF: pass (google.com: domain of isar-users+bncbcwp7lf37ukrbw7h776qkgqet7rgeaq@googlegroups.com designates 209.85.220.55 as permitted sender) client-ip=209.85.220.55; Authentication-Results: mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=KNSbNDQc; arc=pass (i=2 spf=pass spfdomain=mentor.com); spf=pass (google.com: domain of isar-users+bncbcwp7lf37ukrbw7h776qkgqet7rgeaq@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCWP7LF37UKRBW7H776QKGQET7RGEAQ@googlegroups.com ARC-Seal: i=2; a=rsa-sha256; t=1606415323; cv=pass; d=google.com; s=arc-20160816; b=EoYg/shQ91GsLscnnJkE0SnTRP9EpFaLqaj0kT1yeMI6kBp81pAFXiIicsGQl18yBy q1AnPXNTyiqpsldirU4U8+kkqRG1wmSPmsS6wxh3U9i8Vz8PKdFsPl5XdtSQ8RjZ/efS 9JO32XWGsuWu4IiKNXmK76Xk7MCIK7riQk/iwYPOY+MP/3hlwQhNdtqhdVEinUOndtGr NsoneMJwq0yO+A7dxW/1SENUijKse9jV7pLwiyPgl7ttTzRPmBd6LTEXqQI3wIyG6nW0 6/CikJ96Jo3/sqq60/QSed2UbB4blQZzncvXUg4eIJi1HSZ5PkCdNUWTlOZ8oMHGbAqD StpQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:ironport-sdr:ironport-sdr:sender:dkim-signature; bh=U9GE++vFeph/JH6xyqHQZTjr4UYtrF40oj+c4y5IuFY=; b=TXXYKjLjfb6ZYXU4BpchobNUeURF4PcIbPiNMh8yokVYlbHYx+YUtnV6XFJxGt8rNW anU6RtEfLLuYVxBkBHgdB9W6+6I8xdYtTZbKfSPTu57y1++u1gnYo8NgNBpMtPehlURv v3+BkJVGMtgMxHS2AS5iAR5WZgfOUsAsw6IxM8earUaN5flVOgrLd4LGQ4libNDa9yxx Q5aelj7aEK+XOZEtkaCdz/JGDuC5wG0knX/hrvHZ9SJj3BsuN39ixsqST8QtI684Svb8 v6qttMTZTIKSIKBMGyvVnK4flyjIbyfVxucqHT+6m7YitU6xANFqnJ7/HIxIKSlpR735 A37w== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.137.252 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20161025; h=sender:ironport-sdr:ironport-sdr:from:to:cc:subject:date:message-id :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=U9GE++vFeph/JH6xyqHQZTjr4UYtrF40oj+c4y5IuFY=; b=KNSbNDQcF3yhL4R+Bw9AuMgZ3Rwa2SisDId4i3Ho6Oz6Op9v2z72A7Fs0n1mkS+wdz zeucGed8oyOodyydUYZa+kBE2v1jihABQot/gZSsrWscmJKJ7w+4/z9YRbLydY0OtkVN HCAjKzYkmAcQjform9cd91tbpWU7QRRsthegiKX4E5RuMxNujH8Apa2rfUXimesl4o5e aM4YSUAx/p3+s+jrFg84gHs1pQErNHOUFKVBmmYTtvJr9qAbheainQGPwkJbkHfYTG6A 8vCbhGjIyFxm6kVSSsFXUJh0Vqt1qDXEi482REVYwrQdVr783xbjO/d0szOz/ZEZ/38H 0DgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=sender:x-gm-message-state:ironport-sdr:ironport-sdr:from:to:cc :subject:date:message-id:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=U9GE++vFeph/JH6xyqHQZTjr4UYtrF40oj+c4y5IuFY=; b=kuB+a9T3YoOljxGuVQ7gRHQr+sPALCZ8d1pIv4RL3WxTafTLBxkoKsoix2MmiutS1c b9WjouyMVhYjwYeuUqp21LwDM2hyEbi1QVN0yKNXyXurWuPSBkWuWUgjE8pm3CYcoZ6r FtsHwkXEBdwKX5DnwaFNQy6jd8v16BeUb3/sVyaSs5cKbkDQ2a/9mg6BpP1JM7c4wH9j bS+xDprNx0BjFndQ1PXU1pmyZO+RoO+9UQFvdmm+36rNTx4YpTdqX95DYayjI7XUuCVv xgPeZ5NP7+0BCV2ziXWIhMtBwUVLMDaObOU7rIhH6RW+bkhkyEsFIw1C5bJm7nW5M664 wy4Q== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM532puglRJWmFhJxezp+hFIVVTx2SucbBgtL04adYn3Z+OzsCo+PQ vO7uZMWkrgqsWBpleEXukjk= X-Google-Smtp-Source: ABdhPJyBvk2luJYUw9GpSoFjI/T6fyKweIJjiu8Lhbvkqym2PR5PAfPAUwBNAmZqNMjOkgKVhQ3yxQ== X-Received: by 2002:a1c:5585:: with SMTP id j127mr4825423wmb.169.1606415323713; Thu, 26 Nov 2020 10:28:43 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:414c:: with SMTP id h12ls1363107wmm.2.gmail; Thu, 26 Nov 2020 10:28:42 -0800 (PST) X-Received: by 2002:a1c:5f54:: with SMTP id t81mr4738670wmb.77.1606415322777; Thu, 26 Nov 2020 10:28:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1606415322; cv=none; d=google.com; s=arc-20160816; b=pvkxXkmiY7hZBhxLKf60Bchpi/I5//EVYFZ/POLXpGjba+KtLrCJaqhEb79bQcMUxT /G3JZjH6LLji2nRKdWGp7PbxN8pXEWjE5wTKxo0tEbfwYfE5S5/ediBEb2S9vhWNe3yF uSorVlCOpCU+S3FoSkOqAkM5NUmgSgd04fcgjf//SRjnxOJFRU/bxdD7EckGLwup4KuT rcQoKA/KfsEitdpwXBHQIxmajKdMv0QvUfCmwo2XRS85K/4peMZP674WOjyCqh4WppYg HsJfOY0BLo7JIN0n00YIWMW1nXY7JwHpCPx9c4mA4Uu0HsZ5NqxDal6HJQ8QOOQttfF1 sZGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:message-id:date:subject:cc:to:from:ironport-sdr :ironport-sdr; bh=XGX27MHT04mwZFR0v+FnYgQ+LyRhmQmFWq8xqFCxES8=; b=A/jKfRHOlT2N3QN6w/1pYXY8Sf/LtxvM40GG3kT7NS6BbXzjdRWpdJLGVObPicEn+6 tPk7ByE+0fM/UxmcTKU8DdwzOj+rRaWRhWRdAzt8gzNiHWYmFkJl0rd6uDi4xlqTxKn8 w+IArQJPO+x+bmORhBPEHCnje3Gz396Glb5xwHXfn/MHGGAYrK0P3NNPWa+vRixfQxI6 ckXlC4fAmyQnmGulRhD84JmQELwgUiUccBG+XqgJ48LK58wnIcbS4Gy5ttzkvPEPOAmE J6SKFerkA7sp0vccxstUoaKRJ53uw+G9na/jYLt2ZuHCKV9jN7ElKUBJTGPjCv+mGpg6 cHEQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.137.252 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com Received: from esa4.mentor.iphmx.com (esa4.mentor.iphmx.com. [68.232.137.252]) by gmr-mx.google.com with ESMTPS id y187si226499wmd.1.2020.11.26.10.28.42 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 26 Nov 2020 10:28:42 -0800 (PST) Received-SPF: pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.137.252 as permitted sender) client-ip=68.232.137.252; IronPort-SDR: 5swUSITIKorOJF+HBbysUdkiYmnh54ly17rrxCkThMdgVrAoxmIYihlfq3sivNX1hRu6riexkM HlaEyeawsKBXU8oCA/Kg6rWJRybAKl782I5povjVnJ59JdHAlXUDJ+cOneP9sAtu49QbfGyj7J NRYOCtnbT6D8H0t3A5UedSp3aYRs9B/VdtDOII3ruMOdr75WBG8bt3zV3r4BHpZ5XMGPTtJEc4 q5xvx7Osgt5Rmv2zj8XYuqkTpmaeD2dN8sgSgixqJIfKLCRX7QSOVPBj4S1ys07BiJanp7uVzK CKw= X-IronPort-AV: E=Sophos;i="5.78,372,1599552000"; d="scan'208";a="55644663" Received: from orw-gwy-01-in.mentorg.com ([192.94.38.165]) by esa4.mentor.iphmx.com with ESMTP; 26 Nov 2020 10:28:40 -0800 IronPort-SDR: 915xDJOGtunPUmdhvt+YXHYY/NSmgJOcvgjYLGV0L29CHZrz6+A6jooIIlHOZEHBgs/j1S1vj7 fkMgtMEMehXwfqWU3iPm7HI0s7Vm7hEso8sgbl46dlu/4PerhnfjmFADvS9uuZY8YGIT9D89zX A/rXx9DmN5dO42yvIMiAvFswRsWHsnDRpTwmylAnm+yltluLWG1rPMiLqbXPgF2+yMh5hIoU0w 2VKFdjVyRWOSbUo2mamUoA/2XjmMjSb4KfqTHrSArReZ9twje7lZ0ermPqXBI5iIMOQU9Wm2Gs 4jk= From: Vijai Kumar K To: CC: Vijai Kumar K Subject: [PATCH v10 00/14] Debsrc Caching Date: Thu, 26 Nov 2020 23:58:04 +0530 Message-ID: <20201126182818.7261-1-Vijaikumar_Kanagarajan@mentor.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 X-ClientProxiedBy: svr-orw-mbx-04.mgc.mentorg.com (147.34.90.204) To svr-orw-mbx-01.mgc.mentorg.com (147.34.90.201) X-Original-Sender: vijaikumar_kanagarajan@mentor.com X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.137.252 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: =?utf-8?q?1684448555424987877?= X-GMAIL-MSGID: =?utf-8?q?1684448555424987877?= Changes in v10: - Rebased against the latest next - Fixed some typos CI job running here: http://ci.isar-build.org:8080/job/isar_vkk_devel/91/ Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc10 Changes in v9: - Address review comments from Baurzhan Changes in v8: - Fix documentation as per Jan comments. Changes in v7: - Document details in user manual. Changes in v6: - Rebase against latest next - Address review comments from Henning Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3 Changes in v5: The major changes in this series are as below, - (P2) Handle cases where HOST_DISTRO!=DISTRO. This avoids mixing of debs from different distro which inturn helps the src caching logic. - (P5) Avoid downloading package from other builds. - (P8) Add a reprepro based sanity test to check if the repo contains the sources for all the debs. - (P9) Introduce a new variable BASE_REPO_FEATURES which provides means to enable or disable various base-apt features. (cache-deb-src for now) - Some fixes in (P12 & P13) Also, addressed some review comments from Henning. Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc5 Changes in v4: - Use = format instead of just using to download the right version of source package. Changes in v3: - Take care of non-existent downloads/deb-src directory. Changes in v2: - Introduced additional patch to cache deb src - Rebased on top of henning/staging4 tree Vijai Kumar K (14): rootfs: Make rootfs finalize a separate task deb-dl-dir: Cache host distro debs separately meta: cache deb srcs as part of postprocessing deb-dl-dir: Make debsrc_download faster deb-dl-dir: Download files only belonging to the current image deb-dl-dir: Factor out the mounting part deb-dl-dir: Fix skipping of removed files repository: Add a sanity test to check missing sources base-apt: Introduce BASE_REPO_FEATURES repository: Fix failures due to missing section scripts/ci_build.sh: Enable deb-src caching rootfs: Fix possible overwrite of existing resolv.conf rootfs: Handle failures when postprocess is rerun doc/user_manual.md: Document details about deb-src caching doc/user_manual.md | 30 +++++++++ meta-isar/conf/local.conf.sample | 4 ++ meta/classes/deb-dl-dir.bbclass | 61 +++++++++++++++++- meta/classes/dpkg-gbp.bbclass | 8 ++- meta/classes/dpkg.bbclass | 8 ++- meta/classes/image-tools-extension.bbclass | 4 +- meta/classes/image.bbclass | 43 ++++++++++++- meta/classes/repository.bbclass | 13 +++- meta/classes/rootfs.bbclass | 63 +++++++------------ .../isar-bootstrap/isar-bootstrap.inc | 12 +++- meta/recipes-devtools/base-apt/base-apt.bb | 3 + scripts/ci_build.sh | 1 + 12 files changed, 196 insertions(+), 54 deletions(-)