From patchwork Tue Sep 29 22:02:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vijai Kumar K X-Patchwork-Id: 92 Return-Path: Delivered-To: ilbers.mnt@gmail.com Received: by 2002:a4a:eb04:0:0:0:0:0 with SMTP id f4csp4807275ooj; Tue, 29 Sep 2020 23:03:55 -0700 (PDT) X-Received: by 2002:a17:90b:3004:: with SMTP id hg4mr1087019pjb.7.1601445834912; Tue, 29 Sep 2020 23:03:54 -0700 (PDT) ARC-Seal: i=3; a=rsa-sha256; t=1601445834; cv=pass; d=google.com; s=arc-20160816; b=dGfjJAJ+J+QtJUXsDGezKwLRibdhJm/nzj3hQ2mvLN5IUa3ABE0zIvdRq58Iy+iimo 9u2uqRp7OFsW9m9+87bSM0T7JaTvbWlub2AKIoM+7Cbj/P0uqdn3+V2HbHQA2obAD2mj choJA6WSHjItzzcXd/AvPPdAJbaF1CGAmM2714c19tiRCXKmoM0ccE6stMhJDsOyS0+C bIivxyssYLpWgiDsWnUYDtM60r6vUMkpDg1+GL+f+cImARixIOBodbRSpRVWpIaros+j ZF5G2HBn6HKM3/i9IhNxcVNv46oryqNpdoWaWcseLm2isSgnhVCywjHl6NyXDXmb4rAR VVeg== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:ironport-sdr:ironport-sdr:sender:dkim-signature; bh=51uhTD/nv+7PcHinuFk/iFWfzeu2Fj0Rf5G0D3yLNlQ=; b=a1xdpC3S9eVhFJE9KjL0P+JcP6Yy7fuPxzepj6NkIPxOgQWUb5VodfODmegxPgWTd1 PVBOe6su107Aso/SJy9eJnqaMWK5HgTTqU1ToKjN0KDfeDEREnSNXGrjGsZWhWR2P9Fx p0Awn7v2ELzVo+Te4IDpiw/cCHzFecaBDgTLZfDzEhkNyfax9dXThAGmUPruUJIyfzuA 0C1iUGVUaTO49c02OCplmuK3BzSDGhDkmaLGrQmyxC4YFkJ5uPanQ/wE87ESd4rgwrk2 sLSM7x+FWeLNuMaGnM5tiKCBVWf5mJXOqNa1b9otmx/DeDhYzNzI4fIK8Y7KQA4X/8/t nuIw== ARC-Authentication-Results: i=3; mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=VBqhp1wO; arc=pass (i=2 spf=pass spfdomain=mentor.com); spf=pass (google.com: domain of isar-users+bncbcwp7lf37ukrbsv72d5qkgqe3lvo5ai@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCWP7LF37UKRBSV72D5QKGQE3LVO5AI@googlegroups.com Received: from mail-sor-f55.google.com (mail-sor-f55.google.com. [209.85.220.55]) by mx.google.com with SMTPS id 19sor391189pll.6.2020.09.29.23.03.54 (Google Transport Security); Tue, 29 Sep 2020 23:03:54 -0700 (PDT) Received-SPF: pass (google.com: domain of isar-users+bncbcwp7lf37ukrbsv72d5qkgqe3lvo5ai@googlegroups.com designates 209.85.220.55 as permitted sender) client-ip=209.85.220.55; Authentication-Results: mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=VBqhp1wO; arc=pass (i=2 spf=pass spfdomain=mentor.com); spf=pass (google.com: domain of isar-users+bncbcwp7lf37ukrbsv72d5qkgqe3lvo5ai@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCWP7LF37UKRBSV72D5QKGQE3LVO5AI@googlegroups.com ARC-Seal: i=2; a=rsa-sha256; t=1601445834; cv=pass; d=google.com; s=arc-20160816; b=NbqavrO9G+8myiVayFFLSkCRIONEOsnkVN+YHV9O9n5rZlY9uVHHEwseaaNQkbSJKk X/Nq9BxVg0P1rrw9d7+VwVeh4sfJRVhyyQp4VXZEFG3YQgpHT2xmGUAm9wnTvycKrz/w ol9cqcJyNQbvEAb0m6viNGbqeev8Erd7VeLO0NSwJKOraeUajL3UIQ0/wZSM0bjz6oX5 oLz23PXaBhO97Dc1xhnV+tdmNYxV4inybDdGRIwHXGQgsKq1jDRHqBjeYYKF4+jTZ/Q7 awq817rH0krAskys115K5UE8vhuy5GUorr59l+SJ3evmCPEuMuwuTDL8NH8w6HmGzy7d ibjA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:ironport-sdr:ironport-sdr:sender:dkim-signature; bh=51uhTD/nv+7PcHinuFk/iFWfzeu2Fj0Rf5G0D3yLNlQ=; b=deKoE9eJYKIOfmetHqLfPlyhHsEU99quYlP3mMDE1IKqb9S4c95zT3t0SxjViTYybo VKGxthfR2qSzLMIiLLilOOAM3GplLL+4TBnGuM/+o4cDGMJVAVgU6U4SZAEaYEfnDTS0 fAozakmKvmBimbaBjIYA4HiLTuidyApi/QILa1iV+T7Nf5ZcWjoTIy7neq5ScMH7U9it U21M6z+pzxfKOM6bfXax3jaxVLFDr4xHBsyXLaxLU9jblbJhMdXHovf5T5vqTWge9i3V vXAO1KvZaESU7SKlAdt65mbp1lKOv4kYgMiz0Hxe6xqwd450JsIYyE3wd/0gWLX2bfnC qFjw== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.141.98 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20161025; h=sender:ironport-sdr:ironport-sdr:from:to:cc:subject:date:message-id :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=51uhTD/nv+7PcHinuFk/iFWfzeu2Fj0Rf5G0D3yLNlQ=; b=VBqhp1wOnfJ/JzEAkKpLgACete2f8gCduNxNDs6RmNn8hsWQFUyBnNi6NrzbFH1lSG wjBkUP40kzBFDjgV6mz2kuW8qFmh/SGaj66HtnalK7K9StfiXMI3dNlyIY3EZlR0JRZc 5xr8xTDGCN5HsjuKoLVGY/BfImDcZQxRk1BIvLYXFbpIVJqMuPtQNfaHO7S4otT61ATk xccAjWVKLsAoEp6+wO4hU9c8K4IWqck9lBS4Ld1A2n1GmPTfyE0IEj85hyvRy2wiqiTa WZ9M9YqVEkav/uO1OoYer/lIDCrgH+50N5anLniJRwi8tieTt3O6HpArMldC9Oe+nm+w 3XNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=sender:x-gm-message-state:ironport-sdr:ironport-sdr:from:to:cc :subject:date:message-id:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=51uhTD/nv+7PcHinuFk/iFWfzeu2Fj0Rf5G0D3yLNlQ=; b=lRGtHSfRpCY6Hab1SrTHTcWsNqj6fpU1ChNj+wJSZrSaOvwooFlm/oC0rVD/ZeuSfZ eAs1OgR0fFVEcttHsSAzwPrVjwsogJJ/R8Zerh0WjPIjHwi9qTTCfwUp3e0r5e6iYrBJ GGgS/YKXyl/GZft3bO6wWqVKyzCViWJxTf2SyHYd+m9s6CHtVT2598FZ0WB1B7QjCZsI GWkr3SNrVWb9LCjtVmHBhR72sw+uDb+/7QSd0MGOG7KPsERrqPG1HxTklHPmn/9uNZyY IDUmTR31n0gGdzVx30x4WFbmgpvh5YuSO3JQwcBrEqRaWJoapaG8vgGA/hjAWjVJ/OVr WkqQ== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM5332WEpOIm6DGhNxKUWJgFwj5F8W/zUSHM5MD1rt9Zq/MZtc6K3e NJ2hKRE0pvb8KnOVhvaSbR0= X-Google-Smtp-Source: ABdhPJz+Z+Ts5Cwp9Ubfm3XFZfpo3GSztKmzCVaz1+FToGlEbuDLCcxOk2XBy7zS/ODd8zdy4+smCA== X-Received: by 2002:a17:90b:885:: with SMTP id bj5mr1103163pjb.110.1601445834174; Tue, 29 Sep 2020 23:03:54 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:902:7613:: with SMTP id k19ls426376pll.10.gmail; Tue, 29 Sep 2020 23:03:53 -0700 (PDT) X-Received: by 2002:a17:90a:7bcd:: with SMTP id d13mr1182913pjl.18.1601445833523; Tue, 29 Sep 2020 23:03:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1601445833; cv=none; d=google.com; s=arc-20160816; b=zXlpnV1y1cRP289RKUvGFYl3KeXhUzA74y0khP39BIBl2y2JyoS07A0wWV5FKIU8Og knhHOOjBM4IQ9S2pRAarlc+A53B8Fau6ch0itdChHWufSzjZApaUzx64O7ZJHaddULlk 5V47D0l9JncZNUX16M970u4MCLDLIYpD4mJzfR18gtEdaafpvqklCi3UYrBlwPldfuk0 ZlCy3vvKSPO+k/S57YKvP1p8OHC3r+gAMrxziSHs0YjZnXr7tIyOIBkn/+gSwKbMsmPD vqT5caZwYPV0cHu4i7LYhwpB7nD4mqJ0VuY9QK25xeBn/he9CndPNhbCwYcmMV1fI0wP Q4FQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:message-id:date:subject:cc:to:from:ironport-sdr :ironport-sdr; bh=+IcXObClRetBssGR2S/nj3yCDMSRrWDrBjuSsH6YF0U=; b=fj1ACco97e8FnejqPrasK0LkyfW4tbn9LVrVi+EgeRTyAR/A54nUIV8kKP6XsrS6QA ibz+Jo4QzyVthepC8hfNSKuDuCtr4kKiR2BK2H0Lq6kNVD+fJLA28OaprqEde4sT702X vNQNFy2D7bqAoGzgAtZXBY2K3wIlJsamXqwBOtiG275Krmd1Sn3UWPxwqHMlrYueyl98 FxA8a27HKudr/UM4IIl+gZOG+Uva2C8PdyiXYNnUwWi+TW+EUZpwn1PWjXCGxvSkwFL0 je2mcaVfcFeDfcXsmTS8d45kYB2l0BQ1uQOigW86EMhOAmdH5VuJ0dKMNnTwa3h2rBew e3ZA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.141.98 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com Received: from esa2.mentor.iphmx.com (esa2.mentor.iphmx.com. [68.232.141.98]) by gmr-mx.google.com with ESMTPS id mm16si51246pjb.2.2020.09.29.23.03.53 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 29 Sep 2020 23:03:53 -0700 (PDT) Received-SPF: pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.141.98 as permitted sender) client-ip=68.232.141.98; IronPort-SDR: hCRWASK1Psu4BIAIBD0mDPGlN7/YMwWsu0YEWUD/BivDhLloxYFE8wl4LNHryi+OBZPKOaz/T5 cvqJ+NfhyiRDQslO4KotBNSo3TMZNmbcLAEnVJc3yOB67ygED6iJ56PifkHPGaqqjpYpAgLzoV pIKR6M6Y2Lr5AqKY6Uqj0zW3sjpM4bdJS3xZ6B/1BOBuf/RRKoNihgQw0VMOaivls9tVAr3bvA p0eRX1UzZFTTnwI52Jh1K9hBZzcP0Fph7p16nGY23U54wIxIAElC5ZNkLqd8WRIOgaF6qhSXxe D3w= X-IronPort-AV: E=Sophos;i="5.77,321,1596528000"; d="scan'208";a="53461954" Received: from orw-gwy-01-in.mentorg.com ([192.94.38.165]) by esa2.mentor.iphmx.com with ESMTP; 29 Sep 2020 22:03:53 -0800 IronPort-SDR: 4cN/Dtl1aw1+VGT9wu8vdQvF7K+5KiDyJ5H3wP2cw52oupyvKTwM6Dwyt+AYcSb1Zg8cvQVk31 79y3JeMxRSSnWG4WSUCTL6QjkyVqW6SFCmKTFrgfQUalGQSJSulMwKLdC/Z6THjN7lfXKeVjR2 20+9eUSGGVt2ipO6FUGlegl+i0Z9P0jutXuWtLGrwEjiB4UqeVrmxY/iP4sSVrWXZfzHZ/175k rpVt1iZ5u7SpKHSYp7Sj27uqmR/JcQyw3Pk9lZsiqDNVJP/iTbbQHZ4lNFPkdAOUxwJ4wDPPOU Lps= From: Vijai Kumar K To: CC: Vijai Kumar K Subject: [PATCH v6 00/13] Deb-src caching Date: Wed, 30 Sep 2020 11:32:46 +0530 Message-ID: <20200930060259.12422-1-Vijaikumar_Kanagarajan@mentor.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 X-ClientProxiedBy: SVR-ORW-MBX-05.mgc.mentorg.com (147.34.90.205) To svr-orw-mbx-01.mgc.mentorg.com (147.34.90.201) X-Original-Sender: vijaikumar_kanagarajan@mentor.com X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.141.98 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: =?utf-8?q?1679237667711661418?= X-GMAIL-MSGID: =?utf-8?q?1679237667711661418?= Changes in v6: - Rebase against latest next - Address review comments from Henning Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3 Changes in v5: The major changes in this series are as below, - (P2) Handle cases where HOST_DISTRO!=DISTRO. This avoids mixing of debs from different distro which inturn helps the src caching logic. - (P5) Avoid downloading package from other builds. - (P8) Add a reprepro based sanity test to check if the repo contains the sources for all the debs. - (P9) Introduce a new variable BASE_REPO_FEATURES which provides means to enable or disable various base-apt features. (cache-deb-src for now) - Some fixes in (P12 & P13) Also, addressed some review comments from Henning. Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc5 Changes in v4: - Use = format instead of just using to download the right version of source package. Changes in v3: - Take care of non-existent downloads/deb-src directory. Changes in v2: - Introduced additional patch to cache deb src - Rebased on top of henning/staging4 tree Vijai Kumar K (13): rootfs: Make rootfs finalize a separate task deb-dl-dir: Cache host distro debs separately meta: cache deb srcs as part of postprocessing deb-dl-dir: Make debsrc_download faster deb-dl-dir: Download files only belonging to the current image deb-dl-dir: Factor out the mounting part deb-dl-dir: Fix skipping of removed files repository: Add a sanity test to check missing sources base-apt: Introduce BASE_REPO_FEATURES repository: Fix failures due to missing section scripts/ci_build.sh: Enable deb-src caching rootfs: Fix possible overwrite of existing resolv.conf rootfs: Handle failures when postprocess is rerun meta-isar/conf/local.conf.sample | 4 ++ meta/classes/deb-dl-dir.bbclass | 61 +++++++++++++++++- meta/classes/dpkg-gbp.bbclass | 8 ++- meta/classes/dpkg.bbclass | 8 ++- meta/classes/image-tools-extension.bbclass | 4 +- meta/classes/image.bbclass | 43 ++++++++++++- meta/classes/repository.bbclass | 13 +++- meta/classes/rootfs.bbclass | 62 +++++++------------ .../isar-bootstrap/isar-bootstrap.inc | 12 +++- meta/recipes-devtools/base-apt/base-apt.bb | 3 + scripts/ci_build.sh | 1 + 11 files changed, 166 insertions(+), 53 deletions(-)