From patchwork Tue Nov 24 22:02:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vijai Kumar K X-Patchwork-Id: 204 Return-Path: Delivered-To: ilbers.mnt@gmail.com Received: by 2002:a4a:df02:0:0:0:0:0 with SMTP id i2csp152956oou; Wed, 25 Nov 2020 00:07:29 -0800 (PST) X-Received: by 2002:a05:6902:4a5:: with SMTP id r5mr2153744ybs.443.1606291409225; Wed, 25 Nov 2020 00:03:29 -0800 (PST) X-Received: by 2002:a05:6902:4a5:: with SMTP id r5mr2153613ybs.443.1606291407977; Wed, 25 Nov 2020 00:03:27 -0800 (PST) ARC-Seal: i=3; a=rsa-sha256; t=1606291407; cv=pass; d=google.com; s=arc-20160816; b=ssC1+4mF44jq+ju/7fmKBV18BmoIVA4Dwvj9RbAsyarJYvBuSvKs4C7mx7AB8rmcGs h2JULGPuN0x6Zn2dIE03dp6CNQoDUkGnNfIw9GLxr4S4JLrmuTbgIeUBk8dqIEayjfyJ CHJiJRDNodXrkW14P1eB1SssyjKluDeJk3d/uBs2JtmTKjc2XjkY+f1GXtJRvUtyOz1i JL8tF4z7yAEgwVqWIDsSbT+aEk1hBSlETHWk00csCh3hCBJO9Rf8pLyln7HXIz041017 jfGmC/ac3XwNweRl7Cr2WAtErgbALHWDtbMvGo98bbWiak35xRdxPbfT2xNGCyY6OCz5 DgGQ== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:ironport-sdr:ironport-sdr:sender:dkim-signature; bh=nkVKURO+jx8ntta+WJzeq2Tv+kK0j1NlTj4ZRUbDBrg=; b=RvYnDYq5VXCQpNF0MXiY1lZdq8hoZVZxWaNmlyyDupFY7RIWyNRiAMulXvT7QabwzT tLXrOF4YEi6XrWStrbAgJiVrVNRqFdcEAzO6jZruhewGe7SSaaOHOpIRvziBonHwGzX8 ukZRAujUyxl2Yk84DVoUvhopJCThwC4oOaHAVqvUu/Iel23VKWN/IIjpCQKWjIT7eLfm 4PlFuw7pbT5w1uPnjP0QemA3yhToGoWZGaEGWTmbSUk8M8PYabZIa+L4hw6UD5jIDCJb wapvQNyygneb1BuPVauMA8h0G+/t7C4FHCxZGNCNs06tTSCnmBmvVB76wTn2RVCoiuTC 2GXw== ARC-Authentication-Results: i=3; mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=gm4W+hdb; arc=pass (i=2 spf=pass spfdomain=mentor.com); spf=pass (google.com: domain of isar-users+bncbcwp7lf37ukrbt477d6qkgqefefs3na@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCWP7LF37UKRBT477D6QKGQEFEFS3NA@googlegroups.com Received: from mail-sor-f55.google.com (mail-sor-f55.google.com. [209.85.220.55]) by mx.google.com with SMTPS id p206sor403589yba.163.2020.11.25.00.03.27 (Google Transport Security); Wed, 25 Nov 2020 00:03:27 -0800 (PST) Received-SPF: pass (google.com: domain of isar-users+bncbcwp7lf37ukrbt477d6qkgqefefs3na@googlegroups.com designates 209.85.220.55 as permitted sender) client-ip=209.85.220.55; Authentication-Results: mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=gm4W+hdb; arc=pass (i=2 spf=pass spfdomain=mentor.com); spf=pass (google.com: domain of isar-users+bncbcwp7lf37ukrbt477d6qkgqefefs3na@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCWP7LF37UKRBT477D6QKGQEFEFS3NA@googlegroups.com ARC-Seal: i=2; a=rsa-sha256; t=1606291407; cv=pass; d=google.com; s=arc-20160816; b=sATvoa1KVC6jVsrUBMdD1uxLSzq05c56K89tS4le4FlYiDLoHVOWh1BQ57QAOgSAO/ J6nPXiJhuJ6/pn7SvSu5WpFsPJjzlg0ADPZjhiDzYVE0OXsDJHnb+8/aTmqCI0vhzTvT 7OGxs+uU5iq999eQixJkT693s3MihBXswwYv9Hsl+15FNo1MUAVLgOLRQMJOWvj6N+0T PUu0jUWS3yHBimXhfe6eEP9uQYgDJ0Z1OOT3D65JLgt+EhfXwAbOARJ3tvCJ5I79wauY 9XMjVvAbj33xO7xNjuuXxCQTU849lJ08XQr0jMRYvOEUqX5fc95Bs8ipnHXxa1qg0h55 vQWA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:ironport-sdr:ironport-sdr:sender:dkim-signature; bh=nkVKURO+jx8ntta+WJzeq2Tv+kK0j1NlTj4ZRUbDBrg=; b=qqApBb5nUDdp8tCpOAX9scQghaOHwo30Q/3gF6wm008V64o+9M5PHOiN13hjEd57Gb 49WAhntrvXFO8AkopZ4sPH3iFGb69i3JywbA1ymq3kgOvundjXhJ6f/fcXYzTKgCRTNP pNF2lsiUQEWwoi4+6J34L/xPHCXQxb+iQ+pS9ikpXBKFhRlRJctwwFzgk7rRpzcIGVRG XRSLhl2xJFzIKgve3phVTjNUjHdJC3QshBVasPtNH+bVDtF1pBSWm/cdcX7ZSstC2xIZ 6An90cPV9SWlj+1nTAnSj/WvEDqkMX5jiUbNAb7Yrm6dP0m+2dcJSapgIv9/XfFRigB2 IMaA== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.129.153 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20161025; h=sender:ironport-sdr:ironport-sdr:from:to:cc:subject:date:message-id :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=nkVKURO+jx8ntta+WJzeq2Tv+kK0j1NlTj4ZRUbDBrg=; b=gm4W+hdbZ5DcDZu2zrqVbeLjIQiiIsKvbw8HQjuHahPu7YQ8PMlXKuK0ggS/5yKYGg kUKlG100E/nxzq0esQjaRjtJR+qBoyzMc//qaKj/i1dKEAJ0ReoL7Q5P+Wmu1cDfaznc CN7UfXo3rUESaGpYTDIUm4ychzNzA6wSrcx45ArktvW5Z9/Ie1/PepPEPafN8/colj76 fuA6WaqI2wqFCrJDdKUJ+KGnsUDtR8LsoABiP5sNTnMigZoVFhdqRS7y8Esh6FXv8itm FelQ8MQwyofDHrn6UkGsW0Z8msMZSA6lL/uCzCzkwdIKR5f+Ns+XeEIiaZyzsLLr3OK1 BCyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=sender:x-gm-message-state:ironport-sdr:ironport-sdr:from:to:cc :subject:date:message-id:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=nkVKURO+jx8ntta+WJzeq2Tv+kK0j1NlTj4ZRUbDBrg=; b=XRHS34ZfR6WOnTTl2u+zKVuE6+V0I4ojdc50gUyO2Hy9M4rQJSyou4RK+Moz6JR6pT YAOX35disqXntLvymxsNdLrKuk/5JHyJxLbYGLpmYp5Htxx5nYN+Du9DlGJ28DcRukOs zHHcAgPzRneYPiMVuxGknA9hiuClzNPMf9bYioRnpH9hV/GSP+ZXKjr26fheYhVLWavs 7YWlb2uo1s9ItbBv4viZCPQ5sZpUA7LnVkEqF2O9l81x0GEuoYkLQeix+YkucNNv36Q5 IIdwewQkqAGdAwL0oqQOt4YwFHHkhs76A9EL9KgeynWDGHkaDGji7yz1skQlPgonn4Ao KXUA== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM530RV0cCjF+FVMSdAXZ/KaIUn88zB+SLoUTId/U39sJcmqHZcR5Q zGVVJbMnrpDBkc+VszPY9P8= X-Google-Smtp-Source: ABdhPJy4f1cpoZt/5v6Svd5yhccLA8v6YGFEIMnrpA+jdptz5sDAiJXYvDZlJ7GaGoYbuOfzY6D3zg== X-Received: by 2002:a25:4dd5:: with SMTP id a204mr2487262ybb.191.1606291407303; Wed, 25 Nov 2020 00:03:27 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a25:3806:: with SMTP id f6ls809661yba.4.gmail; Wed, 25 Nov 2020 00:03:26 -0800 (PST) X-Received: by 2002:a25:d9cf:: with SMTP id q198mr2171847ybg.243.1606291406749; Wed, 25 Nov 2020 00:03:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1606291406; cv=none; d=google.com; s=arc-20160816; b=eOIDDxEk4fNXfXGhTRqziRcKzNKU6qYbPcabLx30rBdNHzx+sJwct77MUhQsw5DZ9R da6ESOWXxKyV0RpFtFnbOfcHlMKJ4i399Ri496j6YxOXqkWxB1rj2vkwwkI2AYOH0+Ko JW1NlkAb4r1LSqn8+OE9/pZrlcCJKgycWvAGKq0+bmQDNYMrvoL/jbUCVqucSu4idX/j i+wK9rKvXvRJZgRrEVvsXEn1karJljECsmy0WngNaAjHlkzpvnS8mn3OHxM86gYn1z9d NcFApXbPDKc3Dd36agkf2E/93XzvDrEwpP59rdjLpU8gUpl1io0133diGhWZ+07E+9Pc u+JQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:message-id:date:subject:cc:to:from:ironport-sdr :ironport-sdr; bh=04nEdD9GZYSymStf2woAo9K2m2lTDiWk9f+Wu7ndHlU=; b=Mhv6U1IU8C4viJDLOoCsgdZkszf0dYVvo1K8QlU3AdF4tg0Np1a/LLIf/MjRmovu0u 0c0Fcfaw7oERM4cKKpPV2PdL9JDAsf3mAvHpxw/9cY/txzjjlyFcqxN+Hb/pnnEhlppl 3kU0Fh+kjjjdebUpwsZ5AMURDLP/JofGn9gBxC17dcic3Hbl/y8lO+0NwDGIB2unQzks cc9z33aYqN0A7nyeYr5e0++8cV/jitMWCx65I0XvRV2UOxeodtyZGlr4hgL0ZnfkatDY gk+lupYpE0n89l6f6KXZIM8wq5Z2OUVo/zfD6SmiwJhnMjvB6+94pyZ8sWaF4nQu7aXy jbPA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.129.153 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com Received: from esa1.mentor.iphmx.com (esa1.mentor.iphmx.com. [68.232.129.153]) by gmr-mx.google.com with ESMTPS id 199si77218ybd.5.2020.11.25.00.03.26 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Nov 2020 00:03:26 -0800 (PST) Received-SPF: pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.129.153 as permitted sender) client-ip=68.232.129.153; IronPort-SDR: hmdrx7kEhdOgx5tOBE6prfjG2i2w85+j7VXCmAysGKMAs295U453QzrPcTi4gn41vSvc84o4A8 /g0mUwpTp6kVuTzIIlLRPUT0EPabrw/KDoCpfD8NrtXfYb44id4jOV4AMKU64KAbKlyQmRpVvY qrAlVYhOeCTtEv2hrLUYIBvJDIDxdGJtnqq64nK6+f5zrhfRe+gdkxbKE3AkLTKrFTKLMIhFG2 kiFSEbGjTfz/O/pPc4rJL6LpqqmEuzR9/fRfrKzqXGFxsn3mc+wdlcRVGD3txPdbT/rCZzUVht Pxw= X-IronPort-AV: E=Sophos;i="5.78,368,1599552000"; d="scan'208";a="57762463" Received: from orw-gwy-02-in.mentorg.com ([192.94.38.167]) by esa1.mentor.iphmx.com with ESMTP; 25 Nov 2020 00:03:25 -0800 IronPort-SDR: lPSNrkNj3jwhbv79l5D3v6FL3cMFXtLJP33XHDFZlc2lxyCZFjG6g8Pgl+cqNbgC4juIGMnWsk hK0suH9GEvnyX6lMDeV9eTyAYNRI6WhtKq3JljYX8Ezn7wixP8EVsnjhP4NrtrGGI879s6MgqX KYqduXmKUUZy3OahcMqI3DqXbtWpc8tEXT2VkCHZ08JLEhmiOsR3wzHyUUWGCVS3pAHk4NJ17j w9GxpF6k0SWtniqzX176tc6lbqV9bWOOkBG7Bf81IgaMJWBOvzf1LDW6ZHHg5ZyTbgGp1w26w2 H9M= From: Vijai Kumar K To: CC: Vijai Kumar K Subject: [PATCH v7 00/14] Deb-src caching Date: Wed, 25 Nov 2020 13:32:47 +0530 Message-ID: <20201125080301.15818-1-Vijaikumar_Kanagarajan@mentor.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 X-ClientProxiedBy: svr-orw-mbx-04.mgc.mentorg.com (147.34.90.204) To svr-orw-mbx-01.mgc.mentorg.com (147.34.90.201) X-Original-Sender: vijaikumar_kanagarajan@mentor.com X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.129.153 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: =?utf-8?q?1684318619610298412?= X-GMAIL-MSGID: =?utf-8?q?1684318619610298412?= Changes in v7: - Document details in user manual. Changes in v6: - Rebase against latest next - Address review comments from Henning Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3 Changes in v5: The major changes in this series are as below, - (P2) Handle cases where HOST_DISTRO!=DISTRO. This avoids mixing of debs from different distro which inturn helps the src caching logic. - (P5) Avoid downloading package from other builds. - (P8) Add a reprepro based sanity test to check if the repo contains the sources for all the debs. - (P9) Introduce a new variable BASE_REPO_FEATURES which provides means to enable or disable various base-apt features. (cache-deb-src for now) - Some fixes in (P12 & P13) Also, addressed some review comments from Henning. Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc5 Changes in v4: - Use = format instead of just using to download the right version of source package. Changes in v3: - Take care of non-existent downloads/deb-src directory. Changes in v2: - Introduced additional patch to cache deb src - Rebased on top of henning/staging4 tree Vijai Kumar K (14): rootfs: Make rootfs finalize a separate task deb-dl-dir: Cache host distro debs separately meta: cache deb srcs as part of postprocessing deb-dl-dir: Make debsrc_download faster deb-dl-dir: Download files only belonging to the current image deb-dl-dir: Factor out the mounting part deb-dl-dir: Fix skipping of removed files repository: Add a sanity test to check missing sources base-apt: Introduce BASE_REPO_FEATURES repository: Fix failures due to missing section scripts/ci_build.sh: Enable deb-src caching rootfs: Fix possible overwrite of existing resolv.conf rootfs: Handle failures when postprocess is rerun doc/user_manual.md: Document details about deb-src caching doc/user_manual.md | 25 ++++++++ meta-isar/conf/local.conf.sample | 4 ++ meta/classes/deb-dl-dir.bbclass | 61 +++++++++++++++++- meta/classes/dpkg-gbp.bbclass | 8 ++- meta/classes/dpkg.bbclass | 8 ++- meta/classes/image-tools-extension.bbclass | 4 +- meta/classes/image.bbclass | 43 ++++++++++++- meta/classes/repository.bbclass | 13 +++- meta/classes/rootfs.bbclass | 62 +++++++------------ .../isar-bootstrap/isar-bootstrap.inc | 12 +++- meta/recipes-devtools/base-apt/base-apt.bb | 3 + scripts/ci_build.sh | 1 + 12 files changed, 191 insertions(+), 53 deletions(-)