From patchwork Thu Nov 26 01:18:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vijai Kumar K X-Patchwork-Id: 223 Return-Path: Delivered-To: ilbers.mnt@gmail.com Received: by 2002:a4a:df02:0:0:0:0:0 with SMTP id i2csp1229217oou; Thu, 26 Nov 2020 03:19:50 -0800 (PST) X-Received: by 2002:a17:906:6414:: with SMTP id d20mr2220907ejm.82.1606389590102; Thu, 26 Nov 2020 03:19:50 -0800 (PST) ARC-Seal: i=3; a=rsa-sha256; t=1606389590; cv=pass; d=google.com; s=arc-20160816; b=h/uzpjnzOc1QTgXrgLdULumY0610Qb+PprVvJE99YmtJuchGKJCtkr8U9tBd05C3Td nsDBt9iQw5y+cWUv9xTUxuvxqYK0MRsZXWA6xaltNJQxZic4m7PLdoeER10BSpaf2ZFH cAEZmLvblX3+nGKJSZi5funHz8XRQa0E6uoaPX+xrlqTMihIjJIgGsZpyPmtlg5slWwl Sm6ct76lnCJlxJEmgaZjea033L0pL81Q6dD1RI0iBbulDmdhxkUwitTunXrr/fbwaJrR 1rt88TyAmGPPlz9HVPf4n8PCzxaMgEmxgwln6XO7vCDGeB+1VhevVhEB/YgEAINU5nTn 8Oqg== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:ironport-sdr:ironport-sdr:sender:dkim-signature; bh=8uxrWXWeQRq2ZXPQF5xXO/u7pcocYJVX4TLRvICnTGU=; b=hnFTZcqJInvAfkv9VuMh4Hk1sDSK6xvX256a0xFKr21lC09ZsW6BtTQpHtzBDfY6ft JnCIq3D7TodNUllHB4669YgyCmbLMZGC9OhF6PLvYFPIwUyNga5B/HZcMYRWGhyDcwEH 16ifeMINJIWkh6UavHcSJhXlP0v6J9CyjDoXjwGm6c0CTuGMftnbfsvD6ysI7orN3PyC j+ALnMlpXrULp28KVLS8vyUeDbG327lFYRPal1bw/WiBVmqSrZQ/zvKot8ECThpyEpGl 6WuQrT87kQ+Qhw0Y6pcHnEqrBBkGo96HCKGR9HOAYGs6f619DJf5c7djvGORGvwt5cDH XlNQ== ARC-Authentication-Results: i=3; mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=LZBhJaAr; arc=pass (i=2 spf=pass spfdomain=mentor.com); spf=pass (google.com: domain of isar-users+bncbcwp7lf37ukrbvm6736qkgqe4h7ktnq@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCWP7LF37UKRBVM6736QKGQE4H7KTNQ@googlegroups.com Received: from mail-sor-f55.google.com (mail-sor-f55.google.com. [209.85.220.55]) by mx.google.com with SMTPS id h16sor3282599ejj.59.2020.11.26.03.19.49 (Google Transport Security); Thu, 26 Nov 2020 03:19:50 -0800 (PST) Received-SPF: pass (google.com: domain of isar-users+bncbcwp7lf37ukrbvm6736qkgqe4h7ktnq@googlegroups.com designates 209.85.220.55 as permitted sender) client-ip=209.85.220.55; Authentication-Results: mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=LZBhJaAr; arc=pass (i=2 spf=pass spfdomain=mentor.com); spf=pass (google.com: domain of isar-users+bncbcwp7lf37ukrbvm6736qkgqe4h7ktnq@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCWP7LF37UKRBVM6736QKGQE4H7KTNQ@googlegroups.com ARC-Seal: i=2; a=rsa-sha256; t=1606389589; cv=pass; d=google.com; s=arc-20160816; b=HDlZ/RiEFcY1iqu184xyffc3Oq8036XnfYtaybpeQQ1n62exSfZ85/dsnezJICy3Pq E+2l3VsOk34Z+SCmdH2+N66iioQyLU0gYzFfhP74Yf4MLOpExf6BcQ0qLsC6ttBVcbfY V9+Wgx3TyORSUOGmny2RAfcWnr4W/DgGvsHFaevyk1eQGz7KhK0Nh0rbTnEs+YgLLGX/ /aTKG7+6vGfUpxdMOnm/lan1YmOS7kNunda4jJwcvklIpL4xGv5jZLmq6QDToM9MabVz 2wPNSEXtkJzMh2bKSLlINYOSUu3JXsnZYXDG3VikwtVZrDa6oaJrSOWps8XdFW8xcE2r yzGQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:ironport-sdr:ironport-sdr:sender:dkim-signature; bh=8uxrWXWeQRq2ZXPQF5xXO/u7pcocYJVX4TLRvICnTGU=; b=sSq/Q2rcDdIRQ1Z7/I5N6lo8jL+O7UXJ4Gh8mVC7n+RCM/d9Is3cN4+i6oNrTufCLO oe8FiIjDBiQMbWtrefCmqkptoXmgz05XZ5nZZVvaEFPk3r7IBBl2f2pSTGmAL71PgvvF qoHZS0qDEkUegJCQZGFZveZsej9+Fqm8TAMc1Uzqh41LxMiO+aOUGpxom5jX+Zf61uGY ij0sYxrzvQtQYRzK1+CDsMdh3kJEbfh1NEdAVYjVG2kOSwaUoaM4poa90/HYRipCEdOM YTSO7JQddVkMy/fWAbvD8p9JG2Vv77hQQrTZFqkEGFSCqflK/VgJiXq8ZLp3I17ByQGq 3iHQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.137.180 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20161025; h=sender:ironport-sdr:ironport-sdr:from:to:cc:subject:date:message-id :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=8uxrWXWeQRq2ZXPQF5xXO/u7pcocYJVX4TLRvICnTGU=; b=LZBhJaArLu3chzYOPHfPV/AT9phwmPWQpE13Pdh2VRtmdqg5vum6kJgqueQOUIRnmn 0MogqEuSzf9FX4RAIlraHbEYUCQuOxRcPdN9gIt2SHdFs5HXODK0Cb1DeWBQx0AWZb0l EMTWNTLFzZihi4uEpAHoWk0PyWHZkRu9xmlRMTyQpTyNlZHF85LwLZ/lG9wcwI9+kO2V K2emEyaQsDLYY06L47jECxVf6J8CY/M2z+xDBkn54fIeBGHDc0sPZvXUNbjLu+BKjkRH 9fN0VH4jnvkb/IdirYmtd7ictk1XSnZwTUnibLeAI+/h1TBEWbd5GHEHvtyZdhYqxMaP 8QWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=sender:x-gm-message-state:ironport-sdr:ironport-sdr:from:to:cc :subject:date:message-id:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=8uxrWXWeQRq2ZXPQF5xXO/u7pcocYJVX4TLRvICnTGU=; b=s5Ynro9W+KpJGe3V452ziyQZjxTkZv4PV6unHT9eoW6q8uZ5wNfj5LHnrCx+ApRVKO lBwr5490iEPphLCHdFlTlzXBOZsV/lhtJC3H046UScdYuMNRgSYV4IbW9ODGXUBf027a 6cHrB/8N+F5Fu7nNFwfrulGIKz3nUByi8aRKEoYoh/eZpwFaWgvgpOaRZMr4rzRUUDPD EIXseoFLOg77Rz1+Me426befOkUuqTDRcigzrA9K7oe91QIEKNXBkZOnHVPZaCSpzXB3 rrKrszjGzliV5RqAn8N3JKyM7DOr1fACpA7Yw6EHRQNAtUmwdoQ9XqUudmOWOjnp4bzB ruyA== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM5300t64S/+e2PKvBZ8a1c4b6LuOBE6LSoZk4cebOan8rvexaCX0t Zsv5eo6r/j0mb8jIbx1Lkj8= X-Google-Smtp-Source: ABdhPJwtKj4+8u841FwPzmaadnHq4vBxxwawSzr+/p+cOk0mJkUlHUXCd7TYnxwpq0kEEmYP2XZ5sw== X-Received: by 2002:a50:a444:: with SMTP id v4mr1840901edb.175.1606389589415; Thu, 26 Nov 2020 03:19:49 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:aa7:d151:: with SMTP id r17ls1870309edo.1.gmail; Thu, 26 Nov 2020 03:19:48 -0800 (PST) X-Received: by 2002:a05:6402:1813:: with SMTP id g19mr2075821edy.105.1606389588444; Thu, 26 Nov 2020 03:19:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1606389588; cv=none; d=google.com; s=arc-20160816; b=W6RLfclqz1Cy89A2IxYbn/uzfxPpPeRfPhv3BveczYAupNqHRrfRQI/AeMpW+6JdFs sO7oEaeYFl2acPu2w4dSveTM5v4OBhdXvCPBKTQJMZUJdTsT4tHOC1ipvSIAG2VWdWra GUZ+0nZgkjcMdSHXpRukJdWc3+KcNhwZaByALHvZsdBodEPKWAl81SC2IcUAdhAnkBs4 9ssJhn/d/yihKkTRKaPPUrlJP9YKLMswf6WTaza5JFScWVsohzUYqtFpwPd+62hB4x8g XyojSJWdd5oDVs23ySz2DTb4Bn9BuMGvT2uY0fbgF9W70la7zzgSEaIDVQBhi5gl9Po1 LTXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:message-id:date:subject:cc:to:from:ironport-sdr :ironport-sdr; bh=cSZcVe6pZt3Z+p1XyU5Yi1kx7aSEESVhmG+uMlg/uyY=; b=igchXQfdZyz01rOxbkM3cnN1zw8b1LFTdaPLVY/8rm/wWH6A/mtHsU2I7hzEZbzb2J o0CEsHbLqmPpIRFJge9o2JZnb92rGH77XNeuxlbvEGzEKTS6kbO2CtqR4B2BTpD/dImL 9VNXtDnu9XKijOcgoinYndBA0U+Aw9PYwSMWeFkHsUAybQyRHQDIjk55vWAtkcWKfs2N N/776xh54IYliaQhwRUKlMxWLKXjP0hW+dZDXEnCu0xCgHH25e7yzj4nPLnGm7sgfCfy P3nTZ5J0VvenmHtp3/zvqR3Q2U1Pcac624DwBJM0bQSvgVf9SHsRBrHtUAJecNpJar2b QWMg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.137.180 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com Received: from esa3.mentor.iphmx.com (esa3.mentor.iphmx.com. [68.232.137.180]) by gmr-mx.google.com with ESMTPS id v7si292239edj.5.2020.11.26.03.19.47 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 26 Nov 2020 03:19:48 -0800 (PST) Received-SPF: pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.137.180 as permitted sender) client-ip=68.232.137.180; IronPort-SDR: sNM/W1s/6i06kfY+ez8q95mhLwcTDIo9OeUycpJqA+RB1bgq0E1QnY6jgGIn9NeLmbJ1LxA/Ho d7kLFuT1DGO0ycEZ84WOomkKhL7tRgpa9LCbcQ13+TyETqRQ5SS4H46m5cRghl6qbK6H2XzSQM /y7fwT+RlnpLfbqIHHfJG8IeZXv/WLIaqzV7IRDL0b1o1QNuSghCwj121WMmMganqeZKbOitIJ InEK34ok23vzWpm7CCaAPO36SWw/o3O+pRf5Bs6ci1VCJdC57cKm2nsZEEIs3E9PXEhy9TjZir orQ= X-IronPort-AV: E=Sophos;i="5.78,371,1599552000"; d="scan'208";a="55500394" Received: from orw-gwy-01-in.mentorg.com ([192.94.38.165]) by esa3.mentor.iphmx.com with ESMTP; 26 Nov 2020 03:19:46 -0800 IronPort-SDR: +F7ATVPmS5MbM6x9lJCu17/X+Aj/w+bPIYzzVq3mJEM3MQLHglbvaSo+pNdswRcukrbBUrQjgZ gDMk2qioOAP/3FAi5eD2zXHWl55GZSdegujUCKduxsnB5c/qgvSjgllpqZb9sogHq3l/WPx/gU +tunybTT5aDNP1EpJf6mo5X/5popDno+JiDt0eVidbLMAjc3yKFN4EJ6oo5OHAc0IvDGJ3cV3x T8kWlSYO/acjIqgZjDmJ0xhHYErcorngP2NCWY8jAaFpVfVixD2knW/PXm7uILyRO6tqo/Ruo+ 56Y= From: Vijai Kumar K To: , CC: Vijai Kumar K Subject: [PATCH v9 00/14] Debsrc caching Date: Thu, 26 Nov 2020 16:48:58 +0530 Message-ID: <20201126111912.3037-1-Vijaikumar_Kanagarajan@mentor.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 X-ClientProxiedBy: SVR-ORW-MBX-09.mgc.mentorg.com (147.34.90.209) To svr-orw-mbx-01.mgc.mentorg.com (147.34.90.201) X-Original-Sender: vijaikumar_kanagarajan@mentor.com X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.137.180 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: =?utf-8?q?1684421571226644413?= X-GMAIL-MSGID: =?utf-8?q?1684421571226644413?= Changes in v9: - Address review comments from Baurzhan - Rebased on latest next.(no conflicts) Changes in v8: - Fix documentation as per Jan comments. Changes in v7: - Document details in user manual. Changes in v6: - Rebase against latest next - Address review comments from Henning Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3 Changes in v5: The major changes in this series are as below, - (P2) Handle cases where HOST_DISTRO!=DISTRO. This avoids mixing of debs from different distro which inturn helps the src caching logic. - (P5) Avoid downloading package from other builds. - (P8) Add a reprepro based sanity test to check if the repo contains the sources for all the debs. - (P9) Introduce a new variable BASE_REPO_FEATURES which provides means to enable or disable various base-apt features. (cache-deb-src for now) - Some fixes in (P12 & P13) Also, addressed some review comments from Henning. Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc5 Changes in v4: - Use = format instead of just using to download the right version of source package. Changes in v3: - Take care of non-existent downloads/deb-src directory. Changes in v2: - Introduced additional patch to cache deb src - Rebased on top of henning/staging4 tree Vijai Kumar K (14): rootfs: Make rootfs finalize a separate task deb-dl-dir: Cache host distro debs separately meta: cache deb srcs as part of postprocessing deb-dl-dir: Make debsrc_download faster deb-dl-dir: Download files only belonging to the current image deb-dl-dir: Factor out the mounting part deb-dl-dir: Fix skipping of removed files repository: Add a sanity test to check missing sources base-apt: Introduce BASE_REPO_FEATURES repository: Fix failures due to missing section scripts/ci_build.sh: Enable deb-src caching rootfs: Fix possible overwrite of existing resolv.conf rootfs: Handle failures when postprocess is rerun doc/user_manual.md: Document details about deb-src caching doc/user_manual.md | 30 +++++++++ meta-isar/conf/local.conf.sample | 4 ++ meta/classes/deb-dl-dir.bbclass | 61 +++++++++++++++++- meta/classes/dpkg-gbp.bbclass | 8 ++- meta/classes/dpkg.bbclass | 8 ++- meta/classes/image-tools-extension.bbclass | 4 +- meta/classes/image.bbclass | 43 ++++++++++++- meta/classes/repository.bbclass | 13 +++- meta/classes/rootfs.bbclass | 63 +++++++------------ .../isar-bootstrap/isar-bootstrap.inc | 12 +++- meta/recipes-devtools/base-apt/base-apt.bb | 3 + scripts/ci_build.sh | 1 + 12 files changed, 196 insertions(+), 54 deletions(-)