From patchwork Wed Nov 25 03:42:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vijai Kumar K X-Patchwork-Id: 206 Return-Path: Delivered-To: ilbers.mnt@gmail.com Received: by 2002:a4a:df02:0:0:0:0:0 with SMTP id i2csp371366oou; Wed, 25 Nov 2020 05:43:08 -0800 (PST) X-Received: by 2002:adf:f607:: with SMTP id t7mr4202648wrp.169.1606311788263; Wed, 25 Nov 2020 05:43:08 -0800 (PST) ARC-Seal: i=3; a=rsa-sha256; t=1606311788; cv=pass; d=google.com; s=arc-20160816; b=UvfK/iFxQYG879kcK8+IsoGq/gEt67IioKhCl93ApuJBdf/Edr5zhEWYo278EltaM2 /KrMyd1wZhelZ/Qz7rNg6V/hJd8JDD9ofctx4lfGJOmFMSMl3U1JLcObvQJXCe0oQmFb b+JgCWEpymvOGGSjmoWN37/2olrkDa+Wp4Q1oFjI/xRk/e/SJYP1Os6alKAwvi0fFnRJ StSfia6aqVe3o8XSm6bkVCrVuQQGsBdXkWZk0nqggnr4VjCkNPUKs62y6LdOgaLZ7FLB g25+/HIFZQpA3XAsRPB3MjPvqw3ljK8XKwgmX/UJ5XJJSTN8Cfm1dIg1PN1jCcS7EcdQ i4WA== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:ironport-sdr:ironport-sdr:sender:dkim-signature; bh=Nt4yVVl9cJxXuSodGqHvA7m8NaX0XN6rLfhg68w5CHI=; b=uXYjSEioN0WA/JI0a2+x+FcK66h6Ybo3JSZk1euW07eSa2crLjwcl/0MD2Oa9cmqbU 8taub5Stkose07ZaVSUW77k6CriUqMEdOTNNr73sRvzDGevJLgR9l6jvBnhgwAbDEAEY YHs0vB5oLtO1LIefFhlHfrIlf+HVnYaEkoctSGfBLedK9KXJ19V4BuVHLhB2ow6deS+o irmfZ/Ezd0mI7C8D66rAHajn7H2yPHeJrqJNAdKfAI7bEbBINKKCj+hS9/xyakujG1xX 8OrGmFeAwqAWmJmzISBCPMwGbvan2ThhHhCkH9eh8LFd61EYs1CCn8OAX4omggdJBtVf j2JA== ARC-Authentication-Results: i=3; mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=R4R6fbzZ; arc=pass (i=2 spf=pass spfdomain=mentor.com); spf=pass (google.com: domain of isar-users+bncbcwp7lf37ukrb2567h6qkgqevtulk5a@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCWP7LF37UKRB2567H6QKGQEVTULK5A@googlegroups.com Received: from mail-sor-f55.google.com (mail-sor-f55.google.com. [209.85.220.55]) by mx.google.com with SMTPS id f3sor951684wrm.53.2020.11.25.05.43.07 (Google Transport Security); Wed, 25 Nov 2020 05:43:08 -0800 (PST) Received-SPF: pass (google.com: domain of isar-users+bncbcwp7lf37ukrb2567h6qkgqevtulk5a@googlegroups.com designates 209.85.220.55 as permitted sender) client-ip=209.85.220.55; Authentication-Results: mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=R4R6fbzZ; arc=pass (i=2 spf=pass spfdomain=mentor.com); spf=pass (google.com: domain of isar-users+bncbcwp7lf37ukrb2567h6qkgqevtulk5a@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCWP7LF37UKRB2567H6QKGQEVTULK5A@googlegroups.com ARC-Seal: i=2; a=rsa-sha256; t=1606311787; cv=pass; d=google.com; s=arc-20160816; b=qzEBwhwuz6eWzIBLBPYDqVtBwyE55c8NufliWOpAJgQ4O+OMQFjTmMSyPzBXuE9pka Z5Mk65RpdHJytGqHmAv8Ur6yOl02KGIRGqLgUkYcmlbw3K0umVnnc1HksqDqOgv2HVHW 3BUI/uyJTmAcEQqsOMtif90Gi0VYWAjLx8fPWAneHpj/4BY3Ruz61zu23EsJd1jMZNwz 7x+j2PCWCm/pbIL6/yJO6PIcRBgH8P/tAxqJA3DOFOGsgw/zdMz8pR2pHT/kGX8kbdjf CjZk2l4u8gngVBk+pWvyW7fikIPygC2ZjjBFAKsBb1nFXRsVoMonboeHIcYI/nXJMmSa OqCg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:ironport-sdr:ironport-sdr:sender:dkim-signature; bh=Nt4yVVl9cJxXuSodGqHvA7m8NaX0XN6rLfhg68w5CHI=; b=MJQakjlx6WhAQ+/C2J38jG++jeHXukHj/oztWsqft0Is8HAO43TNEboRNp1Ht/ScYc D2de9Gzicsx0Mn1cG9uaNZlJAiZ2dfmwYr8XCPlBlxhWXw8bSX4e9f29syF6uozufGRr UefZCG+AnmPLSfHRg+uTtLMrapjD/N7XKxnuWAtN4MxwAVBWQHYFIpK3lKAR6ePi/WbP Cs9GqW9GqRlHc2NLzPHmLuZ6chtXkmiNxyhAr4/dHrXpj6H4tlaoBNeh6J7YBk80fITu hwmg8tdlY2Fb+O+mhf1Z8Bc/TQIzekzjyCYD/ezNCaDdbO143o4pJ3E9V+mgI+av7anY kYww== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.137.252 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20161025; h=sender:ironport-sdr:ironport-sdr:from:to:cc:subject:date:message-id :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=Nt4yVVl9cJxXuSodGqHvA7m8NaX0XN6rLfhg68w5CHI=; b=R4R6fbzZe4WPbPILbG0siwE3MEmFKpF9bKnu6/rcqpCiJ3HLtMmyvJ3RMFcIzPczhP +IdNX06hd8MGvzOM1ALiomzosSenWDptSHTf1CdyGPoZfKTVBUHmEN0QamnhV8H+p3K4 FiB95gXAdgad9m4uNeI7HGw1Y71PHHrNOk5LXceJsf/FfdLB1EuwW0pudGn+YYyqkDGn zlgYUdE9yU5edDKKHX34MRG20cg/VDzWV/8xAyKHccprbmU4qZtQE3MBh9itAbGn6BBA O4suhLrkF3j2zTizmr2jXY8j7GQYtuiT2C0P+z/PP5sQoMbJNnPEdWIw4HgAWgffadwU +EWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=sender:x-gm-message-state:ironport-sdr:ironport-sdr:from:to:cc :subject:date:message-id:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=Nt4yVVl9cJxXuSodGqHvA7m8NaX0XN6rLfhg68w5CHI=; b=FWhP3yFV6Ex2K16b2ik0oQkf3/1YCisZ4TcYfSEcgg07rT0xkj4PML0z1QYcbVnXBo aEEcXuJX+0T5OxnM5UkTc2G6Sse+UPaf7k8Ml+X47z/Vw7b1C/mw2JZ5ZJKQ5u+Ie4rw zT4sbBJ/dGZensJjz+BjVWCwQTEx1WNf+WXAvcWH9FzOAw5VMERKYAu4YneKn7+I+CJx oW33ey0FG+zBXScBoqNzlLlYiUXV4H3vrAYd2kpW2a3e51By0S5Pw2ucJE+Iy+CSz8R9 zmq/sHTOT03nNdpO+u/53uEtdVplpryNnM+S+/1O+go9KYbWAD3tpJXr96xpQJ3u/yRU g6UA== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM532/mWyNB+6p46692TCMTmjCPfZO1cX4e2/llEVHQ4ybQnW/QEP3 c1bT+/dxG8epADcZC1OgddE= X-Google-Smtp-Source: ABdhPJyKjBWHpLP3cqJC3qzxUM/rJgfIEOFzhky5Jo8iP92d1ghIXBMs01cvDkTkEoXDcL4j1ew5gA== X-Received: by 2002:adf:a495:: with SMTP id g21mr4243858wrb.213.1606311787381; Wed, 25 Nov 2020 05:43:07 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:adf:c689:: with SMTP id j9ls2845619wrg.0.gmail; Wed, 25 Nov 2020 05:43:06 -0800 (PST) X-Received: by 2002:adf:f110:: with SMTP id r16mr4200347wro.350.1606311786266; Wed, 25 Nov 2020 05:43:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1606311786; cv=none; d=google.com; s=arc-20160816; b=rbk6/jEH9Zy5SI1OpLLm30IAssA398u6zuSYO88Z7zxoRz8zCOGN+8odaasOmtIIHn 75ooya7txATKz9cvhl+n8F5xVXPCq6y8WPQkfOKl622LFFaVJlOU1U0u5c9jYk79Hs85 X/75ioPEW9MJqJJD9daZQOJ5GUiI8+vA0lfR2JLC0qFOdirU8nPROodaLZNtOzj3rTZI tu++xdJwvLna6IuvmwOCgZ0Tabmzwnm62vX0akqZ/HSeIcRk27MOKkU2dTmCQqyyjLHi inaASE23NrvCGYABpoInGY1bej3xYoQRnhP/5qpJeQlnLnHg7qPwHW2rcNtzA14GRW9q ZpvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:message-id:date:subject:cc:to:from:ironport-sdr :ironport-sdr; bh=wZeW3CJvQhL6fc5T/AfCCnWzfNLH9XO4C8SYLdgaOJg=; b=qImlVDSJq02xaOS9kPrl+IYXPVUdPv/G3onUXwHygOIa2+PBSEUPKhhnJmjdad0g5j Blni18CgiEPVhqaCVa4Y1QBm98u2CeUBqIVwQhnlIBbhu2IUBqR2EhrdavvL8qd+TYws 9KFXBG5cbRjY1uyAxG1XK/lw4Sr85OGpbmqpkSNyeNOCy9BWnLqHFx6cO8uE9uGndP0u GzNte0lG9UFBjNi1ANwQV6pNgHxXHslbB8nfLCEFrPKniXWvfm/HreFE93nPYBcCSOHt BQKdz78NDuAQSTJ5kAaL72rlrTlxD0rRNb8l70XAugGqQ/46Bbf8LxLpwkV5hgVPw7HQ x6nw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.137.252 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com Received: from esa4.mentor.iphmx.com (esa4.mentor.iphmx.com. [68.232.137.252]) by gmr-mx.google.com with ESMTPS id c20si60917wmd.2.2020.11.25.05.43.05 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Nov 2020 05:43:06 -0800 (PST) Received-SPF: pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.137.252 as permitted sender) client-ip=68.232.137.252; IronPort-SDR: NuhmvQ9HxikZAgBbdBwRNGEL4PVa08ttDuVJ/KbdIO8yyLyyVKOtO+bJVDAi+pjQLgd3fbKbRG /eJIMsovdMyvY4v2V7s1LX56dgKF4Os4qj3CzrnoFzMAHUiye0Sx7Glg7nLXLBlGuWyDR2bVNl xWDp6beK9orcC3o7AuLWFDuJmKPzuA1/7KTUwYW5B6wKgrkhMLS7WNSgnWQUteT9y4UeAcliGB cO5lnKoWhVtYasTMw97ieGrM6YckqMm36pJJBbdotoZVoJo+XRAs+mY6YjeXMfabKXDMvMWmUl ewE= X-IronPort-AV: E=Sophos;i="5.78,368,1599552000"; d="scan'208";a="55597743" Received: from orw-gwy-01-in.mentorg.com ([192.94.38.165]) by esa4.mentor.iphmx.com with ESMTP; 25 Nov 2020 05:43:04 -0800 IronPort-SDR: inId7VggM63sUQMrSJjQYA6+Hje/hQmVRpugWZPlPj7TEl/vGxKS3xkaHj0Idk380Zn9CAKAeY HxBGNiA0tZQ9EdP6FouYNjRLWDCUR6Zm6bzv/9Vjjsl+DVCdbXbLHqZrrI8kH5aG4BblFuTBmE QIlRFxTHlTY/ZGkP4RfYzvBnVi4mrCz2ob1J3q8GvaJQvHSVHzIOp3KpqFMUbG2gGDXvUUgRSR bWEsHXMhFcMcz03PseURmJcKNyEEuKgzyLI24yKNBN8ymPK0y9P18GYys/3dR+qS8e2Gr9FTXF Bek= From: Vijai Kumar K To: CC: Vijai Kumar K Subject: [PATCH v8 00/14] Deb-src caching Date: Wed, 25 Nov 2020 19:12:22 +0530 Message-ID: <20201125134236.25878-1-Vijaikumar_Kanagarajan@mentor.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 X-ClientProxiedBy: SVR-ORW-MBX-06.mgc.mentorg.com (147.34.90.206) To svr-orw-mbx-01.mgc.mentorg.com (147.34.90.201) X-Original-Sender: vijaikumar_kanagarajan@mentor.com X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.137.252 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: =?utf-8?q?1684318619610298412?= X-GMAIL-MSGID: =?utf-8?q?1684339989940291734?= Changes in v8: - Fix documentation as per Jan comments. Changes in v7: - Document details in user manual. Changes in v6: - Rebase against latest next - Address review comments from Henning Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3 Changes in v5: The major changes in this series are as below, - (P2) Handle cases where HOST_DISTRO!=DISTRO. This avoids mixing of debs from different distro which inturn helps the src caching logic. - (P5) Avoid downloading package from other builds. - (P8) Add a reprepro based sanity test to check if the repo contains the sources for all the debs. - (P9) Introduce a new variable BASE_REPO_FEATURES which provides means to enable or disable various base-apt features. (cache-deb-src for now) - Some fixes in (P12 & P13) Also, addressed some review comments from Henning. Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc5 Changes in v4: - Use = format instead of just using to download the right version of source package. Changes in v3: - Take care of non-existent downloads/deb-src directory. Changes in v2: - Introduced additional patch to cache deb src - Rebased on top of henning/staging4 tree Vijai Kumar K (14): rootfs: Make rootfs finalize a separate task deb-dl-dir: Cache host distro debs separately meta: cache deb srcs as part of postprocessing deb-dl-dir: Make debsrc_download faster deb-dl-dir: Download files only belonging to the current image deb-dl-dir: Factor out the mounting part deb-dl-dir: Fix skipping of removed files repository: Add a sanity test to check missing sources base-apt: Introduce BASE_REPO_FEATURES repository: Fix failures due to missing section scripts/ci_build.sh: Enable deb-src caching rootfs: Fix possible overwrite of existing resolv.conf rootfs: Handle failures when postprocess is rerun doc/user_manual.md: Document details about deb-src caching doc/user_manual.md | 30 +++++++++ meta-isar/conf/local.conf.sample | 4 ++ meta/classes/deb-dl-dir.bbclass | 61 +++++++++++++++++- meta/classes/dpkg-gbp.bbclass | 8 ++- meta/classes/dpkg.bbclass | 8 ++- meta/classes/image-tools-extension.bbclass | 4 +- meta/classes/image.bbclass | 43 ++++++++++++- meta/classes/repository.bbclass | 13 +++- meta/classes/rootfs.bbclass | 62 +++++++------------ .../isar-bootstrap/isar-bootstrap.inc | 12 +++- meta/recipes-devtools/base-apt/base-apt.bb | 3 + scripts/ci_build.sh | 1 + 12 files changed, 196 insertions(+), 53 deletions(-)