From patchwork Mon Aug 24 01:27:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vijai Kumar K X-Patchwork-Id: 7 Return-Path: Delivered-To: ilbers.mnt@gmail.com Received: by 2002:a05:7000:4605:0:0:0:0 with SMTP id b5csp2149834mae; Mon, 24 Aug 2020 02:28:20 -0700 (PDT) X-Received: by 2002:a1c:9910:: with SMTP id b16mr2391242wme.79.1598261300127; Mon, 24 Aug 2020 02:28:20 -0700 (PDT) ARC-Seal: i=3; a=rsa-sha256; t=1598261300; cv=pass; d=google.com; s=arc-20160816; b=zHegAP7i+ggjtKY9R5SCNBCg2FgOHg5LUZPHLf7+vDd3fUE4X14MqzSlqTHWIMVT29 91dNOQdGXrT1dZ5STJVVI7mz+QxqrwKngQLBSSjt0aw40VjG2Z+xUGYbiHfRHm+zDpOr 2PrqGXC8x9d8o34NF9i4905RhJvJlKDJGDbG0j/uQR3ObxsS4s3SExYQjNnrlhwOkqbp VjTxFE4LpyOA0M+4tYuMXP9CLbjB4txWr2HSW2RX2GOqyBP4ZW8uE4EPb1q3qbIZ8f4I I/afyWp09EegksVXQ9xEqaVvC7rzX44eq6LtKSJDNLEwSbPLeSuQMieW7uV1Wi/L49zd n/CQ== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:ironport-sdr:ironport-sdr:sender:dkim-signature; bh=KueQMgvPoLQB50Fltim9OqewcLQW+Dof6vdcAcKBAZg=; b=JvdnZ1Ezqa0cRnoCn3BTomwP+jeqzTishseGlbRlG4VudvBZUWmucHzIA0GJqMfeDv gKSSldhvc/4x4eP/XZvwgnK7Wba82Ug4ZZs71Js8ahLKS2At9kgHckyhavVKHDNIoUFb 6/u66E6cifqRlTy+s+PeuYf72VU3SbgTuE7oT61+wHE2KW/GljThQn73hQBQZC+dR52S nhokjv4pqfxqnfCW1a0DgQUAR4URUFwrGlW43DMSiWtqEpedrBR7RoCox3owlCG866I/ oMuY3Fc/NwuMcTOAJTSCJ8Gsj1eWP/MzcCtGuSkdVH0+e/60/yA/pEg0npK2QC+g1fvE eS2w== ARC-Authentication-Results: i=3; mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=n4xvdxeL; arc=pass (i=2 spf=pass spfdomain=mentor.com); spf=pass (google.com: domain of isar-users+bncbcw7ho6fuybrbm4qr35akgqeoydnboa@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCW7HO6FUYBRBM4QR35AKGQEOYDNBOA@googlegroups.com Received: from mail-sor-f55.google.com (mail-sor-f55.google.com. [209.85.220.55]) by mx.google.com with SMTPS id o10sor4188046wrq.65.2020.08.24.02.28.19 (Google Transport Security); Mon, 24 Aug 2020 02:28:20 -0700 (PDT) Received-SPF: pass (google.com: domain of isar-users+bncbcw7ho6fuybrbm4qr35akgqeoydnboa@googlegroups.com designates 209.85.220.55 as permitted sender) client-ip=209.85.220.55; Authentication-Results: mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=n4xvdxeL; arc=pass (i=2 spf=pass spfdomain=mentor.com); spf=pass (google.com: domain of isar-users+bncbcw7ho6fuybrbm4qr35akgqeoydnboa@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCW7HO6FUYBRBM4QR35AKGQEOYDNBOA@googlegroups.com ARC-Seal: i=2; a=rsa-sha256; t=1598261299; cv=pass; d=google.com; s=arc-20160816; b=XApEQYEooE5ADM2HhZEXVn5ILmB873JXBbTT/GgzfAQzO4kOiM/vfBYTwOatIRjJw1 oD4Sf90rizmwSUVMTeH6XibkKPSkyQ2ICuZpYsV1VeQu6ZxRvn2FR7XVBMOjHsPFrAUo Cq9JxzVnT2/VBpO99MNGFuY3mpcdlWaZ22+GROVjx6URvtMP4nm3BYbPx0OuBxHgzkkD oi+l8wYMiTmdf7MXfiEM76VFvgcs2f1jtpviB0vPaLheYGO4BXI1mdfBtbFe0BNtbzXZ FTmPrEY5GBfLo9XNjLYJ1ZcZ/wb1bcvPURwqc1AHRSazlR52PWQjZtJVzXsInfcf2+nS jDQg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:ironport-sdr:ironport-sdr:sender:dkim-signature; bh=KueQMgvPoLQB50Fltim9OqewcLQW+Dof6vdcAcKBAZg=; b=g+WE/edxbQ1LcUXS0gbCcmC2lK1mNAtMZbExZKPKXoi69ZUG5cpR/BN73nNRQQVIDr qoNeZ7JmMSarKK5SJxpc2fxl0L6aJlQwiClE+vru5jgy54kjL1Y5+OF3DOOOjZ+4cTtr DRy4Jeoi68SprhZWlWMIVF6klRmyVd8Gss1xne0KJ20iDy+44z/Ekzdxyvvmguvsc696 eFrpf08+zF5hOUiz8FR7sIirnHJJyuIu1YQOZE5lGc4Qq7OBZDAAhVTO3pqhvHmfbOhz 6WAEoa0+DcLCP+ziKusgVprWMs8+9R+OXB4vGZ8zOlsnU8Wkv216dCJTaczUefz6JO8Q Oztw== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.137.180 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20161025; h=sender:ironport-sdr:ironport-sdr:from:to:cc:subject:date:message-id :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=KueQMgvPoLQB50Fltim9OqewcLQW+Dof6vdcAcKBAZg=; b=n4xvdxeLPW7kCxY2Uw828PUpBzdpNU/bg7MRDjExFS12C3AjuPW5FsmdvuF8xQu1PX Wu77rSYFh6cjUS0epy7j/Yl4+wOriiDRZu4oT9At9XBdBqnofNHuxm6deuW1Hj8Ye6B3 0XFQQNFxMmkS+S/1HrQIUTUCFy2a2mw50sISNK+gLkNyjP3NvpoaLb/L01L+zMyErxXy wx7D+zN0J7DTKBUYMbbmzfNYQ+YHlZgwFmxKadJplgKbUE8EeyrIy598uDJbJ7d8UA8u ng5ahz50uaE1IJ4nu4pV56NWryFPkyBvIHDQzWVKRhOQazzzlFgzGaB8K5eNS+bMF5y4 eVtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=sender:x-gm-message-state:ironport-sdr:ironport-sdr:from:to:cc :subject:date:message-id:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=KueQMgvPoLQB50Fltim9OqewcLQW+Dof6vdcAcKBAZg=; b=d7YmBusZwtmN32DxwAKz0CTg3wBIoYZuKBEZj+tpImANQkfLbvji8q7JbVwBEk+Jw3 5kbdApxtL601ZrGWi1P4ZGh+pPUSV8NAuC8ZRZXfUMjwsMLZrHIgYF0RsQJW2O3GDp79 PSRfXLCL1sNcWO9XQtYL+1YgAuE/mBxzGr+yknwgLlmkQP1XQf2ed5TKxY4ObRDuAYyJ PN+x+bDoTD86KgPlPItmo+vsnSrLoj982VgCqgo/sXcddOf3gmRLP9/tyZy24uPqkTcU cQ/6hOuq8eI9Le/bEt93XYnwQTWOTj3JngaHFtXvbJsUNPI7Y0/LjRMoGehd3mTmV97U tX9Q== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM530aDDongUeUVeoZ0ZEPriY4GtzjVIVw+7kZCrfA1njXL+/YyE50 88Z0j3LqsjnEUhgvmNCR3j0= X-Google-Smtp-Source: ABdhPJwH5PMBwA9+Kz4n44ogC2LDyKfRo2C6m9X8Ygslv1dqy8L33tgHhKansYM5ND6TBY3rU2dJ2g== X-Received: by 2002:a5d:67c2:: with SMTP id n2mr5255768wrw.326.1598261299520; Mon, 24 Aug 2020 02:28:19 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:6592:: with SMTP id q18ls392731wru.0.gmail; Mon, 24 Aug 2020 02:28:18 -0700 (PDT) X-Received: by 2002:adf:e812:: with SMTP id o18mr4805058wrm.29.1598261298935; Mon, 24 Aug 2020 02:28:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1598261298; cv=none; d=google.com; s=arc-20160816; b=RV+GNidohOfLbfdpApu7Jl++Xms+d669vWuh1N44Wfu+DlIC6oK69rIw8EHHWG0159 x8cWUFLZZ+1euReq22OJ2oWOzBfvq0BPRxInsF07jZhqMZpKzawsCLnzuYsh/HuTVOcn /LLBHgxPQozcrDOapeKWLAX5DsZgEOPb1lEA4kcKDFrAbDHlMfi2pjYYtxZHwZlewmtY YhXZp8ZPX+VgHH3tua9xtkoWebJQlCd4Lh9ezVW0PhRjpdeZ+UPecVMkr+LFdGyTUYM/ pZFNUdoleW9WpoLyleaYY5/0knt6halir1GD1A87q4sOQrIx24h/Isr1S0m0432bedKr jlsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:message-id:date:subject:cc:to:from:ironport-sdr :ironport-sdr; bh=Mv85uo/2n/+r9yNAkqFKswHnNfaADBgAymoPwMYZyao=; b=vAJMnchPgylfvDgHLV1aeIuFQsneZ2Uz5JcPUjrJFFAY5WW2tDpms9/7d/h2WWeWqb jW2nSVIS1Gbzo8nyfdN1xOgbTn8H9v5Xis7iVdjaFWGyXOZO7dSgBCT9378O+AwAP5eX AXO6yj02dDry50dlY2/YY2XF/QAgPEiv/2QmdfD9xx7/5mnKDlbMI3sAxy8CTYP8Ohvi ErCVhI5ynxsR4ToiY3exuFhAd2y80NRjhNxNNkf/5tDGkX1GM0+fheh0PLQxrtTOScPT 3euBeu5DIXf64lnCApnvPKbf0mm/O89oAZXo3aYpCQmKzZc1LdSQ1+tKMNBj4hf5G3N2 97nQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.137.180 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com Received: from esa3.mentor.iphmx.com (esa3.mentor.iphmx.com. [68.232.137.180]) by gmr-mx.google.com with ESMTPS id i11si325166wra.3.2020.08.24.02.28.18 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 24 Aug 2020 02:28:18 -0700 (PDT) Received-SPF: pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.137.180 as permitted sender) client-ip=68.232.137.180; IronPort-SDR: osH15P0Ii9ccPgQNlQBtL3Ye3AAytLRWO5nBiGEY8iLlYLhqyqm3BiBfhX/ofne8C6Ucrb5dkI C1948Q94z2yji/6AuH5zvIyDcGSKzfFrpWP1ugAPRDwJLqPy8+N2+bauNU8m6HbjUFboDYowqO 9YdvAdxBWzcidP54JrOQ9oRrOsm+ZY/Jv6WE9cKivM+sPGWFU2wYDZL5cW5aWQUbH8mfqyS6cb x9VUlJQrUg3WzLd25ssiPS5zlVTLnCGhnCjmH7sMpZZ3rJ0eEsJkVju/E4rHjHbP3Va/WT36B0 WxQ= X-IronPort-AV: E=Sophos;i="5.76,348,1592899200"; d="scan'208";a="52173147" Received: from orw-gwy-01-in.mentorg.com ([192.94.38.165]) by esa3.mentor.iphmx.com with ESMTP; 24 Aug 2020 01:28:16 -0800 IronPort-SDR: 8UhbcyawtfhTFnayjw1X0gpSn6lHze0gIb3ZVuh4yZzIAr4rbsLSFaDt1bKy6ZlWKCrGdDCGFy 7myxSCEaFumf0RXIEjiTGcvpY/0/wlHQTDs0vbxl1PA9gnU99oaM7HRB8G6Rri4IxFRR6tQV6y RXd8tQtcFuglqDSgb8k71Ie9mqvBWih0SMXkJaedp3LUOWj+3sYFY7rvzuaGQv0SwyTA+huj3y okKaAa8D1ahMlZU9g9GbAqvhC6bZnwEdAu6ZZcZancR796CRBgTmhbg7cZ/7u9C1ZcT/D606ES fCs= From: Vijai Kumar K To: CC: Vijai Kumar K Subject: [RFC PATCH 00/10] WIC update Date: Mon, 24 Aug 2020 14:57:40 +0530 Message-ID: <20200824092750.19642-1-Vijaikumar_Kanagarajan@mentor.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 X-ClientProxiedBy: SVR-ORW-MBX-05.mgc.mentorg.com (147.34.90.205) To svr-orw-mbx-01.mgc.mentorg.com (147.34.90.201) X-Original-Sender: vijaikumar_kanagarajan@mentor.com X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of vijaikumar_kanagarajan@mentor.com designates 68.232.137.180 as permitted sender) smtp.mailfrom=Vijaikumar_Kanagarajan@mentor.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: =?utf-8?q?1675898441118360991?= X-GMAIL-MSGID: =?utf-8?q?1675898441118360991?= The following series updates the wic to the latest one from openembedded core. The git tree is available here. https://github.com/vj-kumar/isar/tree/wic/staging0 Below are summary of key changes in the below series. P3 & P6: The latest version of WIC detects the presence of fakeroot using pseudo_dir(${IMAGE_ROOTFS}/../pseudo). If this is non-existent it doesnot use fakeroot. P3 handles that quirk. There is also a standalone invocation of pseudo which P6 tries to handle via wic_fakeroot. In case of ISAR, we need wic_fakeroot as fakeroot only for 2 purposes. 1. To workaround fsck return values for stretch 2. To handle standalone invocation of pseudo(P4 in series) Number 2 is not needed if we dont use fakeroot. The alternative approach to fakeroot is to carry patches on top of wic directory instead of working around such quirks using fakeroot. The downside being the need to rebase the local changes on future updates of wic. P4 & P5: These cherry-picked changes address the --exclude-path issues which Henning has addressed in [2]. Should be dropped once that is merged. P9: P9 is needed for --include-path to work. I have observed that --include-path does not yet work/broken in the latest OE-core/poky. Posted a query in [1]. In ISAR it works with P9 fix. P9 fix could also be relevant upstream and can be sent there for review. Apart from the normal qemu build test, Below are the list of features tested. KS API: --exclude-path --include-path --fsuuid --overhead-factor --fixed-size --use-label --use-uuid Standalone invocations: wic ls wic cp wic rm Currently in the process of finding a way to test wic features automatically. Some wic features like -fsuuid cannot be validated till we boot the target. Need to find a generalized way for testing the available wic features. The approach in hand is to use a wic-test.wks file and generate qemu image with that to validate the wic features. [1] https://lists.yoctoproject.org/g/poky/message/12151 [2] https://groups.google.com/g/isar-users/c/xePCyEl2qhM/m/mwFdeou0AQAJ Henning Schild (1): lib/oe/path: try hardlinking instead of guessing when it might fail Paul Barker (1): oe.path: Add copyhardlink() helper function Vijai Kumar K (8): wic: Update to the latest wic from openembedded core wic/plugins: Fix wic plugins to work with the latest wic wic-img: Satisfy the quirks of latest wic wic_fakeroot: Handle standalone pseudo invocations meta-isar/conf: Add provision to debug WIC debian-common: Add tar as a dependency for wic scripts/lib/wic: Add /bin to the list of searchpaths meta-isar/canned-wks: Remove /boot mountpoint meta-isar/conf/local.conf.sample | 3 + .../lib/wic/canned-wks/common-isar.wks.inc | 2 +- .../scripts/lib/wic/canned-wks/hikey.wks | 2 +- .../lib/wic/canned-wks/sdimage-efi.wks | 2 +- meta/classes/wic-img.bbclass | 15 +- meta/conf/distro/debian-common.conf | 3 +- meta/lib/oe/path.py | 27 +- .../wic/plugins/source/bootimg-efi-isar.py | 2 +- .../wic/plugins/source/bootimg-pcbios-isar.py | 9 +- .../lib/wic/plugins/source/rootfs-u-boot.py | 2 +- scripts/lib/scriptpath.py | 32 ++ scripts/lib/wic/__init__.py | 14 +- scripts/lib/wic/canned-wks/common.wks.inc | 2 +- .../directdisk-bootloader-config.cfg | 8 +- .../lib/wic/canned-wks/efi-bootdisk.wks.in | 3 + scripts/lib/wic/canned-wks/mkhybridiso.wks | 2 +- scripts/lib/wic/canned-wks/qemuriscv.wks | 3 + .../lib/wic/canned-wks/qemux86-directdisk.wks | 2 +- .../lib/wic/canned-wks/sdimage-bootpart.wks | 4 +- .../lib/wic/canned-wks/systemd-bootdisk.wks | 4 +- scripts/lib/wic/engine.py | 421 +++++++++++++++- scripts/lib/wic/filemap.py | 170 ++++--- scripts/lib/wic/help.py | 401 ++++++++++++++-- scripts/lib/wic/ksparser.py | 115 +++-- scripts/lib/wic/{utils => }/misc.py | 100 ++-- scripts/lib/wic/partition.py | 234 ++++----- scripts/lib/wic/pluginbase.py | 36 +- scripts/lib/wic/plugins/imager/direct.py | 175 ++++--- .../wic/plugins/source/bootimg-biosplusefi.py | 213 +++++++++ scripts/lib/wic/plugins/source/bootimg-efi.py | 111 +++-- .../wic/plugins/source/bootimg-partition.py | 153 ++++-- .../lib/wic/plugins/source/bootimg-pcbios.py | 91 ++-- scripts/lib/wic/plugins/source/fsimage.py | 56 --- .../wic/plugins/source/isoimage-isohybrid.py | 185 +++---- scripts/lib/wic/plugins/source/rawcopy.py | 44 +- scripts/lib/wic/plugins/source/rootfs.py | 159 ++++-- scripts/lib/wic/utils/__init__.py | 0 scripts/lib/wic/utils/runner.py | 114 ----- scripts/wic | 452 +++++++++++++----- scripts/wic_fakeroot | 5 + 40 files changed, 2340 insertions(+), 1036 deletions(-) create mode 100644 scripts/lib/scriptpath.py create mode 100644 scripts/lib/wic/canned-wks/efi-bootdisk.wks.in create mode 100644 scripts/lib/wic/canned-wks/qemuriscv.wks rename scripts/lib/wic/{utils => }/misc.py (70%) create mode 100644 scripts/lib/wic/plugins/source/bootimg-biosplusefi.py delete mode 100644 scripts/lib/wic/plugins/source/fsimage.py delete mode 100644 scripts/lib/wic/utils/__init__.py delete mode 100644 scripts/lib/wic/utils/runner.py