From patchwork Wed May 8 14:37:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Mikanovich X-Patchwork-Id: 3535 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Wed, 08 May 2024 16:38:25 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f185.google.com (mail-lj1-f185.google.com [209.85.208.185]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 448EcM96032759 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 8 May 2024 16:38:23 +0200 Received: by mail-lj1-f185.google.com with SMTP id 38308e7fff4ca-2e23b3fae6csf39231461fa.0 for ; Wed, 08 May 2024 07:38:23 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1715179097; cv=pass; d=google.com; s=arc-20160816; b=EsQocDbyDu2Bsr8QTBMv35FMG6A2RFFEDVQI+s1SIJnL5B8XIixRpE7ObCDEW4NdVN jRJ1dEloYdjVbqm0M101c2ECDlnf3DNhNK3w55L8SFkq95ygipPINIVs30gwtgZm21Fo dT+aS8ZR/j+9/B66XsgyEReaCEjHkjxr/ELRXyl3otIN+V8n7YUaScC/GU+hqfOfI3yo gMhUFIoP7ofymwIUY1dDsouFnmZlom+TsRuWrHVCTXrI8D93iWv0dvA0B+A5Zbhe8Sk8 q1qnjMdqMrZxFbnS7NFH2tQghMzv31s2uFHHJ21uPpN7pc6CCTtGhn4Km5vyxvmotfCU gp2w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=AA4JkUdwiG/zMvyQt1ZZaQUulzKjk19zs0gt6At7RbU=; fh=aUNwKOv66jKHoH9vCE6eV1G3ejH8nzhzEchO07PrYeA=; b=lEu3zM3z2f0NgPI+7/zE9Ok2ZQEqrJ3nPCZXKGXZvroMt88T7zk07dB9H0qzQJVsKZ ylxi84Jkb+XiipmONYCxzL0HV84LwCzkH4QSt+UJvAGJsWDSoot1I8k+/icUZyBK1Qa0 lnpYjtOmk1jfPZUAQ0TylRDj++W6AY4Q/MsgATgdeaiIFM+2DocoDxkmij5QlP5SAm/z G+PtzZab75L09xAgT6Ibp6k59mtz9x1+GC97bA540nRgJJHcFpoRf7j8lT4IKbVVD4Y/ lToLpMR3cCQEGApCpKWlb3juCI37+jU2ss15+W9bqFjT49CFvLBxVNJJi8u8pawiYhNi rI5w==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1715179097; x=1715783897; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=AA4JkUdwiG/zMvyQt1ZZaQUulzKjk19zs0gt6At7RbU=; b=PzIT+265C8UuLOvAE2EwTX/DGoznH2FjvHI1PUI9vh4IagfU10uIugO/6TEfxOuEMz 0EKTB+TjkEziZIc1QsYsZnio8PtvJRvvYa9PTMO90f5wqyOPI4bwl567yqecT0jB6sQx jH83Mf1l0vblbcnYpBWTyX7GisdONzHspphP2jE2VPxenJenCJpSD6F2h8SZlmmX6tbv 370AIufchNimippOE2PL1wJLcO6XZKDvQCXasrX7AM2NwRONxY/LVcpI7AbLtAurERNm e/MohAhr8RRulhIL8jaXM1KNW7pOqtJ2i6juYR6jn6WVJ9G9b5bC5oz6X+jmEhjfZnuH EM1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715179097; x=1715783897; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :x-beenthere:x-gm-message-state:sender:from:to:cc:subject:date :message-id:reply-to; bh=AA4JkUdwiG/zMvyQt1ZZaQUulzKjk19zs0gt6At7RbU=; b=UsOaiua1vvrShq0CKcqWztLFEsNPoZULosiBtb/Lgx1jw/QGP6EOrLw8hrqZDCWeGL UJaRH7eV6ShzSLvREFE/KVlfkMnUJkqdYORnx8pykwO7pw7aoW2jae/krx/OCka1B1+d /o7ouyT4jNuAWpecrDG8xtgr0npDwa9HuX1fh1qSYjon0a1OwUNTKug8pvBQCBc5DwPT Ytp2/ksASWzOfn3/WnvwcnOHrDvvdkt6RtPMgGovklSE9pMTqhY8GibMdMzEq1vRKN1Y 3lbog1gZyamzI+0m73/FevZS9kWF6dB/mPUUsx7g93QUGWXmX/L+DDBf9q8SjPftrmxT /ouA== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCUntdL+FgzEcjiuP5gMyhqs0N8C+MyBUG2oVkB4TH85fnAf0YgkV+gGJPJKQTjYkeYfTofWdSsnSVKyl2WqsPLwwdz3y58= X-Gm-Message-State: AOJu0Yzz6XhLdNaqIfl/8JvM/zuD32/1jYixZrLPaJsYSx5jYCX6aftk GHyT46c9mnAwjIWkyhXsIYmTMYK/yvoN31ZnvSFCWeX+846lR9b6 X-Google-Smtp-Source: AGHT+IFeiQaSv9v3vSs0s8OQ8tEV2bsI4Y/rzQ/0IHHw3xP3TQNbD2OrWKKNKaKd03GhdqDJisOA4A== X-Received: by 2002:a2e:80c2:0:b0:2e1:f267:374 with SMTP id 38308e7fff4ca-2e446d830cfmr18679791fa.5.1715179096485; Wed, 08 May 2024 07:38:16 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:900b:0:b0:2e1:a117:3c01 with SMTP id 38308e7fff4ca-2e1bbdb29bfls28265291fa.0.-pod-prod-06-eu; Wed, 08 May 2024 07:38:14 -0700 (PDT) X-Received: by 2002:ac2:4985:0:b0:51b:e46c:19fc with SMTP id 2adb3069b0e04-5217cd47390mr1753846e87.58.1715179094210; Wed, 08 May 2024 07:38:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1715179094; cv=none; d=google.com; s=arc-20160816; b=DwBAz9Yv4FcWC4cVemiCy0EuR1HizQy36PmaWPsHV8wWjh7J9sW8st/sgvvfm/lJub 7fpJ/HUHweewHmn9eI/ahx3wJnhG7NTQPjl5pXGzYoB6F3j/n1JMs/W71DKbSBCLH5uQ FHGfjQdq77Hpz40DVZvLcIM1wUM1+meBlVXi0aPOoJXMYD2RhzD7KrI7/LVHisAZpLM+ XfgJTl+uDCTNIMumbIELLTJRrx1IO5IFZ3tRzWZDYqxFeUztc6UYQcrjbH8od+b6pXeC LhDjROrV6aBGeb4nx4s64d53aGQPPcebERHPtoPj0s1TFMn0ii1XUnxPmyU9WII+kiO+ 9GNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=wBWaNjogBXatQ6VXojTAHTnCTFsh45cCv54UfQByEsg=; fh=/h9QQkzJ8EboVkWg45aWwpaUro6WMavIVd2OhN45RtE=; b=xgqskVgvxEOR7z1mVszYWUjxFZ7SAnlnHe3OjdOtGclCL9Lbi9c6jnha8ly5RJFtIw +ah1tV3mMn5OgKB60aJ+gDVocQxZpeLlrLnOX4JdYRfH5VQhYV+8+pLwnhHTKyPcqOhB coIS1TPA5lpfo+MEPpnvUxUDA600lCzjkT5Bplz+54uNpFIU2leFqB6eQe7UFuvdkkti WYD7X85jVSI31zzn7gdSdWyAd0vH4yqyhM+omNr+/kfxBmpEzU5OLyezPd+zqsxlvr8P Slv2zqppEb5yakv0x34ZfrgG3BeEYdTXK1OdmezRZl5xPdH7jWcCy2fHdT+A2VkQ2bvE Pv+g==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id eo16-20020a056512481000b0051976dcfdf8si430791lfb.10.2024.05.08.07.38.14 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 May 2024 07:38:14 -0700 (PDT) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from user-B660.promwad.corp ([159.148.83.123]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 448EcAtl032711 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 8 May 2024 16:38:12 +0200 From: Anton Mikanovich To: isar-users@googlegroups.com Cc: Anton Mikanovich Subject: [PATCH 2/4] wic: Update to the latest revision Date: Wed, 8 May 2024 17:37:59 +0300 Message-Id: <20240508143801.557383-3-amikan@ilbers.de> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240508143801.557383-1-amikan@ilbers.de> References: <20240508143801.557383-1-amikan@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: amikan@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= OE-core Revision: 7ef767d84d56b25498e45db83bb8f9d9caebeaf9. Signed-off-by: Anton Mikanovich --- .../lib/wic/canned-wks/efi-bootdisk.wks.in | 2 +- .../lib/wic/canned-wks/qemux86-directdisk.wks | 2 +- scripts/lib/wic/filemap.py | 7 + scripts/lib/wic/help.py | 2 +- scripts/lib/wic/ksparser.py | 6 +- scripts/lib/wic/misc.py | 2 +- scripts/lib/wic/partition.py | 47 ++++++- scripts/lib/wic/plugins/imager/direct.py | 122 ++++++++++++++---- scripts/lib/wic/plugins/source/bootimg-efi.py | 108 +++++++++++++--- .../wic/plugins/source/bootimg-partition.py | 9 +- .../lib/wic/plugins/source/bootimg-pcbios.py | 6 +- scripts/lib/wic/plugins/source/empty.py | 59 ++++++++- scripts/lib/wic/plugins/source/rawcopy.py | 9 +- scripts/lib/wic/plugins/source/rootfs.py | 4 +- 14 files changed, 324 insertions(+), 61 deletions(-) diff --git a/scripts/lib/wic/canned-wks/efi-bootdisk.wks.in b/scripts/lib/wic/canned-wks/efi-bootdisk.wks.in index 7300e65e..2fd286ff 100644 --- a/scripts/lib/wic/canned-wks/efi-bootdisk.wks.in +++ b/scripts/lib/wic/canned-wks/efi-bootdisk.wks.in @@ -1,3 +1,3 @@ bootloader --ptable gpt -part /boot --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/boot --fstype=vfat --label boot --active --align 1024 --use-uuid --overhead-factor 1.0 +part /boot --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/boot --fstype=vfat --label boot --active --align 1024 --use-uuid --overhead-factor 1.1 part / --source rootfs --fstype=ext4 --label root --align 1024 --exclude-path boot/ diff --git a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks index 22b45217..80899761 100644 --- a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks +++ b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks @@ -4,5 +4,5 @@ include common.wks.inc -bootloader --timeout=0 --append="rw oprofile.timer=1 rootfstype=ext4 " +bootloader --timeout=0 --append="rw oprofile.timer=1 rootfstype=ext4 console=tty console=ttyS0 " diff --git a/scripts/lib/wic/filemap.py b/scripts/lib/wic/filemap.py index 4d9da281..85b39d5d 100644 --- a/scripts/lib/wic/filemap.py +++ b/scripts/lib/wic/filemap.py @@ -46,6 +46,13 @@ def get_block_size(file_obj): bsize = stat.st_blksize else: raise IOError("Unable to determine block size") + + # The logic in this script only supports a maximum of a 4KB + # block size + max_block_size = 4 * 1024 + if bsize > max_block_size: + bsize = max_block_size + return bsize class ErrorNotSupp(Exception): diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py index 73e3380c..163535e4 100644 --- a/scripts/lib/wic/help.py +++ b/scripts/lib/wic/help.py @@ -1118,7 +1118,7 @@ COMMAND: TOPIC: overview - Presents an overall overview of Wic plugins - Presents an overview and API for Wic plugins - kickstart - Presents a Wic kicstart file reference + kickstart - Presents a Wic kickstart file reference Examples: diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py index a49b7b97..7ef3dc83 100644 --- a/scripts/lib/wic/ksparser.py +++ b/scripts/lib/wic/ksparser.py @@ -159,7 +159,7 @@ class KickStart(): part.add_argument('--fstype', default='vfat', choices=('ext2', 'ext3', 'ext4', 'btrfs', 'squashfs', 'vfat', 'msdos', 'erofs', - 'swap')) + 'swap', 'none')) part.add_argument('--mkfs-extraopts', default='') part.add_argument('--label') part.add_argument('--use-label', action='store_true') @@ -171,6 +171,7 @@ class KickStart(): part.add_argument('--rootfs-dir') part.add_argument('--type', default='primary', choices = ('primary', 'logical')) + part.add_argument('--hidden', action='store_true') # --size and --fixed-size cannot be specified together; options # ----extra-space and --overhead-factor should also raise a parser @@ -187,11 +188,12 @@ class KickStart(): part.add_argument('--uuid') part.add_argument('--fsuuid') part.add_argument('--no-fstab-update', action='store_true') + part.add_argument('--mbr', action='store_true') bootloader = subparsers.add_parser('bootloader') bootloader.add_argument('--append') bootloader.add_argument('--configfile') - bootloader.add_argument('--ptable', choices=('msdos', 'gpt'), + bootloader.add_argument('--ptable', choices=('msdos', 'gpt', 'gpt-hybrid'), default='msdos') bootloader.add_argument('--timeout', type=int) bootloader.add_argument('--source') diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py index 2b90821b..1a7c140f 100644 --- a/scripts/lib/wic/misc.py +++ b/scripts/lib/wic/misc.py @@ -25,7 +25,7 @@ from wic import WicError logger = logging.getLogger('wic') # executable -> recipe pairs for exec_native_cmd -NATIVE_RECIPES = {"bmaptool": "bmap-tools", +NATIVE_RECIPES = {"bmaptool": "bmaptool", "dumpe2fs": "e2fsprogs", "grub-mkimage": "grub-efi", "isohybrid": "syslinux", diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py index e50871b8..795707ec 100644 --- a/scripts/lib/wic/partition.py +++ b/scripts/lib/wic/partition.py @@ -59,6 +59,8 @@ class Partition(): self.updated_fstab_path = None self.has_fstab = False self.update_fstab_in_rootfs = False + self.hidden = args.hidden + self.mbr = args.mbr self.lineno = lineno self.source_file = "" @@ -133,6 +135,8 @@ class Partition(): self.update_fstab_in_rootfs = True if not self.source: + if self.fstype == "none" or self.no_table: + return if not self.size and not self.fixed_size: raise WicError("The %s partition has a size of zero. Please " "specify a non-zero --size/--fixed-size for that " @@ -280,6 +284,20 @@ class Partition(): extraopts = self.mkfs_extraopts or "-F -i 8192" + if os.getenv('SOURCE_DATE_EPOCH'): + sde_time = int(os.getenv('SOURCE_DATE_EPOCH')) + if pseudo: + pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s " % (sde_time, pseudo) + else: + pseudo = "export E2FSPROGS_FAKE_TIME=%s; " % sde_time + + # Set hash_seed to generate deterministic directory indexes + namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-2f2fdf33d460") + if self.fsuuid: + namespace = uuid.UUID(self.fsuuid) + hash_seed = str(uuid.uuid5(namespace, str(sde_time))) + extraopts += " -E hash_seed=%s" % hash_seed + label_str = "" if self.label: label_str = "-L %s" % self.label @@ -300,6 +318,30 @@ class Partition(): mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) + if os.getenv('SOURCE_DATE_EPOCH'): + sde_time = hex(int(os.getenv('SOURCE_DATE_EPOCH'))) + debugfs_script_path = os.path.join(cr_workdir, "debugfs_script") + files = [] + for root, dirs, others in os.walk(rootfs_dir): + base = root.replace(rootfs_dir, "").rstrip(os.sep) + files += [ "/" if base == "" else base ] + files += [ base + "/" + n for n in dirs + others ] + with open(debugfs_script_path, "w") as f: + f.write("set_current_time %s\n" % (sde_time)) + if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update: + f.write("set_inode_field /etc/fstab mtime %s\n" % (sde_time)) + f.write("set_inode_field /etc/fstab mtime_extra 0\n") + for file in set(files): + for time in ["atime", "ctime", "crtime"]: + f.write("set_inode_field \"%s\" %s %s\n" % (file, time, sde_time)) + f.write("set_inode_field \"%s\" %s_extra 0\n" % (file, time)) + for time in ["wtime", "mkfs_time", "lastcheck"]: + f.write("set_super_value %s %s\n" % (time, sde_time)) + for time in ["mtime", "first_error_time", "last_error_time"]: + f.write("set_super_value %s 0\n" % (time)) + debugfs_cmd = "debugfs -w -f %s %s" % (debugfs_script_path, rootfs) + exec_native_cmd(debugfs_cmd, native_sysroot) + self.check_for_Y2038_problem(rootfs, native_sysroot) def prepare_rootfs_btrfs(self, rootfs, cr_workdir, oe_builddir, rootfs_dir, @@ -353,7 +395,7 @@ class Partition(): exec_native_cmd(mcopy_cmd, native_sysroot) if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update: - mcopy_cmd = "mcopy -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path) + mcopy_cmd = "mcopy -m -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path) exec_native_cmd(mcopy_cmd, native_sysroot) chmod_cmd = "chmod 644 %s" % rootfs @@ -381,6 +423,9 @@ class Partition(): (extraopts, self.fsuuid, rootfs, rootfs_dir) exec_native_cmd(erofs_cmd, native_sysroot, pseudo=pseudo) + def prepare_empty_partition_none(self, rootfs, oe_builddir, native_sysroot): + pass + def prepare_empty_partition_ext(self, rootfs, oe_builddir, native_sysroot): """ diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py index da483dae..a1d15265 100644 --- a/scripts/lib/wic/plugins/imager/direct.py +++ b/scripts/lib/wic/plugins/imager/direct.py @@ -117,7 +117,7 @@ class DirectPlugin(ImagerPlugin): updated = False for part in self.parts: if not part.realnum or not part.mountpoint \ - or part.mountpoint == "/" or not part.mountpoint.startswith('/'): + or part.mountpoint == "/" or not (part.mountpoint.startswith('/') or part.mountpoint == "swap"): continue if part.use_uuid: @@ -149,6 +149,9 @@ class DirectPlugin(ImagerPlugin): self.updated_fstab_path = os.path.join(self.workdir, "fstab") with open(self.updated_fstab_path, "w") as f: f.writelines(fstab_lines) + if os.getenv('SOURCE_DATE_EPOCH'): + fstab_time = int(os.getenv('SOURCE_DATE_EPOCH')) + os.utime(self.updated_fstab_path, (fstab_time, fstab_time)) def _full_path(self, path, name, extention): """ Construct full file path to a file we generate. """ @@ -310,7 +313,10 @@ class PartitionedImage(): # all partitions (in bytes) self.ptable_format = ptable_format # Partition table format # Disk system identifier - self.identifier = random.SystemRandom().randint(1, 0xffffffff) + if os.getenv('SOURCE_DATE_EPOCH'): + self.identifier = random.Random(int(os.getenv('SOURCE_DATE_EPOCH'))).randint(1, 0xffffffff) + else: + self.identifier = random.SystemRandom().randint(1, 0xffffffff) self.partitions = partitions self.partimages = [] @@ -336,7 +342,7 @@ class PartitionedImage(): # generate parition and filesystem UUIDs for part in self.partitions: if not part.uuid and part.use_uuid: - if self.ptable_format == 'gpt': + if self.ptable_format in ('gpt', 'gpt-hybrid'): part.uuid = str(uuid.uuid4()) else: # msdos partition table part.uuid = '%08x-%02d' % (self.identifier, part.realnum) @@ -392,6 +398,10 @@ class PartitionedImage(): raise WicError("setting custom partition type is not " \ "implemented for msdos partitions") + if part.mbr and self.ptable_format != 'gpt-hybrid': + raise WicError("Partition may only be included in MBR with " \ + "a gpt-hybrid partition table") + # Get the disk where the partition is located self.numpart += 1 if not part.no_table: @@ -400,7 +410,7 @@ class PartitionedImage(): if self.numpart == 1: if self.ptable_format == "msdos": overhead = MBR_OVERHEAD - elif self.ptable_format == "gpt": + elif self.ptable_format in ("gpt", "gpt-hybrid"): overhead = GPT_OVERHEAD # Skip one sector required for the partitioning scheme overhead @@ -484,7 +494,7 @@ class PartitionedImage(): # Once all the partitions have been layed out, we can calculate the # minumim disk size self.min_size = self.offset - if self.ptable_format == "gpt": + if self.ptable_format in ("gpt", "gpt-hybrid"): self.min_size += GPT_OVERHEAD self.min_size *= self.sector_size @@ -505,22 +515,49 @@ class PartitionedImage(): return exec_native_cmd(cmd, self.native_sysroot) + def _write_identifier(self, device, identifier): + logger.debug("Set disk identifier %x", identifier) + with open(device, 'r+b') as img: + img.seek(0x1B8) + img.write(identifier.to_bytes(4, 'little')) + + def _make_disk(self, device, ptable_format, min_size): + logger.debug("Creating sparse file %s", device) + with open(device, 'w') as sparse: + os.ftruncate(sparse.fileno(), min_size) + + logger.debug("Initializing partition table for %s", device) + exec_native_cmd("parted -s %s mklabel %s" % (device, ptable_format), + self.native_sysroot) + + def _write_disk_guid(self): + if self.ptable_format in ('gpt', 'gpt-hybrid'): + if os.getenv('SOURCE_DATE_EPOCH'): + self.disk_guid = uuid.UUID(int=int(os.getenv('SOURCE_DATE_EPOCH'))) + else: + self.disk_guid = uuid.uuid4() + + logger.debug("Set disk guid %s", self.disk_guid) + sfdisk_cmd = "sfdisk --disk-id %s %s" % (self.path, self.disk_guid) + exec_native_cmd(sfdisk_cmd, self.native_sysroot) + def create(self): - logger.debug("Creating sparse file %s", self.path) - with open(self.path, 'w') as sparse: - os.ftruncate(sparse.fileno(), self.min_size) + self._make_disk(self.path, + "gpt" if self.ptable_format == "gpt-hybrid" else self.ptable_format, + self.min_size) - logger.debug("Initializing partition table for %s", self.path) - exec_native_cmd("parted -s %s mklabel %s" % - (self.path, self.ptable_format), self.native_sysroot) + self._write_identifier(self.path, self.identifier) + self._write_disk_guid() - logger.debug("Set disk identifier %x", self.identifier) - with open(self.path, 'r+b') as img: - img.seek(0x1B8) - img.write(self.identifier.to_bytes(4, 'little')) + if self.ptable_format == "gpt-hybrid": + mbr_path = self.path + ".mbr" + self._make_disk(mbr_path, "msdos", self.min_size) + self._write_identifier(mbr_path, self.identifier) logger.debug("Creating partitions") + hybrid_mbr_part_num = 0 + for part in self.partitions: if part.num == 0: continue @@ -565,11 +602,19 @@ class PartitionedImage(): self._create_partition(self.path, part.type, parted_fs_type, part.start, part.size_sec) - if part.part_name: + if self.ptable_format == "gpt-hybrid" and part.mbr: + hybrid_mbr_part_num += 1 + if hybrid_mbr_part_num > 4: + raise WicError("Extended MBR partitions are not supported in hybrid MBR") + self._create_partition(mbr_path, "primary", + parted_fs_type, part.start, part.size_sec) + + if self.ptable_format in ("gpt", "gpt-hybrid") and (part.part_name or part.label): + partition_label = part.part_name if part.part_name else part.label logger.debug("partition %d: set name to %s", - part.num, part.part_name) + part.num, partition_label) exec_native_cmd("sgdisk --change-name=%d:%s %s" % \ - (part.num, part.part_name, + (part.num, partition_label, self.path), self.native_sysroot) if part.part_type: @@ -579,32 +624,55 @@ class PartitionedImage(): (part.num, part.part_type, self.path), self.native_sysroot) - if part.uuid and self.ptable_format == "gpt": + if part.uuid and self.ptable_format in ("gpt", "gpt-hybrid"): logger.debug("partition %d: set UUID to %s", part.num, part.uuid) exec_native_cmd("sgdisk --partition-guid=%d:%s %s" % \ (part.num, part.uuid, self.path), self.native_sysroot) - if part.label and self.ptable_format == "gpt": - logger.debug("partition %d: set name to %s", - part.num, part.label) - exec_native_cmd("parted -s %s name %d %s" % \ - (self.path, part.num, part.label), - self.native_sysroot) - if part.active: - flag_name = "legacy_boot" if self.ptable_format == 'gpt' else "boot" + flag_name = "legacy_boot" if self.ptable_format in ('gpt', 'gpt-hybrid') else "boot" logger.debug("Set '%s' flag for partition '%s' on disk '%s'", flag_name, part.num, self.path) exec_native_cmd("parted -s %s set %d %s on" % \ (self.path, part.num, flag_name), self.native_sysroot) + if self.ptable_format == 'gpt-hybrid' and part.mbr: + exec_native_cmd("parted -s %s set %d %s on" % \ + (mbr_path, hybrid_mbr_part_num, "boot"), + self.native_sysroot) if part.system_id: exec_native_cmd("sfdisk --part-type %s %s %s" % \ (self.path, part.num, part.system_id), self.native_sysroot) + if part.hidden and self.ptable_format == "gpt": + logger.debug("Set hidden attribute for partition '%s' on disk '%s'", + part.num, self.path) + exec_native_cmd("sfdisk --part-attrs %s %s RequiredPartition" % \ + (self.path, part.num), + self.native_sysroot) + + if self.ptable_format == "gpt-hybrid": + # Write a protective GPT partition + hybrid_mbr_part_num += 1 + if hybrid_mbr_part_num > 4: + raise WicError("Extended MBR partitions are not supported in hybrid MBR") + + # parted cannot directly create a protective GPT partition, so + # create with an arbitrary type, then change it to the correct type + # with sfdisk + self._create_partition(mbr_path, "primary", "fat32", 1, GPT_OVERHEAD) + exec_native_cmd("sfdisk --part-type %s %d 0xee" % (mbr_path, hybrid_mbr_part_num), + self.native_sysroot) + + # Copy hybrid MBR + with open(mbr_path, "rb") as mbr_file: + with open(self.path, "r+b") as image_file: + mbr = mbr_file.read(512) + image_file.write(mbr) + def cleanup(self): pass diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py index 634a808d..13a9cddf 100644 --- a/scripts/lib/wic/plugins/source/bootimg-efi.py +++ b/scripts/lib/wic/plugins/source/bootimg-efi.py @@ -220,6 +220,8 @@ class BootimgEFIPlugin(SourcePlugin): cls.do_configure_grubefi(hdddir, creator, cr_workdir, source_params) elif source_params['loader'] == 'systemd-boot': cls.do_configure_systemdboot(hdddir, creator, cr_workdir, source_params) + elif source_params['loader'] == 'uefi-kernel': + pass else: raise WicError("unrecognized bootimg-efi loader: %s" % source_params['loader']) except KeyError: @@ -330,40 +332,78 @@ class BootimgEFIPlugin(SourcePlugin): shutil.copyfileobj(in_file, initrd) initrd.close() + # Searched by systemd-boot: + # https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-images + install_cmd = "install -d %s/EFI/Linux" % hdddir + exec_cmd(install_cmd) + + staging_dir_host = get_bitbake_var("STAGING_DIR_HOST") + target_sys = get_bitbake_var("TARGET_SYS") + + objdump_cmd = "%s-objdump" % target_sys + objdump_cmd += " -p %s" % efi_stub + objdump_cmd += " | awk '{ if ($1 == \"SectionAlignment\"){print $2} }'" + + ret, align_str = exec_native_cmd(objdump_cmd, native_sysroot) + align = int(align_str, 16) + + objdump_cmd = "%s-objdump" % target_sys + objdump_cmd += " -h %s | tail -2" % efi_stub + ret, output = exec_native_cmd(objdump_cmd, native_sysroot) + + offset = int(output.split()[2], 16) + int(output.split()[3], 16) + + osrel_off = offset + align - offset % align + osrel_path = "%s/usr/lib/os-release" % staging_dir_host + osrel_sz = os.stat(osrel_path).st_size + + cmdline_off = osrel_off + osrel_sz + cmdline_off = cmdline_off + align - cmdline_off % align + cmdline_sz = os.stat(cmdline.name).st_size + + dtb_off = cmdline_off + cmdline_sz + dtb_off = dtb_off + align - dtb_off % align + dtb = source_params.get('dtb') if dtb: if ';' in dtb: raise WicError("Only one DTB supported, exiting") - dtb_params = '--add-section .dtb=%s/%s --change-section-vma .dtb=0x40000' % \ - (deploy_dir, dtb) + dtb_path = "%s/%s" % (deploy_dir, dtb) + dtb_params = '--add-section .dtb=%s --change-section-vma .dtb=0x%x' % \ + (dtb_path, dtb_off) + linux_off = dtb_off + os.stat(dtb_path).st_size + linux_off = linux_off + align - linux_off % align else: dtb_params = '' + linux_off = dtb_off - # Searched by systemd-boot: - # https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-images - install_cmd = "install -d %s/EFI/Linux" % hdddir - exec_cmd(install_cmd) + linux_path = "%s/%s" % (staging_kernel_dir, kernel) + linux_sz = os.stat(linux_path).st_size - staging_dir_host = get_bitbake_var("STAGING_DIR_HOST") - target_sys = get_bitbake_var("TARGET_SYS") + initrd_off = linux_off + linux_sz + initrd_off = initrd_off + align - initrd_off % align # https://www.freedesktop.org/software/systemd/man/systemd-stub.html objcopy_cmd = "%s-objcopy" % target_sys - objcopy_cmd += " --add-section .osrel=%s/usr/lib/os-release" % staging_dir_host - objcopy_cmd += " --change-section-vma .osrel=0x20000" + objcopy_cmd += " --enable-deterministic-archives" + objcopy_cmd += " --preserve-dates" + objcopy_cmd += " --add-section .osrel=%s" % osrel_path + objcopy_cmd += " --change-section-vma .osrel=0x%x" % osrel_off objcopy_cmd += " --add-section .cmdline=%s" % cmdline.name - objcopy_cmd += " --change-section-vma .cmdline=0x30000" + objcopy_cmd += " --change-section-vma .cmdline=0x%x" % cmdline_off objcopy_cmd += dtb_params - objcopy_cmd += " --add-section .linux=%s/%s" % (staging_kernel_dir, kernel) - objcopy_cmd += " --change-section-vma .linux=0x2000000" + objcopy_cmd += " --add-section .linux=%s" % linux_path + objcopy_cmd += " --change-section-vma .linux=0x%x" % linux_off objcopy_cmd += " --add-section .initrd=%s" % initrd.name - objcopy_cmd += " --change-section-vma .initrd=0x3000000" + objcopy_cmd += " --change-section-vma .initrd=0x%x" % initrd_off objcopy_cmd += " %s %s/EFI/Linux/linux.efi" % (efi_stub, hdddir) + exec_native_cmd(objcopy_cmd, native_sysroot) else: - install_cmd = "install -m 0644 %s/%s %s/%s" % \ - (staging_kernel_dir, kernel, hdddir, kernel) - exec_cmd(install_cmd) + if source_params.get('install-kernel-into-boot-dir') != 'false': + install_cmd = "install -m 0644 %s/%s %s/%s" % \ + (staging_kernel_dir, kernel, hdddir, kernel) + exec_cmd(install_cmd) if get_bitbake_var("IMAGE_EFI_BOOT_FILES"): for src_path, dst_path in cls.install_task: @@ -385,6 +425,28 @@ class BootimgEFIPlugin(SourcePlugin): for mod in [x for x in os.listdir(kernel_dir) if x.startswith("systemd-")]: cp_cmd = "cp %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, hdddir, mod[8:]) exec_cmd(cp_cmd, True) + elif source_params['loader'] == 'uefi-kernel': + kernel = get_bitbake_var("KERNEL_IMAGETYPE") + if not kernel: + raise WicError("Empty KERNEL_IMAGETYPE %s\n" % target) + target = get_bitbake_var("TARGET_SYS") + if not target: + raise WicError("Unknown arch (TARGET_SYS) %s\n" % target) + + if re.match("x86_64", target): + kernel_efi_image = "bootx64.efi" + elif re.match('i.86', target): + kernel_efi_image = "bootia32.efi" + elif re.match('aarch64', target): + kernel_efi_image = "bootaa64.efi" + elif re.match('arm', target): + kernel_efi_image = "bootarm.efi" + else: + raise WicError("UEFI stub kernel is incompatible with target %s" % target) + + for mod in [x for x in os.listdir(kernel_dir) if x.startswith(kernel)]: + cp_cmd = "cp %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, hdddir, kernel_efi_image) + exec_cmd(cp_cmd, True) else: raise WicError("unrecognized bootimg-efi loader: %s" % source_params['loader']) @@ -396,6 +458,11 @@ class BootimgEFIPlugin(SourcePlugin): cp_cmd = "cp %s %s/" % (startup, hdddir) exec_cmd(cp_cmd, True) + for paths in part.include_path or []: + for path in paths: + cp_cmd = "cp -r %s %s/" % (path, hdddir) + exec_cmd(cp_cmd, True) + du_cmd = "du -bks %s" % hdddir out = exec_cmd(du_cmd) blocks = int(out.split()[0]) @@ -410,6 +477,13 @@ class BootimgEFIPlugin(SourcePlugin): logger.debug("Added %d extra blocks to %s to get to %d total blocks", extra_blocks, part.mountpoint, blocks) + # required for compatibility with certain devices expecting file system + # block count to be equal to partition block count + if blocks < part.fixed_size: + blocks = part.fixed_size + logger.debug("Overriding %s to %d total blocks for compatibility", + part.mountpoint, blocks) + # dosfs image, created by mkdosfs bootimg = "%s/boot.img" % cr_workdir diff --git a/scripts/lib/wic/plugins/source/bootimg-partition.py b/scripts/lib/wic/plugins/source/bootimg-partition.py index 5dbe2558..94183174 100644 --- a/scripts/lib/wic/plugins/source/bootimg-partition.py +++ b/scripts/lib/wic/plugins/source/bootimg-partition.py @@ -1,4 +1,6 @@ # +# Imported from openembedded-core +# # SPDX-License-Identifier: GPL-2.0-only # # DESCRIPTION @@ -30,6 +32,7 @@ class BootimgPartitionPlugin(SourcePlugin): """ name = 'bootimg-partition' + image_boot_files_var_name = 'IMAGE_BOOT_FILES' @classmethod def do_configure_partition(cls, part, source_params, cr, cr_workdir, @@ -54,12 +57,12 @@ class BootimgPartitionPlugin(SourcePlugin): else: var = "" - boot_files = get_bitbake_var("IMAGE_BOOT_FILES" + var) + boot_files = get_bitbake_var(cls.image_boot_files_var_name + var) if boot_files is not None: break if boot_files is None: - raise WicError('No boot files defined, IMAGE_BOOT_FILES unset for entry #%d' % part.lineno) + raise WicError('No boot files defined, %s unset for entry #%d' % (cls.image_boot_files_var_name, part.lineno)) logger.debug('Boot files: %s', boot_files) @@ -110,7 +113,7 @@ class BootimgPartitionPlugin(SourcePlugin): # Use a custom configuration for extlinux.conf extlinux_conf = custom_cfg logger.debug("Using custom configuration file " - "%s for extlinux.cfg", configfile) + "%s for extlinux.conf", configfile) else: raise WicError("configfile is specified but failed to " "get it from %s." % configfile) diff --git a/scripts/lib/wic/plugins/source/bootimg-pcbios.py b/scripts/lib/wic/plugins/source/bootimg-pcbios.py index 32e47f18..a207a835 100644 --- a/scripts/lib/wic/plugins/source/bootimg-pcbios.py +++ b/scripts/lib/wic/plugins/source/bootimg-pcbios.py @@ -122,7 +122,7 @@ class BootimgPcbiosPlugin(SourcePlugin): syslinux_conf += "DEFAULT boot\n" syslinux_conf += "LABEL boot\n" - kernel = "/vmlinuz" + kernel = "/" + get_bitbake_var("KERNEL_IMAGETYPE") syslinux_conf += "KERNEL " + kernel + "\n" syslinux_conf += "APPEND label=boot root=%s %s\n" % \ @@ -155,8 +155,8 @@ class BootimgPcbiosPlugin(SourcePlugin): kernel = "%s-%s.bin" % \ (get_bitbake_var("KERNEL_IMAGETYPE"), get_bitbake_var("INITRAMFS_LINK_NAME")) - cmds = ("install -m 0644 %s/%s %s/vmlinuz" % - (staging_kernel_dir, kernel, hdddir), + cmds = ("install -m 0644 %s/%s %s/%s" % + (staging_kernel_dir, kernel, hdddir, get_bitbake_var("KERNEL_IMAGETYPE")), "install -m 444 %s/syslinux/ldlinux.sys %s/ldlinux.sys" % (bootimg_dir, hdddir), "install -m 0644 %s/syslinux/vesamenu.c32 %s/vesamenu.c32" % diff --git a/scripts/lib/wic/plugins/source/empty.py b/scripts/lib/wic/plugins/source/empty.py index 041617d6..c1903ca5 100644 --- a/scripts/lib/wic/plugins/source/empty.py +++ b/scripts/lib/wic/plugins/source/empty.py @@ -1,4 +1,6 @@ # +# Imported from openembedded-core +# # SPDX-License-Identifier: MIT # @@ -7,9 +9,19 @@ # To use it you must pass "empty" as argument for the "--source" parameter in # the wks file. For example: # part foo --source empty --ondisk sda --size="1024" --align 1024 +# +# The plugin supports writing zeros to the start of the +# partition. This is useful to overwrite old content like +# filesystem signatures which may be re-recognized otherwise. +# This feature can be enabled with +# '--sourceparams="[fill|size=[S|s|K|k|M|G]][,][bs=[S|s|K|k|M|G]]"' +# Conflicting or missing options throw errors. import logging +import os +from wic import WicError +from wic.ksparser import sizetype from wic.pluginbase import SourcePlugin logger = logging.getLogger('wic') @@ -17,6 +29,16 @@ logger = logging.getLogger('wic') class EmptyPartitionPlugin(SourcePlugin): """ Populate unformatted empty partition. + + The following sourceparams are supported: + - fill + Fill the entire partition with zeros. Requires '--fixed-size' option + to be set. + - size=[S|s|K|k|M|G] + Set the first N bytes of the partition to zero. Default unit is 'K'. + - bs=[S|s|K|k|M|G] + Write at most N bytes at a time during source file creation. + Defaults to '1M'. Default unit is 'K'. """ name = 'empty' @@ -29,4 +51,39 @@ class EmptyPartitionPlugin(SourcePlugin): Called to do the actual content population for a partition i.e. it 'prepares' the partition to be incorporated into the image. """ - return + get_byte_count = sizetype('K', True) + size = 0 + + if 'fill' in source_params and 'size' in source_params: + raise WicError("Conflicting source parameters 'fill' and 'size' specified, exiting.") + + # Set the size of the zeros to be written to the partition + if 'fill' in source_params: + if part.fixed_size == 0: + raise WicError("Source parameter 'fill' only works with the '--fixed-size' option, exiting.") + size = get_byte_count(part.fixed_size) + elif 'size' in source_params: + size = get_byte_count(source_params['size']) + + if size == 0: + # Nothing to do, create empty partition + return + + if 'bs' in source_params: + bs = get_byte_count(source_params['bs']) + else: + bs = get_byte_count('1M') + + # Create a binary file of the requested size filled with zeros + source_file = os.path.join(cr_workdir, 'empty-plugin-zeros%s.bin' % part.lineno) + if not os.path.exists(os.path.dirname(source_file)): + os.makedirs(os.path.dirname(source_file)) + + quotient, remainder = divmod(size, bs) + with open(source_file, 'wb') as file: + for _ in range(quotient): + file.write(bytearray(bs)) + file.write(bytearray(remainder)) + + part.size = (size + 1024 - 1) // 1024 # size in KB rounded up + part.source_file = source_file diff --git a/scripts/lib/wic/plugins/source/rawcopy.py b/scripts/lib/wic/plugins/source/rawcopy.py index 7c90cd3c..73017e86 100644 --- a/scripts/lib/wic/plugins/source/rawcopy.py +++ b/scripts/lib/wic/plugins/source/rawcopy.py @@ -1,4 +1,6 @@ # +# Imported from openembedded-core +# # SPDX-License-Identifier: GPL-2.0-only # @@ -23,6 +25,10 @@ class RawCopyPlugin(SourcePlugin): @staticmethod def do_image_label(fstype, dst, label): + # don't create label when fstype is none + if fstype == 'none': + return + if fstype.startswith('ext'): cmd = 'tune2fs -L %s %s' % (label, dst) elif fstype in ('msdos', 'vfat'): @@ -52,7 +58,8 @@ class RawCopyPlugin(SourcePlugin): decompressor = { ".bz2": "bzip2", ".gz": "gzip", - ".xz": "xz" + ".xz": "xz", + ".zst": "zstd -f", }.get(extension) if not decompressor: raise WicError("Not supported compressor filename extension: %s" % extension) diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py index fc06312e..c990143c 100644 --- a/scripts/lib/wic/plugins/source/rootfs.py +++ b/scripts/lib/wic/plugins/source/rootfs.py @@ -43,7 +43,7 @@ class RootfsPlugin(SourcePlugin): # directory, or modify a directory outside OpenEmbedded). full_path = os.path.realpath(os.path.join(rootfs_dir, path)) if not full_path.startswith(os.path.realpath(rootfs_dir)): - logger.error("%s: Must point inside the rootfs:" % (cmd, path)) + logger.error("%s: Must point inside the rootfs: %s" % (cmd, path)) sys.exit(1) return full_path @@ -224,7 +224,7 @@ class RootfsPlugin(SourcePlugin): if part.update_fstab_in_rootfs and part.has_fstab and not part.no_fstab_update: fstab_path = os.path.join(new_rootfs, "etc/fstab") # Assume that fstab should always be owned by root with fixed permissions - install_cmd = "install -m 0644 %s %s" % (part.updated_fstab_path, fstab_path) + install_cmd = "install -m 0644 -p %s %s" % (part.updated_fstab_path, fstab_path) if new_pseudo: pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo) else: