From patchwork Fri Jan 23 08:24:41 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhihang Wei X-Patchwork-Id: 336 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 23 Jan 2026 09:25:10 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-ed1-f61.google.com (mail-ed1-f61.google.com [209.85.208.61]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 60N8PAfk015033 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 23 Jan 2026 09:25:10 +0100 Received: by mail-ed1-f61.google.com with SMTP id 4fb4d7f45d1cf-658188b600asf2063399a12.3 for ; Fri, 23 Jan 2026 00:25:10 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1769156705; cv=pass; d=google.com; s=arc-20240605; b=M9GpuRW9+HmKxkw9fYy91e9MtF53rtKaPjLfer/hiNWebbV3O3T0ltBeRqjkj/Gxhg nHlkUfioIWlWe+izSxe2Xl483papn7VeDveWgPhm5RcaGFgUMuRvaOrXT+zQZ6I5Obzu HOiOkn3XV8JNfhsY249ttex5oqAaSHNq+ARdIPYwxg2w2XU7aXB0sr9rlyD9Xp1k35Og uUBBUqiPq/xQ6wm9dgrHxZxo3FeChXWx0DkSL+P9ijQubQ0PLiwbWXUkYLw7XMI1gZ1e C7dbyQbDjFRi8+efDKQ6ZPAG1hmx2NQ6h4NtzVZpCwcjfhtmlnssdclGR4N5MVExONwl NudQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=S5Dwq33GqBKe9xxsFlDOevCoo4cuEyntzeizPHTm85M=; fh=RSK1AdZ7cbRI8OLyabdlOmKXPQQ6HX/NWmx65XDIBc0=; b=RrCGGqBepXX7wFByot2sDKa3jQRqB8OdNvLw0chCW78gkYPHhbvrIchuNLS2gq4U8b FERo7+pTa+d9Q2OFZUBNAm0kLthJddba1J5GjsbTWMvbKdLyq2Ajcw883GdugwnfQlEc pvOdgZx3Cfm7raHxlMQCBjr1qVqkAAfr3BwWmsUNn8z0VonGDR4X0wn+4tHCuqhm5UQv DhUJMVqCsKMf68+to2qBvjrenne0Xu1fTu9b7JRauiwmAAICphc1XHI/8EVjovURdelq g7okGhTi1bOITDBHK2fcMfcEQ8nhpVOIqp0K8fO+BTp0mUkLcTbrDuzXy+EdmLJIxrAg d76A==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of wzh@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=wzh@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1769156705; x=1769761505; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:message-id:date:subject:cc:to:from :sender:from:to:cc:subject:date:message-id:reply-to; bh=S5Dwq33GqBKe9xxsFlDOevCoo4cuEyntzeizPHTm85M=; b=ogEZ6L0eqGYmxzNH72D7sdWjpoeigseDNpS4D32uHNjGbeB55qYZzREKrh0pblt+g3 GHRmgZEM4VDxl+IEkyzpUOJgUJnMQmdHezH4C0qB+G2bOnxeaR/zHwvUdXeakeRzcAV1 nQbtVpqsrPbOopZY5ZIjuCoqDVdUEMDG/yd9BPiTEXPRQWdQwWpTnAfpfUxuJ6uT2UiP EMQsILYGrXW9U8yR9n4vf4IoHJ84eiCqTWRTbWOviwMRBCiFqhGfc+Krdm6C5TjJtvf4 /u5cLwCfB019+8UXbuf8FVXLl5QZS2Ul1kn45b5steWWLIeauP5sYUVxiMWoSXywRskh utPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1769156705; x=1769761505; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:x-beenthere:x-gm-message-state :sender:from:to:cc:subject:date:message-id:reply-to; bh=S5Dwq33GqBKe9xxsFlDOevCoo4cuEyntzeizPHTm85M=; b=egxoo7BEuANP9AMaM8pLtel6lTVJOX3h83jKPN6h3mWfqoHrUvnXTa0DlylTMemIj2 iaIriV44XVYeESDx4Ij4ypO8BIjNJ+VpkCrS9RmynVQCA3pcmoI5ONUCBHYuR3VHFPRI IfwGCHVWvFnuqW78Ddnj6o5FspXVVpUJdpf5DRxSGDdJNvOFO1/+vv3cl5L6Dnlis5um 47aQ2RrhDE2tGCiz2AwQzRSAVKDc0s//urUmBuFnsu8HoesmiwokxrW01UHddOv+s0+I 51CGYFnKAz/N6el9Z5DOH3jvr+Aa5XpHlSQeMJWMKzCW98gqGnCKztV8VbD6wr/79EW4 hFdg== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCWhbT0TQklAM+xByyfPv9J5vcOakRviqCKMKKfG894r5iL3AAMgQ0L/dmTV2sMwYyRZ0nKgeDw=@isar-build.org X-Gm-Message-State: AOJu0Yxa1DYIUXQGNBkF2a2opqipnClcdQ9W+fXRfYFrQju1Cqu711rT SBl21NV5NrxB/wj+PwFEEx8d0d1zLj7u++SUgcBxEJZ7o47SmoOFnXHe X-Received: by 2002:a05:6402:1448:b0:64b:73d5:e2b4 with SMTP id 4fb4d7f45d1cf-6584874be52mr1563696a12.5.1769156704459; Fri, 23 Jan 2026 00:25:04 -0800 (PST) X-BeenThere: isar-users@googlegroups.com; h="AV1CL+F97IBhmENDALP2xePZS/HqHrG9ws969yDEDUb9KGdq0A==" Received: by 2002:aa7:d14b:0:b0:64b:9695:8dac with SMTP id 4fb4d7f45d1cf-65832d93590ls1321952a12.2.-pod-prod-05-eu; Fri, 23 Jan 2026 00:25:02 -0800 (PST) X-Received: by 2002:a17:907:7284:b0:b76:6020:ed2b with SMTP id a640c23a62f3a-b885ae0bb1amr130681166b.45.1769156702190; Fri, 23 Jan 2026 00:25:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1769156702; cv=none; d=google.com; s=arc-20240605; b=KyKup+XOtwoVOzhertnkaF98IFPoHTPKZ6yhWMphEtO2ijVoBOa40/CCrT6yYODne+ zyP6CYVW/27gf/Is2c29cCWhJdKAYSVMMMk0j1WJ5r9naqMJksVqQWEDUprc2306XKVh yA261ja+sVZqquBq/SiR2VyEMQn1An5ZswQLowLdicB2DkZgm5hyqRS9sRwPLPozrHwT SJnXcLKVzV0jAt1sV+5VRWtMWC2JAOp2JejANST1B3vjqgWVJLw8RcQVKUWtc6ddW33Z tBe/p/7pzpip8cRJnH52NT7z76DB7QwmVYDiCCXlQG0orHtDkn4Kg5sVOcOTGhNtvfoe i5xQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=aUnHsv8L+y4zUusV0+1gFMChveMmNXsgK5OERmbuoAA=; fh=V42d4Y/lNDOgZ/9/qsphFBeZCU4sz3YugfOOjBc+1H8=; b=Lre6L7VTEvztsqdUSnAodJNZzZL9WXd+3KjL12vHtD/wP1CquJaUln6FNZs19tbHtU iTn5TUIl821SdJUF2NIuMQmZWak5hXkjMYt2Q49sd2U1PKtUQEt/WyaBAi07qnlXzY9U JYYWRb3E4QNqf5cVnc76yF7SPG+eY8WWklODhdL+xtuSfTqBfNzC083XH2n9UkcAiS1x 0iV6yPob3oqlN3gTx4mZvb2rLiZswm5xcN5885HWnIGIo517+SXqFIb3ARRrPlcskYkQ QsgAPcW16hu6ezydBhSzVQ0NgjCQV9U7V+by3grpUDpV6FfqpofxZQyuUT6dgoo4PkX8 UFOw==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of wzh@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=wzh@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id a640c23a62f3a-b885b67f781si4041066b.2.2026.01.23.00.25.02 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 23 Jan 2026 00:25:02 -0800 (PST) Received-SPF: pass (google.com: domain of wzh@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from debian-zwei.m.ilbers.de ([88.130.203.42]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPA id 60N8P1Xf014977; Fri, 23 Jan 2026 09:25:01 +0100 From: Zhihang Wei To: isar-users@googlegroups.com, felix.moessbauer@siemens.com, amikan@ilbers.de Cc: wzh@ilbers.de, cedric.hombourger@siemens.com Subject: [PATCH v3 00/20] Various improvements to the testsuite Date: Fri, 23 Jan 2026 09:24:41 +0100 Message-Id: <20260123082501.240751-1-wzh@ilbers.de> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2, RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: wzh@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of wzh@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=wzh@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This series aims at simplifying and speeding up the testsuite execution. This should enable contributors to actually run it and test their patches. As pointed out in [1], the testsuite currently tests various features over and over again, while others are not tested at all. We try to improve this by focusing on "feature" tests, whereby each test case checks a set of features instead of a whole image. We further identified long running tasks (like the kernel builds) and move them to a dedicated test. The execution time reduction varies greatly depending on the hardware, network and sstate usage. When applying these changes on recently added patches, the time was often decreased by more than 50% on "fast". Note, that more refactorings of the testsuite are needed - and planned, but this series should already address the biggest pain points. [1] https://groups.google.com/g/isar-users/c/uZBTIHzLp8Q/m/8k0L1vfMDAAJ Changes since v2: - move startvm tests after each build to avoid inter-testsuite dependency - remove SstateTest from full Changes since v1: - continued work on testsuite refactoring, this supersedes: - testsuite: fix SignatureTest by avoiding absolute path in bblayers - testsuite: add rootfs target for rootfs only tests - Enable SState cache in GitLab CI - testsuite: fix typo in log message in perform_signature_lint - completed exclusion / refactoring of db2m sbuild chroot - GitLab CI: increased after script timeout to give sstate upload more time - GitLab CI: do not upload sstate artifacts of images (as almost no reuse) - do not add SignatureTest to fast tag (as otherwise VM tests break...) - rebuild container test images in VM test to not depend on test order - rebased onto next Anton Mikanovich (1): testsuite: Group prebuilt_containers related test cases together MOESSBAUER, Felix (17): testsuite: move targets with custom kernel to separate test testsuite: enable ccache on kernel tests testsuite: make prebuilt container a feature test testsuite: make compat test standalone test testsuite: handle IMAGE_INSTALL solely in cibuilder.py testsuite: limit cross_debsrc test to subset of packages testsuite: forward SSTATE_MIRRORS into CI env on sstate ci: add support for sstate cache testsuite: make test_cross_deps more specific testsuite: fix typo in log message in perform_signature_lint testsuite: fix SignatureTest by avoiding absolute path in bblayers testsuite: use more recent distros in SignatureTest testsuite: make SignatureTest idempotent testsuite: add rootfs target for rootfs only tests testsuite: refactor sbom tests to avoid overhead testsuite: make sbuild-flavor test standalone testsuite: skip VM tests if images are not available Zhihang Wei (2): testsuite: Execute startvm testcases after building testsuite: Remove SstateTest from full testsuite .gitlab-ci.yml | 36 +- meta-test/conf/bblayers.conf.sample | 8 +- meta-test/conf/local.conf.sample | 10 - .../hello-isar/hello-isar.bbappend | 3 + .../recipes-app/libhello/libhello.bbappend | 5 + .../recipes-core/images/isar-image-ci.bb | 2 - .../recipes-core/images/isar-rootfs-ci.bb | 17 + testsuite/cibase.py | 34 +- testsuite/cibuilder.py | 49 +- testsuite/citest.py | 740 +++++++++++------- 10 files changed, 602 insertions(+), 302 deletions(-) create mode 100644 meta-test/recipes-app/libhello/libhello.bbappend create mode 100644 meta-test/recipes-core/images/isar-rootfs-ci.bb