mbox series

[v2,00/17] Various improvements to the testsuite

Message ID 20251219131923.1504046-1-felix.moessbauer@siemens.com
Headers show
Series Various improvements to the testsuite | expand

Message

Felix Moessbauer Dec. 19, 2025, 1:19 p.m. UTC 
This series aims at simplifying and speeding up the testsuite execution.
This should enable contributors to actually run it and test their patches.

As pointed out in [1], the testsuite currently tests various features over
and over again, while others are not tested at all. We try to improve this
by focusing on "feature" tests, whereby each test case checks a set of
features instead of a whole image. We further identified long running tasks
(like the kernel builds) and move them to a dedicated test.

The execution time reduction varies greatly depending on the hardware,
network and sstate usage. When applying these changes on recently added
patches, the time was often decreased by more than 50% on "fast".

Note, that more refactorings of the testsuite are needed - and planned,
but this series should already address the biggest pain points.

[1] https://groups.google.com/g/isar-users/c/uZBTIHzLp8Q/m/8k0L1vfMDAAJ

Changes since v1:

- continued work on testsuite refactoring, this supersedes:
  - testsuite: fix SignatureTest by avoiding absolute path in bblayers
  - testsuite: add rootfs target for rootfs only tests
  - Enable SState cache in GitLab CI
  - testsuite: fix typo in log message in perform_signature_lint
- completed exclusion / refactoring of db2m sbuild chroot
- GitLab CI: increased after script timeout to give sstate upload more time
- GitLab CI: do not upload sstate artifacts of images (as almost no reuse)
- do not add SignatureTest to fast tag (as otherwise VM tests break...)
- rebuild container test images in VM test to not depend on test order
- rebased onto next

As always, the patches are written in a linear way, so only parts of them
can already be integrated.

That's a wrap for this year, isar developers!
See you next year!

Best regards,
Felix Moessbauer
Siemens AG

Felix Moessbauer (17):
  testsuite: move targets with custom kernel to separate test
  testsuite: enable ccache on kernel tests
  testsuite: make prebuilt container a feature test
  testsuite: make compat test standalone test
  testsuite: handle IMAGE_INSTALL solely in cibuilder.py
  testsuite: limit cross_debsrc test to subset of packages
  testsuite: forward SSTATE_MIRRORS into CI env on sstate
  ci: add support for sstate cache
  testsuite: make test_cross_deps more specific
  testsuite: fix typo in log message in perform_signature_lint
  testsuite: fix SignatureTest by avoiding absolute path in bblayers
  testsuite: use more recent distros in SignatureTest
  testsuite: make SignatureTest idempotent
  testsuite: add rootfs target for rootfs only tests
  testsuite: refactor sbom tests to avoid overhead
  testsuite: make sbuild-flavor test standalone
  testsuite: skip VM tests if images are not available

 .gitlab-ci.yml                                |  36 ++++-
 meta-test/conf/bblayers.conf.sample           |   8 +-
 meta-test/conf/local.conf.sample              |  10 --
 .../hello-isar/hello-isar.bbappend            |   3 +
 .../recipes-app/libhello/libhello.bbappend    |   5 +
 .../recipes-core/images/isar-image-ci.bb      |   2 -
 .../recipes-core/images/isar-rootfs-ci.bb     |  17 +++
 testsuite/cibase.py                           |  34 ++++-
 testsuite/cibuilder.py                        |  49 ++++++-
 testsuite/citest.py                           | 131 ++++++++++++++++--
 10 files changed, 255 insertions(+), 40 deletions(-)
 create mode 100644 meta-test/recipes-app/libhello/libhello.bbappend
 create mode 100644 meta-test/recipes-core/images/isar-rootfs-ci.bb