mbox series

[0/9] Add more signature cachability tests to the testsuite

Message ID 20240402172813.418770-1-chris.larson@siemens.com
Headers show
Series Add more signature cachability tests to the testsuite | expand

Message

kergoth@gmail.com April 2, 2024, 5:28 p.m. UTC 
From: Christopher Larson <chris.larson@siemens.com>

This series improves isar-sstate's lint command to support checking sigdata in tmp/stamps and to check for absolute paths in SRC_URI, and adds an additional test to the testsuite to exercise this capability. The new test checks for signature cachability issues much more quickly, as it does so without a full build, so it checks more target configurations. This uses bitbake's `-S none` option to avoid building anything, and then runs isar-sstate lint on the resulting sigdata in tmp/stamps.

This test will accept a verbose parameter on the command-line to pass --verbose to isar-sstate lint, which will show the "other" absolute paths found, not just source and build directory. This series also fixes two of the failures which were identified by the new test.

Here we can see the issues identified by the new test, which are fixed in this series:
```
$ avocado run testsuite/citest.py -t signatures --max-parallel-tasks=1
JOB ID     : 3a3308967946663d9b239f638b030502fa80ef0a
JOB LOG    : /builder/avocado/job-results/job-2024-03-29T19.24-3a33089/job.log
 (1/1) testsuite/citest.py:SignatureTest.test_signature_lint: STARTED
 (1/1) testsuite/citest.py:SignatureTest.test_signature_lint: FAIL: Detected cachability issues (42.93 s)
RESULTS    : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 44.63 s

Test summary:
1-testsuite/citest.py:SignatureTest.test_signature_lint: FAIL
```

Applicable section of the avocado full.log:
```
1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29 19:25:10,818 avocado.test cibase           L0142 ERROR| ==== issues found in ubuntu-focal-amd64:isar-bootstrap-target:unpack (95af4581) ====
1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29 19:25:10,818 avocado.test cibase           L0142 ERROR| -> path in sources-dir: SRC_URI entry "file:///home/kergoth/Code/industrial/signatures/isar/meta-isar/conf/distro/ubuntu.public.key;sha256sum=36a38199a4bf4eae1e7f574891f7dfcb79b91b87a33a499383265e1224b5e989"
1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29 19:25:10,818 avocado.test cibase           L0142 ERROR| ==== issues found in ubuntu-focal-amd64:isar-bootstrap-target:fetch (e8249cb2) ====
1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29 19:25:10,818 avocado.test cibase           L0142 ERROR| -> path in sources-dir: SRC_URI entry "file:///home/kergoth/Code/industrial/signatures/isar/meta-isar/conf/distro/ubuntu.public.key;sha256sum=36a38199a4bf4eae1e7f574891f7dfcb79b91b87a33a499383265e1224b5e989"
1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29 19:25:10,818 avocado.test cibase           L0142 ERROR| ==== issues found in debian-bullseye-amd64:isar-ci-ssh-setup:install (1c0a8b21) ====
1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29 19:25:10,818 avocado.test cibase           L0142 ERROR| -> path in sources-dir: TESTSUITEDIR = "/home/kergoth/Code/industrial/signatures/isar/testsuite"
1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29 19:25:10,818 avocado.test cibase           L0142 ERROR| found cachability issues (scanned 602 signatures)
1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29 19:25:10,818 avocado.test cibase           L0142 ERROR| -> absolute paths: sources-dir 3, build-dir 0, other 184
```

If we strip off the prefix for legibility:
```
ERROR| ==== issues found in ubuntu-focal-amd64:isar-bootstrap-target:unpack (95af4581) ====
ERROR| -> path in sources-dir: SRC_URI entry "file:///home/kergoth/Code/industrial/signatures/isar/meta-isar/conf/distro/ubuntu.public.key;sha256sum=36a38199a4bf4eae1e7f574891f7dfcb79b91b87a33a499383265e1224b5e989"
ERROR| ==== issues found in ubuntu-focal-amd64:isar-bootstrap-target:fetch (e8249cb2) ====
ERROR| -> path in sources-dir: SRC_URI entry "file:///home/kergoth/Code/industrial/signatures/isar/meta-isar/conf/distro/ubuntu.public.key;sha256sum=36a38199a4bf4eae1e7f574891f7dfcb79b91b87a33a499383265e1224b5e989"
ERROR| ==== issues found in debian-bullseye-amd64:isar-ci-ssh-setup:install (1c0a8b21) ====
ERROR| -> path in sources-dir: TESTSUITEDIR = "/home/kergoth/Code/industrial/signatures/isar/testsuite"
ERROR| found cachability issues (scanned 602 signatures)
ERROR| -> absolute paths: sources-dir 3, build-dir 0, other 184
```

The sstate/signatures tests were run for each commit in this series using `git rebase -x` to ensure it does not break a bisect.

I welcome any and all feedback on this. In particular I don't love extending the hardcoded list of image tasks being ignored in isar-sstate lint, but it's a start. I'm open to suggestions.

Christopher Larson (9):
  isar-bootstrap: avoid forced early expansion of key vars
  isar-ci-ssh-setup: avoid abs path in signatures
  isar-sstate: lint: check for absolute paths in SRC_URI
  isar-sstate: lint: add support for checking stamps
  isar-sstate: lint: ignore more image tasks
  isar-sstate: add --excluded-tasks argument
  cibuilder.py: add -S support to the bitbake method
  testsuite: add perform_signature_lint method
  testsuite: add signature cachability checks

 .../isar-ci-ssh-setup_0.1.bb                  |  3 +
 .../isar-bootstrap/isar-bootstrap.inc         |  8 ++-
 scripts/isar-sstate                           | 60 +++++++++++++++----
 testsuite/cibase.py                           | 19 ++++++
 testsuite/cibuilder.py                        |  5 +-
 testsuite/citest.py                           | 22 ++++++-
 6 files changed, 100 insertions(+), 17 deletions(-)