mbox

[0/2] Sstate maintenance script

Message ID 20220413063534.799526-1-adriaan.schmidt@siemens.com
State Superseded, archived
Headers show

Message

Schmidt, Adriaan April 12, 2022, 10:35 p.m. UTC 
Hi,

We have been running CI with shared sstate caches for some months now, in
several downstream projects. This is the cache maintenance script that has
evolved during that time. Detailed documentation is in the script itself.
Main features:
- upload cache artifacts to shared caches on filesystem, http, or s3
- clean old artifacts from shared caches
- analyze in detail why cache misses happen (what has changed in the signatures)

The last one is especially interesting, and has already yielded some
improvements to the cacheability of Isar ([PATCH v2 0/4] Improve cacheability);
analysis is still ongoing.
This feature becomes more robust with a patch to bitbake (p2 of this series,
also submitted upstream).

One issue: testing!
This is not easy, because it involves infrastructure, and artificial tests
that provide decent coverage would be quite complex to design.

If we declare that we sufficiently trust the sstate code, we could add a
shared/persistent cache to the Isar CI infrastructure. This would further test
the sstate feature and all steps involved in maintaining such a setup.
In addition, it would significantly speed up CI builds.

Adriaan

Adriaan Schmidt (2):
  scripts: add isar-sstate
  bitbake-diffsigs: make finding of changed signatures more robust

 bitbake/lib/bb/siggen.py |  10 +-
 scripts/isar-sstate      | 743 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 748 insertions(+), 5 deletions(-)
 create mode 100755 scripts/isar-sstate