| Message ID | 20220509101604.3249558-1-adriaan.schmidt@siemens.com |
|---|---|
| Headers | show |
| Series | Sstate maintenance script | expand |
09.05.2022 13:15, Adriaan Schmidt wrote: > We have been running CI with shared sstate caches for some months now, in > several downstream projects. This is the cache maintenance script that has > evolved during that time. Detailed documentation is in the script itself. > Main features: > - upload cache artifacts to shared caches on filesystem, http, or s3 > - clean old artifacts from shared caches > - analyze in detail why cache misses happen (what has changed in the signatures) > - check the sstate signatures for absolute paths pointing to the build host > > The last two are especially interesting, and have already yielded some > improvements to the cacheability of Isar, some already merged, and some > more in "[PATCH 0/7] Further improve cachability of ISAR". > > p1 handles another absolute path in a variable (LAYERDIR_isar). > p2..3 are minor patches to bitbake (both already upstream) that greatly > improve accuracy and performance of the sstate analysis. > p4 refactors handling of the apt_* tasks. This was motivated by the sstate > analysis, but I think it also makes the code cleaner. > p5..6 add the sstate maintenance script (2 authors, hence 2 patches). > p7 includes a signature check into the sstate test case. This requires the > changes from "[PATCH 0/7] Further improve cachability of ISAR" for the > test to pass. > > One issue: testing! > This is not easy, because it involves infrastructure, and artificial tests > that provide decent coverage would be quite complex to design. > > If we declare that we sufficiently trust the sstate code, we could add a > shared/persistent cache to the Isar CI infrastructure. This would further test > the sstate feature and all steps involved in maintaining such a setup. > In addition, it would significantly speed up CI builds. > > changes since v1: > - generally improved script > - analysis and cachability improvements in bitbake, dpkg-base, and meta-isar > - added "sstate linting" to the testsuite > > Adriaan Schmidt (6): > meta-isar: improve cachability > bitbake-diffsigs: make finding of changed signatures more robust > bitbake-diffsigs: break on first dependent task difference > dpkg-base: refactor dependencies of apt_* tasks > scripts: add isar-sstate > testsuite: add cachability analysis to sstate test > > Felix Moessbauer (1): > isar-sstate: add tool to check for caching issues > > bitbake/lib/bb/siggen.py | 11 +- > meta-isar/conf/layer.conf | 1 + > meta/classes/dpkg-base.bbclass | 14 +- > meta/classes/dpkg.bbclass | 2 +- > scripts/isar-sstate | 863 +++++++++++++++++++++++++++++++++ > testsuite/cibase.py | 8 +- > 6 files changed, 884 insertions(+), 15 deletions(-) > create mode 100755 scripts/isar-sstate > Applied to next, thanks.