| Message ID | 20230110121748.14917-1-henning.schild@siemens.com |
|---|---|
| Headers | show |
| Series | CI rework of gitlab fast job | expand |
I did not dare touching any of this for a very long time. But we have reached a state where the CI is just a better room heater and not meaningful, at least when done with gitlab and running the fast. There is much to be improved, this is just a start. It is tested on our gitlab and we save about 1.5h per run while we now vm-test all images we build. Henning Am Tue, 10 Jan 2023 13:17:40 +0100 schrieb Henning Schild <henning.schild@siemens.com>: > This series contains some changes to the CI as done on gitlab but also > affects testing in general. The main idea is to run qemu basically > every time we build an image (only fast for gitlab at the moment), so > we do not just build it without testing. > On the way i also improved shell style and removed some tests from the > fast set, since they seem more advanced and just waste time on "fast" > > Henning Schild (8): > CI: move to avocado to 99.0 > CI: fix shell coding style > CI: install qemu-system when qemu testing is requested > testsuite: remove Ccache test from "fast" set > testsuite: remove Sdk test from "fast" set > testsuite: remove ContainerImage test from "fast" set > testsuite: remove ContainerSdk test from "fast" set > testsuite: remove Sstate test from "fast" set > > .gitlab-ci.yml | 2 +- > scripts/ci_build.sh | 26 ++++++++++++++++---------- > testsuite/citest.py | 10 +++++----- > 3 files changed, 22 insertions(+), 16 deletions(-) >
On Tue, Jan 10, 2023 at 01:17:40PM +0100, Henning Schild wrote: > This series contains some changes to the CI as done on gitlab but also > affects testing in general. The main idea is to run qemu basically every > time we build an image (only fast for gitlab at the moment), so we do > not just build it without testing. > On the way i also improved shell style and removed some tests from the > fast set, since they seem more advanced and just waste time on "fast" We are working on a new "dev" testsuite [1] which should be: * reasonably fast, * cover "main" arches (in separate testcases) and common (-ly broken) cases * and kept green most of the time, while keeping "fast" and "full" as maintainer suites (massive multiconfigs, etc.). So, your series is a good input what is not interesting for a typical downstream. Let's discuss when Anton is back next week. 1. https://patchwork.isar-build.org/project/isar/list/?series=670&state=* With kind regards, Baurzhan
Am Thu, 12 Jan 2023 13:23:17 +0100 schrieb Baurzhan Ismagulov <ibr@radix50.net>: > On Tue, Jan 10, 2023 at 01:17:40PM +0100, Henning Schild wrote: > > This series contains some changes to the CI as done on gitlab but > > also affects testing in general. The main idea is to run qemu > > basically every time we build an image (only fast for gitlab at the > > moment), so we do not just build it without testing. > > On the way i also improved shell style and removed some tests from > > the fast set, since they seem more advanced and just waste time on > > "fast" > > We are working on a new "dev" testsuite [1] which should be: > > * reasonably fast, > * cover "main" arches (in separate testcases) and common (-ly broken) > cases > * and kept green most of the time, > > while keeping "fast" and "full" as maintainer suites (massive > multiconfigs, etc.). > > So, your series is a good input what is not interesting for a typical > downstream. Let's discuss when Anton is back next week. Yes the two should be orthogonal at the moment. I am just making fast a bit faster and go deeper ... because it now boots. Since there have been no other review comments so far, i will prepare that v2 where all the "remove from fast" will have a generic reason and will be squashed. Henning > 1. > https://patchwork.isar-build.org/project/isar/list/?series=670&state=* > > With kind regards, > Baurzhan
On Thu, Jan 12, 2023 at 04:12:50PM +0100, Henning Schild wrote: > Yes the two should be orthogonal at the moment. I am just making fast a > bit faster and go deeper ... because it now boots. As "fast" should remain the maintainer testsuite, I'd like to discuss the coverage implications with Anton as well. > Since there have been no other review comments so far, i will prepare > that v2 where all the "remove from fast" will have a generic reason and > will be squashed. Yes, would be good. With kind regards, Baurzhan.
Am Thu, 12 Jan 2023 16:20:56 +0100 schrieb Baurzhan Ismagulov <ibr@radix50.net>: > On Thu, Jan 12, 2023 at 04:12:50PM +0100, Henning Schild wrote: > > Yes the two should be orthogonal at the moment. I am just making > > fast a bit faster and go deeper ... because it now boots. > > As "fast" should remain the maintainer testsuite, I'd like to discuss > the coverage implications with Anton as well. At the end of the day i want any variant of the testsuite to run on any setup just like it does on ilbers jenkins. That should be based on the kas containers for anyone not using your jenkins. With that any contributor would be able to run it manually or in their CI. We currently can not run "full" in gitlab CI because it takes too much time and our gitlab runners would take such jobs for at most 12h. But that is a value i could adjust if we find that it really needs to take that long. I hope we can get much faster and still have the same coverage. Henning > > Since there have been no other review comments so far, i will > > prepare that v2 where all the "remove from fast" will have a > > generic reason and will be squashed. > > Yes, would be good. > > > With kind regards, > Baurzhan.
On Fri, 2023-01-13 at 00:43 +0100, Henning Schild wrote: > Am Thu, 12 Jan 2023 16:20:56 +0100 > schrieb Baurzhan Ismagulov <ibr@radix50.net>: > > > On Thu, Jan 12, 2023 at 04:12:50PM +0100, Henning Schild wrote: > > > Yes the two should be orthogonal at the moment. I am just making > > > fast a bit faster and go deeper ... because it now boots. > > > > As "fast" should remain the maintainer testsuite, I'd like to > > discuss > > the coverage implications with Anton as well. > > At the end of the day i want any variant of the testsuite to run on > any > setup just like it does on ilbers jenkins. I fully agree. Currently it is really hard and time consuming to run the testsuite, even just for basic tests. > > That should be based on the kas containers for anyone not using your > jenkins. This would also help a lot, to run it locally. > > With that any contributor would be able to run it manually or in > their > CI. I guess this would also reduce the maintenance effort at Ilbers as less patches would be sent to the ML which do not pass the CI. Felix > > We currently can not run "full" in gitlab CI because it takes too > much > time and our gitlab runners would take such jobs for at most 12h. But > that is a value i could adjust if we find that it really needs to > take > that long. I hope we can get much faster and still have the same > coverage. > > Henning > > > > Since there have been no other review comments so far, i will > > > prepare that v2 where all the "remove from fast" will have a > > > generic reason and will be squashed. > > > > Yes, would be good. > > > > > > With kind regards, > > Baurzhan. >
On Fri, Jan 13, 2023 at 12:43:15AM +0100, Henning Schild wrote: > At the end of the day i want any variant of the testsuite to run on any > setup just like it does on ilbers jenkins. > > That should be based on the kas containers for anyone not using your > jenkins. > > With that any contributor would be able to run it manually or in their > CI. > > We currently can not run "full" in gitlab CI because it takes too much > time and our gitlab runners would take such jobs for at most 12h. But > that is a value i could adjust if we find that it really needs to take > that long. I hope we can get much faster and still have the same > coverage. Thanks for the summary, that's fully in line with our last off-list discussion: * The preferred downstream testsuite is "dev" (runs for ~ 2 h IIRC). * The testsuite runs manually on the host or in kas. * The testsuite runs automatically in GitLab, Jenkins or Azure using kas. * Separate testcases for every arch / distro combo. One can run the complete testsuite or individual testcases. * The maintainer CI provides test status and build control on a publicly accessible, easily scalable infrastructure. * The CI setup is easily clonable for the downstreams. * Downstreams can run Isar testsuites and / or their own testsuites. So, I propose that we update and apply the "dev" testsuite series, then apply yours with the following changes: 1. Use "dev" in "[PATCH 3/8] CI: install qemu-system when qemu testing is requested". 2. Patches 4-8 "testsuite: Remove X test from "fast" set" will be unnecessary (we'd take this into account during the "dev" series update -- those testcases are not part of the "dev" v1 anyway). For the final review of the use cases I'll wait for Anton. With kind regards, Baurzhan
Am Fri, 13 Jan 2023 14:14:58 +0100 schrieb Baurzhan Ismagulov <ibr@radix50.net>: > On Fri, Jan 13, 2023 at 12:43:15AM +0100, Henning Schild wrote: > > At the end of the day i want any variant of the testsuite to run on > > any setup just like it does on ilbers jenkins. > > > > That should be based on the kas containers for anyone not using your > > jenkins. > > > > With that any contributor would be able to run it manually or in > > their CI. > > > > We currently can not run "full" in gitlab CI because it takes too > > much time and our gitlab runners would take such jobs for at most > > 12h. But that is a value i could adjust if we find that it really > > needs to take that long. I hope we can get much faster and still > > have the same coverage. > > Thanks for the summary, that's fully in line with our last off-list > discussion: > > * The preferred downstream testsuite is "dev" (runs for ~ 2 h IIRC). > > * The testsuite runs manually on the host or in kas. > > * The testsuite runs automatically in GitLab, Jenkins or Azure using > kas. > > * Separate testcases for every arch / distro combo. One can run the > complete testsuite or individual testcases. > > * The maintainer CI provides test status and build control on a > publicly accessible, easily scalable infrastructure. > > * The CI setup is easily clonable for the downstreams. > > * Downstreams can run Isar testsuites and / or their own testsuites. > > > So, I propose that we update and apply the "dev" testsuite series, > then apply yours with the following changes: > > 1. Use "dev" in "[PATCH 3/8] CI: install qemu-system when qemu > testing is requested". > > 2. Patches 4-8 "testsuite: Remove X test from "fast" set" will be > unnecessary (we'd take this into account during the "dev" series > update -- those testcases are not part of the "dev" v1 anyway). > > For the final review of the use cases I'll wait for Anton. Let us see how we merge all that. I never looked at that dev series and think we should not add a third pipeline. Too many switches and variations just mean that we will need testing for the tests and nobody will know which to run when. I did not try but i bet if "fast" would simply use Sstate caches it would run in less than 30min for small changes. And we would naturally test Sstate and actually become much better on "partial rebuild". So in my book "dev" is "fast" or even "full" with warm caches. Henning > > With kind regards, > Baurzhan
On Fri, Jan 13, 2023 at 06:14:01PM +0100, Henning Schild wrote: > think we should not add a third pipeline. Too many switches and > variations just mean that we will need testing for the tests and nobody > will know which to run when. This is a very valid point. Our problem is, we need two (or maybe more :( ) testsuites for efficient handling of maintenance tasks, and they are very unsuitable for an average contributor. So this has appeared out of necessity. I fully agree regarding the switches, I also hate them for the same reason. We've considered moving to the raw avocado command line, but it is neither short nor obvious, so we left ci_build.sh wrapper at this stage. So, my proposal is to make "dev" the default and everyone should normally run ci_build.sh without any switches (unless anyone explicitly wants to experience some diversity in life). Of course, the maintainers should then regularly run it. > I did not try but i bet if "fast" would simply use Sstate caches it > would run in less than 30min for small changes. And we would naturally > test Sstate and actually become much better on "partial rebuild". > > So in my book "dev" is "fast" or even "full" with warm caches. Caching is also a good point. We can run "dev" with sstate by default; we just need a way to specify local site configuration out of git (e.g., in an environment variable). I wouldn't like to have "dev" = "full" + caching because we're evaluating adding rebuild testcases with and without specific options (like sstate, etc.) and running both in "full". "Fast" is rather a misnomer; its goal is to cover some 95+% of functionality in somewhat reasonable time. The mentioned testcases have been very consciously included there (e.g., sdk was broken several times); we can discuss what to include but in general we'd like to keep and increase the current coverage (execution time permitting) without affecting the users who don't need it. "Dev" would further diverge from "fast" due to e.g.: * Omitted "pedantic / infrastructure" bits as you mentioned * Chosen rebuild testcases as this has often been broken in the past * To be useful, "dev" should be kept green as far as possible, with all the implications. E.g., testcases can fail due to Debian issues, so "dev" shouldn't include Debian-ports (riscv). But the maintainers need to "quickly" assess the state of all supported arches. * Testcase granularity (one testcase covering one arch + distro) to have a clear indication in the test results which specific combos have failed and to be able to re-run that failed combo only. "Fast" needs to keep the multiconfig building to catch races. With kind regards, Baurzhan
Am Sat, 14 Jan 2023 15:40:47 +0100 schrieb Baurzhan Ismagulov <ibr@radix50.net>: > On Fri, Jan 13, 2023 at 06:14:01PM +0100, Henning Schild wrote: > > think we should not add a third pipeline. Too many switches and > > variations just mean that we will need testing for the tests and > > nobody will know which to run when. > > This is a very valid point. Our problem is, we need two (or maybe > more :( ) testsuites for efficient handling of maintenance tasks, and > they are very unsuitable for an average contributor. So this has > appeared out of necessity. > > I fully agree regarding the switches, I also hate them for the same > reason. We've considered moving to the raw avocado command line, but > it is neither short nor obvious, so we left ci_build.sh wrapper at > this stage. I would suggest a wrapper for every pipeline, rather than a switch to one of the wrappers. We can leave switches for advanced users but maybe only $@ to be passed to avocado. So a simple ls would tell one which tests are available, where the files hopefully have good names and the README or CONTRIBUTING files say which tests to best run after changing anything. Henning > > So, my proposal is to make "dev" the default and everyone should > normally run ci_build.sh without any switches (unless anyone > explicitly wants to experience some diversity in life). Of course, > the maintainers should then regularly run it. > > > > I did not try but i bet if "fast" would simply use Sstate caches it > > would run in less than 30min for small changes. And we would > > naturally test Sstate and actually become much better on "partial > > rebuild". > > > > So in my book "dev" is "fast" or even "full" with warm caches. > > Caching is also a good point. We can run "dev" with sstate by > default; we just need a way to specify local site configuration out > of git (e.g., in an environment variable). > > I wouldn't like to have "dev" = "full" + caching because we're > evaluating adding rebuild testcases with and without specific options > (like sstate, etc.) and running both in "full". > > "Fast" is rather a misnomer; its goal is to cover some 95+% of > functionality in somewhat reasonable time. The mentioned testcases > have been very consciously included there (e.g., sdk was broken > several times); we can discuss what to include but in general we'd > like to keep and increase the current coverage (execution time > permitting) without affecting the users who don't need it. > > "Dev" would further diverge from "fast" due to e.g.: > > * Omitted "pedantic / infrastructure" bits as you mentioned > > * Chosen rebuild testcases as this has often been broken in the past > > * To be useful, "dev" should be kept green as far as possible, with > all the implications. > > E.g., testcases can fail due to Debian issues, so "dev" shouldn't > include Debian-ports (riscv). But the maintainers need to "quickly" > assess the state of all supported arches. > > * Testcase granularity (one testcase covering one arch + distro) to > have a clear indication in the test results which specific combos > have failed and to be able to re-run that failed combo only. > > "Fast" needs to keep the multiconfig building to catch races. > > > With kind regards, > Baurzhan >