| Message ID | 20220706114655.119026-3-Quirin.Gylstorff@siemens.com |
|---|---|
| State | Accepted, archived |
| Headers | show |
| Series | Fix .gitlab-ci build | expand |
On 7/6/22 13:46, Quirin Gylstorff wrote: > From: Quirin Gylstorff <quirin.gylstorff@siemens.com> > > With avocado-framework version 97.0 the following error occurs: > ``` > Successfully installed avocado-framework-97.0 > + avocado --show=app,test run /builds/debian/isar/testsuite/citest.py -t fast,-repro,-startvm --test-runner=runner --disable-sysinfo -p quiet=1 -p cross=1 -p time_to_wait=300 > Suite creation for runner "runner" is not supported > ``` > Reason: the legacy test runner was removed with https://github.com/avocado-framework/avocado/commit/c961522567539bdb3859524f1087e4c5c69c600a. Quirin > Force for now version 96.0. > > Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com> > --- > scripts/ci_build.sh | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh > index 2513f1a0..b1ec6912 100755 > --- a/scripts/ci_build.sh > +++ b/scripts/ci_build.sh > @@ -21,7 +21,7 @@ if ! command -v avocado > /dev/null; then > rm -rf /tmp/avocado_venv > virtualenv --python python3 /tmp/avocado_venv > source /tmp/avocado_venv/bin/activate > - pip install avocado-framework > + pip install avocado-framework==96.0 > fi > > # Get Avocado build tests path -
Am Wed, 6 Jul 2022 13:46:55 +0200 schrieb Quirin Gylstorff <Quirin.Gylstorff@siemens.com>: > From: Quirin Gylstorff <quirin.gylstorff@siemens.com> > > With avocado-framework version 97.0 the following error occurs: > ``` > Successfully installed avocado-framework-97.0 > + avocado --show=app,test run /builds/debian/isar/testsuite/citest.py > -t fast,-repro,-startvm --test-runner=runner --disable-sysinfo -p > quiet=1 -p cross=1 -p time_to_wait=300 Suite creation for runner > "runner" is not supported ``` > > Force for now version 96.0. > > Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com> > --- > scripts/ci_build.sh | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh > index 2513f1a0..b1ec6912 100755 > --- a/scripts/ci_build.sh > +++ b/scripts/ci_build.sh > @@ -21,7 +21,7 @@ if ! command -v avocado > /dev/null; then > rm -rf /tmp/avocado_venv > virtualenv --python python3 /tmp/avocado_venv > source /tmp/avocado_venv/bin/activate > - pip install avocado-framework > + pip install avocado-framework==96.0 Once the code is reworked so it does not use the legacy feature any longer this pinning should probably not be dropped, it can be moved every once in a while, but getting a random version is simply not good. As we see again here. Especially not when ilbers does not run this bit in their CI, having avocado already installed in some version. Maybe we should always use that venv/pip version no matter if the host is contaminated already with some other random version. How is the mainline issue for debian going anyhow? Time to grab some tequila to wash down the guacamole and sing some "i told you so" songs. pip pip hurray! When it comes to testing the maintainer should really make it as simple as possible for people to run the tests, ideally with the very same outcome as the pipelines of the maintainer. Also it should be easy for users to contribute to the test suite, so it will be reasonable to ask people to bring tests with their features. IMHO the whole avocado story made both aspects worse than before. Only for some features that the maintainer wanted, but what they get is ontested patches because they are the only ones able to run that beast. Henning > fi > > # Get Avocado build tests path
On 7/6/22 14:54, Gylstorff Quirin wrote: > > > On 7/6/22 13:46, Quirin Gylstorff wrote: >> From: Quirin Gylstorff <quirin.gylstorff@siemens.com> >> >> With avocado-framework version 97.0 the following error occurs: >> ``` >> Successfully installed avocado-framework-97.0 >> + avocado --show=app,test run /builds/debian/isar/testsuite/citest.py >> -t fast,-repro,-startvm --test-runner=runner --disable-sysinfo -p >> quiet=1 -p cross=1 -p time_to_wait=300 >> Suite creation for runner "runner" is not supported >> ``` >> > Reason: the legacy test runner was removed with > https://github.com/avocado-framework/avocado/commit/c961522567539bdb3859524f1087e4c5c69c600a. > > Quirin Also the new runner breaks the CI build. Attached is the build log. This is on next with the following changes: ``` modified scripts/ci_build.sh @@ -21,7 +21,7 @@ if ! command -v avocado > /dev/null; then rm -rf /tmp/avocado_venv virtualenv --python python3 /tmp/avocado_venv source /tmp/avocado_venv/bin/activate - pip install avocado-framework + pip install avocado-framework==97.0 fi # Get Avocado build tests path @@ -143,5 +143,5 @@ export VIRTUAL_ENV="./" set -x avocado $VERBOSE run "$TESTSUITE_DIR/citest.py" \ - -t $TAGS --test-runner=runner --disable-sysinfo \ + -t $TAGS --disable-sysinfo \ -p quiet=$QUIET -p cross=$CROSS_BUILD -p time_to_wait=$TIMEOUT ``` Quirin >> Force for now version 96.0. >> >> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com> >> --- >> scripts/ci_build.sh | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh >> index 2513f1a0..b1ec6912 100755 >> --- a/scripts/ci_build.sh >> +++ b/scripts/ci_build.sh >> @@ -21,7 +21,7 @@ if ! command -v avocado > /dev/null; then >> rm -rf /tmp/avocado_venv >> virtualenv --python python3 /tmp/avocado_venv >> source /tmp/avocado_venv/bin/activate >> - pip install avocado-framework >> + pip install avocado-framework==96.0 >> fi >> # Get Avocado build tests path > > - >
On Wed, Jul 06, 2022 at 04:01:48PM +0200, Henning Schild wrote: > > diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh > > index 2513f1a0..b1ec6912 100755 > > --- a/scripts/ci_build.sh > > +++ b/scripts/ci_build.sh > > @@ -21,7 +21,7 @@ if ! command -v avocado > /dev/null; then > > rm -rf /tmp/avocado_venv > > virtualenv --python python3 /tmp/avocado_venv > > source /tmp/avocado_venv/bin/activate > > - pip install avocado-framework > > + pip install avocado-framework==96.0 > > Once the code is reworked so it does not use the legacy feature any > longer this pinning should probably not be dropped, it can be moved > every once in a while, but getting a random version is simply not good. > As we see again here. Not including avocado in kas-container is kas's decision. This workaround was requested on this list and we applied it. If you ask me, I don't want to have it and still suggest to use debs. > Especially not when ilbers does not run this bit in their CI, having > avocado already installed in some version. Maybe we should always use > that venv/pip version no matter if the host is contaminated already > with some other random version. This is one of the reasons to avoid pip, which we have repeatedly suggested. > How is the mainline issue for debian going anyhow? We've contacted Debian, got initial feedback and fixed most of that. The remaining issues will be clarified with the upstream. We'll send the updated version and ask for the review. > Time to grab some tequila to wash down the guacamole and sing some "i > told you so" songs. pip pip hurray! I call everyone on this list to report the issues and discuss specific solutions. I can also start "told you to use debs" songs which won't solve anyone's problems. > When it comes to testing the maintainer should really make it as simple > as possible for people to run the tests, ideally with the very same > outcome as the pipelines of the maintainer. I think providing a deb in an apt repo is as simple as possible for people to run the tests. Way simpler and cleaner than the pip stuff. And using the same deb would result in the same outcome. > Also it should be easy for > users to contribute to the test suite, so it will be reasonable to ask > people to bring tests with their features. > IMHO the whole avocado story made both aspects worse than before. If you have specific difficulties contributing to the testsuite, feel free to share them here. From the frameworks that you had suggested, any of those would require getting familiar with writing testcases, so currently I don't see major differences in this specific aspect, but there are important advantages in other aspects. I admit that since we spend huge amounts of machine-hours for running CI on every patch set, we do have special requirements for the test framework. And I think the result is good for downstreams as well. > Only > for some features that the maintainer wanted, but what they get is > ontested patches because they are the only ones able to run that beast. This is not true, you are running avocado in your CI. You can also run specific testcases in kas-container, just like with any other framework. If desired, we could contribute docs or make a community workshop on using the testsuite. With kind regards, Baurzhan
Am Wed, 6 Jul 2022 18:27:41 +0200 schrieb Baurzhan Ismagulov <ibr@radix50.net>: > On Wed, Jul 06, 2022 at 04:01:48PM +0200, Henning Schild wrote: > > > diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh > > > index 2513f1a0..b1ec6912 100755 > > > --- a/scripts/ci_build.sh > > > +++ b/scripts/ci_build.sh > > > @@ -21,7 +21,7 @@ if ! command -v avocado > /dev/null; then > > > rm -rf /tmp/avocado_venv > > > virtualenv --python python3 /tmp/avocado_venv > > > source /tmp/avocado_venv/bin/activate > > > - pip install avocado-framework > > > + pip install avocado-framework==96.0 > > > > Once the code is reworked so it does not use the legacy feature any > > longer this pinning should probably not be dropped, it can be moved > > every once in a while, but getting a random version is simply not > > good. As we see again here. > > Not including avocado in kas-container is kas's decision. This > workaround was requested on this list and we applied it. If you ask > me, I don't want to have it and still suggest to use debs. pip is kind of ok when pinning the version, removing the pin was the problem > > > Especially not when ilbers does not run this bit in their CI, having > > avocado already installed in some version. Maybe we should always > > use that venv/pip version no matter if the host is contaminated > > already with some other random version. > > This is one of the reasons to avoid pip, which we have repeatedly > suggested. That deb of yours does not count as a deb. It is as unmaintained as the debianization in avocado itself, probably not tested in bookworm ... and what have you. The pip way is at least maintained and recent. If one never updates avocado one will happily write tests using deprecated features ... forcing oneself to stick to the old version forever. Not to mention that some people might want to run stuff on non-debian distros. i.e. the one i tried to get in with the arm64 efi booting, the build itself done in a container (debian) the boot tests initially done/developed on the host (gentoo). > > > How is the mainline issue for debian going anyhow? > > We've contacted Debian, got initial feedback and fixed most of that. > The remaining issues will be clarified with the upstream. We'll send > the updated version and ask for the review. > > > > Time to grab some tequila to wash down the guacamole and sing some > > "i told you so" songs. pip pip hurray! > > I call everyone on this list to report the issues and discuss specific > solutions. I can also start "told you to use debs" songs which won't > solve anyone's problems. Both lack debian upstream quality. Again it is fine as long as it has the == ... maybe together with a --no-deps if possible. > > > When it comes to testing the maintainer should really make it as > > simple as possible for people to run the tests, ideally with the > > very same outcome as the pipelines of the maintainer. > > I think providing a deb in an apt repo is as simple as possible for > people to run the tests. Way simpler and cleaner than the pip stuff. > And using the same deb would result in the same outcome. I do not think so. What one could do is fetch upstream avocado, build their package and install it. Do that every single time, also in the jenkins setups. > > Also it should be easy for > > users to contribute to the test suite, so it will be reasonable to > > ask people to bring tests with their features. > > IMHO the whole avocado story made both aspects worse than before. > > If you have specific difficulties contributing to the testsuite, feel > free to share them here. From the frameworks that you had suggested, > any of those would require getting familiar with writing testcases, > so currently I don't see major differences in this specific aspect, > but there are important advantages in other aspects. I admit that > since we spend huge amounts of machine-hours for running CI on every > patch set, we do have special requirements for the test framework. > And I think the result is good for downstreams as well. From what i saw the tests are pretty convoluted and python is not properly used. But that has nothing to do with avocado, just the way the tests have been written. > > Only > > for some features that the maintainer wanted, but what they get is > > ontested patches because they are the only ones able to run that > > beast. > > This is not true, you are running avocado in your CI. You can also > run specific testcases in kas-container, just like with any other > framework. If desired, we could contribute docs or make a community > workshop on using the testsuite. We are trying to ... sometimes ... if it works. But for sure not always. I am still often finding myself in the place where i send untested stuff ... well local manual tested and then send. And one reason people skip tests is still that it is pretty hard and time consuming to run them. And they are very random and likely to fail anyhow. So i just send ... maybe i push to my poller branch of your jenkins ... but this will always go red anyhow ... like next does all the time. The testsuite failing in more than 20% in the cases anyways ... means people stop caring about it. It is "rainy thunderstorm" in jenkins anyhow or "red" in gitlab ... it was before your patch and it remains with your patch. At least the reports ease that a bit, if people still care to look what is behind "red" or "thunderstorm" Henning > > With kind regards, > Baurzhan
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh index 2513f1a0..b1ec6912 100755 --- a/scripts/ci_build.sh +++ b/scripts/ci_build.sh @@ -21,7 +21,7 @@ if ! command -v avocado > /dev/null; then rm -rf /tmp/avocado_venv virtualenv --python python3 /tmp/avocado_venv source /tmp/avocado_venv/bin/activate - pip install avocado-framework + pip install avocado-framework==96.0 fi # Get Avocado build tests path