diff mbox series

[v4,4/5] ci: Add test cases for container fetching and loading

Message ID 3a0b76b54aca9cee33d249a2df73d233edf08708.1721407122.git.jan.kiszka@siemens.com
State Superseded, archived
Headers show
Series Introduce container fetcher and pre-loader | expand

Commit Message

Jan Kiszka July 19, 2024, 4:38 p.m. UTC 
From: Jan Kiszka <jan.kiszka@siemens.com>

This plugs the two example recipes for loading container images into
VM-based testing. The test consists of running 'true' in the installed
alpine images.

Rather than enabling the ci user to do password-less sudo, this uses su
with the piped-in password. Another trick needed is to poll for the
images because loading is performed asynchronously.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 .../recipes-core/images/isar-image-ci.bb      |  2 ++
 testsuite/citest.py                           | 21 +++++++++++++++++++
 2 files changed, 23 insertions(+)

Comments

Uladzimir Bely July 30, 2024, 7:44 a.m. UTC | #1 
On Fri, 2024-07-19 at 18:38 +0200, 'Jan Kiszka' via isar-users wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This plugs the two example recipes for loading container images into
> VM-based testing. The test consists of running 'true' in the
> installed
> alpine images.
> 
> Rather than enabling the ci user to do password-less sudo, this uses
> su
> with the piped-in password. Another trick needed is to poll for the
> images because loading is performed asynchronously.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  .../recipes-core/images/isar-image-ci.bb      |  2 ++
>  testsuite/citest.py                           | 21
> +++++++++++++++++++
>  2 files changed, 23 insertions(+)
> 
> diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-
> test/recipes-core/images/isar-image-ci.bb
> index e5d51e6e..9133da74 100644
> --- a/meta-test/recipes-core/images/isar-image-ci.bb
> +++ b/meta-test/recipes-core/images/isar-image-ci.bb
> @@ -16,6 +16,7 @@ IMAGE_INSTALL += "sshd-regen-keys"
>  
>  # qemuamd64-bookworm
>  WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
> +IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-
> img prebuilt-podman-img"
>  
>  # qemuamd64-bullseye
>  IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz tar.gz"
> @@ -51,3 +52,4 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= "
> ${SYSTEMD_BOOTLOADER_INSTALL}
>  # qemuarm64-bookworm
>  IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
>  IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= "
> ${GRUB_BOOTLOADER_INSTALL}"
> +IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-
> img prebuilt-podman-img"
> diff --git a/testsuite/citest.py b/testsuite/citest.py
> index 8dd907d0..539c9440 100755
> --- a/testsuite/citest.py
> +++ b/testsuite/citest.py
> @@ -522,3 +522,24 @@ class VmBootTestFull(CIBaseTest):
>          self.init()
>          self.vm_start('mipsel','bookworm', image='isar-image-ci',
>                        script='test_kernel_module.sh example_module')
> +
> +
> +    def test_amd64_bookworm_prebuilt_containers(self):
> +        self.init()
> +        self.vm_start('amd64', 'bookworm', image='isar-image-ci',
> +                      cmd='echo root | su -c \'' \
> +                          'PATH=\$PATH:/usr/sbin;' \
> +                          'for n in \$(seq 30); do docker images |
> grep -q alpine && break; sleep 10; done;' \
> +                          'docker run --rm
> quay.io/libpod/alpine:3.10.2 true && ' \
> +                          'for n in \$(seq 30); do podman images |
> grep -q alpine && break; sleep 10; done;' \
> +                          'podman run --rm
> quay.io/libpod/alpine:latest true\'')
> +
> +    def test_arm64_bookworm_prebuilt_containers(self):
> +        self.init()
> +        self.vm_start('arm64', 'bookworm', image='isar-image-ci',
> +                      cmd='echo root | su -c \'' \
> +                          'PATH=\$PATH:/usr/sbin;' \
> +                          'for n in \$(seq 30); do docker images |
> grep -q alpine && break; sleep 10; done;' \
> +                          'docker run --rm
> quay.io/libpod/alpine:3.10.2 true && ' \
> +                          'for n in \$(seq 30); do podman images |
> grep -q alpine && break; sleep 10; done;' \
> +                          'podman run --rm
> quay.io/libpod/alpine:latest true\'')

This appears to fail in CI:

[stdlog] 2024-07-30 07:29:37,721 avocado.app ERROR| qemu-system-
aarch64: -drive
file=/workspace/build/isar_ub_devel/49/build/tmp/deploy/images/qemuarm6
4/isar-image-ci-debian-bookworm-
qemuarm64.ext4,if=none,format=raw,id=hd0: Could not open
'/workspace/build/isar_ub_devel/49/build/tmp/deploy/images/qemuarm64/is
ar-image-ci-debian-bookworm-qemuarm64.ext4': No such file or directory

Should be either changed the bullseye here (not yet checked if it
works). Or, we need to build arm64-bullseye image in build tests
before.

> -- 
> 2.43.0
>
Jan Kiszka July 30, 2024, 9:45 p.m. UTC | #2 
On 30.07.24 09:44, Uladzimir Bely wrote:
> On Fri, 2024-07-19 at 18:38 +0200, 'Jan Kiszka' via isar-users wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This plugs the two example recipes for loading container images into
>> VM-based testing. The test consists of running 'true' in the
>> installed
>> alpine images.
>>
>> Rather than enabling the ci user to do password-less sudo, this uses
>> su
>> with the piped-in password. Another trick needed is to poll for the
>> images because loading is performed asynchronously.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>  .../recipes-core/images/isar-image-ci.bb      |  2 ++
>>  testsuite/citest.py                           | 21
>> +++++++++++++++++++
>>  2 files changed, 23 insertions(+)
>>
>> diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-
>> test/recipes-core/images/isar-image-ci.bb
>> index e5d51e6e..9133da74 100644
>> --- a/meta-test/recipes-core/images/isar-image-ci.bb
>> +++ b/meta-test/recipes-core/images/isar-image-ci.bb
>> @@ -16,6 +16,7 @@ IMAGE_INSTALL += "sshd-regen-keys"
>>  
>>  # qemuamd64-bookworm
>>  WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
>> +IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-
>> img prebuilt-podman-img"
>>  
>>  # qemuamd64-bullseye
>>  IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz tar.gz"
>> @@ -51,3 +52,4 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= "
>> ${SYSTEMD_BOOTLOADER_INSTALL}
>>  # qemuarm64-bookworm
>>  IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
>>  IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= "
>> ${GRUB_BOOTLOADER_INSTALL}"
>> +IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-
>> img prebuilt-podman-img"
>> diff --git a/testsuite/citest.py b/testsuite/citest.py
>> index 8dd907d0..539c9440 100755
>> --- a/testsuite/citest.py
>> +++ b/testsuite/citest.py
>> @@ -522,3 +522,24 @@ class VmBootTestFull(CIBaseTest):
>>          self.init()
>>          self.vm_start('mipsel','bookworm', image='isar-image-ci',
>>                        script='test_kernel_module.sh example_module')
>> +
>> +
>> +    def test_amd64_bookworm_prebuilt_containers(self):
>> +        self.init()
>> +        self.vm_start('amd64', 'bookworm', image='isar-image-ci',
>> +                      cmd='echo root | su -c \'' \
>> +                          'PATH=\$PATH:/usr/sbin;' \
>> +                          'for n in \$(seq 30); do docker images |
>> grep -q alpine && break; sleep 10; done;' \
>> +                          'docker run --rm
>> quay.io/libpod/alpine:3.10.2 true && ' \
>> +                          'for n in \$(seq 30); do podman images |
>> grep -q alpine && break; sleep 10; done;' \
>> +                          'podman run --rm
>> quay.io/libpod/alpine:latest true\'')
>> +
>> +    def test_arm64_bookworm_prebuilt_containers(self):
>> +        self.init()
>> +        self.vm_start('arm64', 'bookworm', image='isar-image-ci',
>> +                      cmd='echo root | su -c \'' \
>> +                          'PATH=\$PATH:/usr/sbin;' \
>> +                          'for n in \$(seq 30); do docker images |
>> grep -q alpine && break; sleep 10; done;' \
>> +                          'docker run --rm
>> quay.io/libpod/alpine:3.10.2 true && ' \
>> +                          'for n in \$(seq 30); do podman images |
>> grep -q alpine && break; sleep 10; done;' \
>> +                          'podman run --rm
>> quay.io/libpod/alpine:latest true\'')
>
> This appears to fail in CI:
>
> [stdlog] 2024-07-30 07:29:37,721 avocado.app ERROR| qemu-system-
> aarch64: -drive
> file=/workspace/build/isar_ub_devel/49/build/tmp/deploy/images/qemuarm6
> 4/isar-image-ci-debian-bookworm-
> qemuarm64.ext4,if=none,format=raw,id=hd0: Could not open
> '/workspace/build/isar_ub_devel/49/build/tmp/deploy/images/qemuarm64/is
> ar-image-ci-debian-bookworm-qemuarm64.ext4': No such file or directory
>
> Should be either changed the bullseye here (not yet checked if it
> works). Or, we need to build arm64-bullseye image in build tests
> before.

bullseye is legacy.

Unfortunately, the testsuite lacks a dependency model. Here,
VmBootTestFull depends on CrossTest and NoCrossTest - apparently this
was not the case so far.

I'm still wondering why we are using such an unhandy and non-standard
test framework if not even such basic things are expressed - or are even
expressible?

Jan
Uladzimir Bely Aug. 5, 2024, 7:20 a.m. UTC | #3 
On Tue, 2024-07-30 at 23:45 +0200, Jan Kiszka wrote:
> On 30.07.24 09:44, Uladzimir Bely wrote:
> > On Fri, 2024-07-19 at 18:38 +0200, 'Jan Kiszka' via isar-users
> > wrote:
> > > From: Jan Kiszka <jan.kiszka@siemens.com>
> > > 
> > > This plugs the two example recipes for loading container images
> > > into
> > > VM-based testing. The test consists of running 'true' in the
> > > installed
> > > alpine images.
> > > 
> > > Rather than enabling the ci user to do password-less sudo, this
> > > uses
> > > su
> > > with the piped-in password. Another trick needed is to poll for
> > > the
> > > images because loading is performed asynchronously.
> > > 
> > > Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> > > ---
> > >  .../recipes-core/images/isar-image-ci.bb      |  2 ++
> > >  testsuite/citest.py                           | 21
> > > +++++++++++++++++++
> > >  2 files changed, 23 insertions(+)
> > > 
> > > diff --git a/meta-test/recipes-core/images/isar-image-ci.bb
> > > b/meta-
> > > test/recipes-core/images/isar-image-ci.bb
> > > index e5d51e6e..9133da74 100644
> > > --- a/meta-test/recipes-core/images/isar-image-ci.bb
> > > +++ b/meta-test/recipes-core/images/isar-image-ci.bb
> > > @@ -16,6 +16,7 @@ IMAGE_INSTALL += "sshd-regen-keys"
> > >  
> > >  # qemuamd64-bookworm
> > >  WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
> > > +IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-
> > > docker-
> > > img prebuilt-podman-img"
> > >  
> > >  # qemuamd64-bullseye
> > >  IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz
> > > tar.gz"
> > > @@ -51,3 +52,4 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm
> > > ?= "
> > > ${SYSTEMD_BOOTLOADER_INSTALL}
> > >  # qemuarm64-bookworm
> > >  IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
> > >  IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= "
> > > ${GRUB_BOOTLOADER_INSTALL}"
> > > +IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-
> > > docker-
> > > img prebuilt-podman-img"
> > > diff --git a/testsuite/citest.py b/testsuite/citest.py
> > > index 8dd907d0..539c9440 100755
> > > --- a/testsuite/citest.py
> > > +++ b/testsuite/citest.py
> > > @@ -522,3 +522,24 @@ class VmBootTestFull(CIBaseTest):
> > >          self.init()
> > >          self.vm_start('mipsel','bookworm', image='isar-image-
> > > ci',
> > >                        script='test_kernel_module.sh
> > > example_module')
> > > +
> > > +
> > > +    def test_amd64_bookworm_prebuilt_containers(self):
> > > +        self.init()
> > > +        self.vm_start('amd64', 'bookworm', image='isar-image-
> > > ci',
> > > +                      cmd='echo root | su -c \'' \
> > > +                          'PATH=\$PATH:/usr/sbin;' \
> > > +                          'for n in \$(seq 30); do docker images
> > > |
> > > grep -q alpine && break; sleep 10; done;' \
> > > +                          'docker run --rm
> > > quay.io/libpod/alpine:3.10.2 true && ' \
> > > +                          'for n in \$(seq 30); do podman images
> > > |
> > > grep -q alpine && break; sleep 10; done;' \
> > > +                          'podman run --rm
> > > quay.io/libpod/alpine:latest true\'')
> > > +
> > > +    def test_arm64_bookworm_prebuilt_containers(self):
> > > +        self.init()
> > > +        self.vm_start('arm64', 'bookworm', image='isar-image-
> > > ci',
> > > +                      cmd='echo root | su -c \'' \
> > > +                          'PATH=\$PATH:/usr/sbin;' \
> > > +                          'for n in \$(seq 30); do docker images
> > > |
> > > grep -q alpine && break; sleep 10; done;' \
> > > +                          'docker run --rm
> > > quay.io/libpod/alpine:3.10.2 true && ' \
> > > +                          'for n in \$(seq 30); do podman images
> > > |
> > > grep -q alpine && break; sleep 10; done;' \
> > > +                          'podman run --rm
> > > quay.io/libpod/alpine:latest true\'')
> > 
> > This appears to fail in CI:
> > 
> > [stdlog] 2024-07-30 07:29:37,721 avocado.app ERROR| qemu-system-
> > aarch64: -drive
> > file=/workspace/build/isar_ub_devel/49/build/tmp/deploy/images/qemu
> > arm6
> > 4/isar-image-ci-debian-bookworm-
> > qemuarm64.ext4,if=none,format=raw,id=hd0: Could not open
> > '/workspace/build/isar_ub_devel/49/build/tmp/deploy/images/qemuarm6
> > 4/is
> > ar-image-ci-debian-bookworm-qemuarm64.ext4': No such file or
> > directory
> > 
> > Should be either changed the bullseye here (not yet checked if it
> > works). Or, we need to build arm64-bullseye image in build tests
> > before.
> 
> bullseye is legacy.
> 

Hello.

I rebased the patchset on top of "[PATCH] testsuite: Build bookworm
arm64 image in no-cross test". So, now this test doesn't fail this way.

But there were other issues related to the backslashed '\$'. I couldn't
catch this until applied one more useful patch: "[PATCH] testsuite:
Save ssh command stdout/stderr in case of failure" that allowed to see
syntax error here.

Also, flake8 appeared to recognize this as "W605 invalid escape
sequence '\$'".

I've just sent a replacement of the patch that fixes the issue and
python style here.

You could prepare pathset v5, or I could merge it as is after testing.

> Unfortunately, the testsuite lacks a dependency model. Here,
> VmBootTestFull depends on CrossTest and NoCrossTest - apparently this
> was not the case so far.
> 
> I'm still wondering why we are using such an unhandy and non-standard
> test framework if not even such basic things are expressed - or are
> even
> expressible?
> 
> Jan
>
diff mbox series

Patch

diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb
index e5d51e6e..9133da74 100644
--- a/meta-test/recipes-core/images/isar-image-ci.bb
+++ b/meta-test/recipes-core/images/isar-image-ci.bb
@@ -16,6 +16,7 @@  IMAGE_INSTALL += "sshd-regen-keys"
 
 # qemuamd64-bookworm
 WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
+IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
 
 # qemuamd64-bullseye
 IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz tar.gz"
@@ -51,3 +52,4 @@  IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= " ${SYSTEMD_BOOTLOADER_INSTALL}
 # qemuarm64-bookworm
 IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
 IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= " ${GRUB_BOOTLOADER_INSTALL}"
+IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 8dd907d0..539c9440 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -522,3 +522,24 @@  class VmBootTestFull(CIBaseTest):
         self.init()
         self.vm_start('mipsel','bookworm', image='isar-image-ci',
                       script='test_kernel_module.sh example_module')
+
+
+    def test_amd64_bookworm_prebuilt_containers(self):
+        self.init()
+        self.vm_start('amd64', 'bookworm', image='isar-image-ci',
+                      cmd='echo root | su -c \'' \
+                          'PATH=\$PATH:/usr/sbin;' \
+                          'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \
+                          'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \
+                          'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \
+                          'podman run --rm quay.io/libpod/alpine:latest true\'')
+
+    def test_arm64_bookworm_prebuilt_containers(self):
+        self.init()
+        self.vm_start('arm64', 'bookworm', image='isar-image-ci',
+                      cmd='echo root | su -c \'' \
+                          'PATH=\$PATH:/usr/sbin;' \
+                          'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \
+                          'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \
+                          'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \
+                          'podman run --rm quay.io/libpod/alpine:latest true\'')