mbox

[v14,0/2] CPIO & OVA Images

Message ID 20210708084238.2933-1-Quirin.Gylstorff@siemens.com
State Superseded, archived
Headers show

Message

Quirin Gylstorff July 8, 2021, 12:42 a.m. UTC
From: Quirin Gylstorff <quirin.gylstorff@siemens.com>

- Generate cpio images for rescue images
- generate ova images for VMWare & Virtualbox

Changes V2:
- initramfs modification is now a package
- ova-wic-img was renamed to virtual-machine-image
- virtual-machine-image do_convert_wic supports now other image
formats
- Add tests
- cpiogz now use chown instead of userspec

Changes V3:
- Add tests to build-ci
- add postrm to initramfs-config
- use sdimage-efi for virtualbox images
- changed maintainer

Changes V4:
- use templates from debhelper for postinst and postrm scripts

Changes V5:
- add virtual-machine-template package to add the template as extra file to the build

Changes V6:
- whitespace fix
- initramfs-config is now a inc file instead of a recipe
- meta isar contains a default implementation of intramfs-config

Changes V7:
- drop initramfs-config in favour of https://groups.google.com/g/isar-users/c/47EZAUTclZs/m/npCfE6zqCwAJ
- clarify OVA commit message
- rebase on  origin/next f5a6fdfd9aad49202093aab2158f625429eaf7c5

Changes V8:
- formatting
- correct cpio commit message

Changes in v9 by Jan Kiszka:
 - renaming "virtual-machine" -> "vm"

Changes in v10 by Jan Kiszka:
 - fix incomplete renaming "virtual-machine" -> "vm"
   (the classic "posting before testing"...)

Changes in v11:
 - fix full ci build by Jan Kiszka
 - fix missing in ovf file after envsubst

Changes in v12:
 - remove var BOOTLOADER

Changes in v13:
 - add image_undo_mounts

Changes in v14:
 - remove get_disksize to reduce image_do_mounts calls

Quirin Gylstorff (2):
  meta/classes: Generate ova image for VMWare or Virtualbox
  meta/classes: add cpiogz-img

 doc/user_manual.md                            |   1 +
 meta-isar/conf/local.conf.sample              |   2 +
 meta-isar/conf/machine/virtualbox.conf        |  15 ++
 meta-isar/conf/machine/vmware.conf            |  15 ++
 .../multiconfig/qemuamd64-buster-cpiogz.conf  |   9 +
 .../multiconfig/virtualbox-ova-buster.conf    |   8 +
 meta/classes/cpiogz-img.bbclass               |  22 +++
 meta/classes/vm-img.bbclass                   | 118 +++++++++++++
 .../vm-template/files/vm-template.ovf.tmpl    | 155 ++++++++++++++++++
 .../vm-template/vm-template_0.1.bb            |  16 ++
 scripts/ci_build.sh                           |   2 +
 11 files changed, 363 insertions(+)
 create mode 100644 meta-isar/conf/machine/virtualbox.conf
 create mode 100644 meta-isar/conf/machine/vmware.conf
 create mode 100644 meta-isar/conf/multiconfig/qemuamd64-buster-cpiogz.conf
 create mode 100644 meta-isar/conf/multiconfig/virtualbox-ova-buster.conf
 create mode 100644 meta/classes/cpiogz-img.bbclass
 create mode 100644 meta/classes/vm-img.bbclass
 create mode 100644 meta/recipes-devtools/vm-template/files/vm-template.ovf.tmpl
 create mode 100644 meta/recipes-devtools/vm-template/vm-template_0.1.bb

Comments

Anton Mikanovich Aug. 20, 2021, 9:07 a.m. UTC | #1
08.07.2021 11:42, Q. Gylstorff wrote:
> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>
> - Generate cpio images for rescue images
> - generate ova images for VMWare & Virtualbox
>
We've experienced floating issue during full CI testing if include this 
patchset:

00:14:20 INFO: Creating image(s)...
00:14:20
00:14:20 ERROR: _exec_cmd: export 
PATH=//sbin://usr/sbin://usr/bin://bin:None:$PATH;export 
PSEUDO_PREFIX=//usr;export 
PSEUDO_LOCALSTATEDIR=/workspace/build/isar_am_devel/110/build/tmp/work/debian-buster-amd64/isar-image-base-qemuamd64-wic-img/1.0-r0/rootfs/../pseudo;export 
PSEUDO_PASSWD=/workspace/build/isar_am_devel/110/build/tmp/work/debian-buster-amd64/isar-image-base-qemuamd64-wic-img/1.0-r0/rootfs;export 
PSEUDO_NOSYMLINKEXP=1;export 
PSEUDO_IGNORE_PATHS=/tmp/tmp.nSYNbkMHKq/isar-image-base-debian-buster-qemuamd64.wic/tmp.wic.4t1qlax6/rootfs_platform.2.ext4,/;/workspace/build/isar_am_devel/110/scripts/wic_fakeroot 
fsck.ext4 -pvfD 
/tmp/tmp.nSYNbkMHKq/isar-image-base-debian-buster-qemuamd64.wic/tmp.wic.4t1qlax6/rootfs_platform.2.ext4 
returned '127' instead of 0
00:14:20 output: /bin/sh: 1: 
/workspace/build/isar_am_devel/110/scripts/wic_fakeroot: not found
00:14:20
00:14:20 WARNING: exit code 1 from a shell command.

It doesn't looks like something virtual images related, so not sure if 
it was caused by this patchset.
But for some reason it was happened only if CPIO&OVA v14 was included.

The failed builds are:
http://ci.isar-build.org:8080/job/isar_am_devel/110/consoleFull
http://ci.isar-build.org:8080/job/isar_am_devel/111/consoleFull
http://ci.isar-build.org:8080/job/isar_am_devel/115/consoleFull
http://ci.isar-build.org:8080/job/isar_am_devel/123/consoleFull
Quirin Gylstorff Sept. 7, 2021, 10:48 a.m. UTC | #2
I will test it on our site. with the patchset applied to next( 
e274130b870b7e31532fe191b9932cde1d819b4b).

Quirin

On 8/20/21 11:07 AM, Anton Mikanovich wrote:
> 08.07.2021 11:42, Q. Gylstorff wrote:
>> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>
>> - Generate cpio images for rescue images
>> - generate ova images for VMWare & Virtualbox
>>
> We've experienced floating issue during full CI testing if include this 
> patchset:
> 
> 00:14:20 INFO: Creating image(s)...
> 00:14:20
> 00:14:20 ERROR: _exec_cmd: export 
> PATH=//sbin://usr/sbin://usr/bin://bin:None:$PATH;export 
> PSEUDO_PREFIX=//usr;export 
> PSEUDO_LOCALSTATEDIR=/workspace/build/isar_am_devel/110/build/tmp/work/debian-buster-amd64/isar-image-base-qemuamd64-wic-img/1.0-r0/rootfs/../pseudo;export 
> PSEUDO_PASSWD=/workspace/build/isar_am_devel/110/build/tmp/work/debian-buster-amd64/isar-image-base-qemuamd64-wic-img/1.0-r0/rootfs;export 
> PSEUDO_NOSYMLINKEXP=1;export 
> PSEUDO_IGNORE_PATHS=/tmp/tmp.nSYNbkMHKq/isar-image-base-debian-buster-qemuamd64.wic/tmp.wic.4t1qlax6/rootfs_platform.2.ext4,/;/workspace/build/isar_am_devel/110/scripts/wic_fakeroot 
> fsck.ext4 -pvfD 
> /tmp/tmp.nSYNbkMHKq/isar-image-base-debian-buster-qemuamd64.wic/tmp.wic.4t1qlax6/rootfs_platform.2.ext4 
> returned '127' instead of 0
> 00:14:20 output: /bin/sh: 1: 
> /workspace/build/isar_am_devel/110/scripts/wic_fakeroot: not found
> 00:14:20
> 00:14:20 WARNING: exit code 1 from a shell command.
> 
> It doesn't looks like something virtual images related, so not sure if 
> it was caused by this patchset.
> But for some reason it was happened only if CPIO&OVA v14 was included.
> 
> The failed builds are:
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.isar-build.org%3A8080%2Fjob%2Fisar_am_devel%2F110%2FconsoleFull&amp;data=04%7C01%7C6549bcd2-981c-4c06-8e1b-b5c6cc3441b4%40ad011.siemens.com%7C442a9219ee3b41688f2d08d963ba04f1%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637650472859000759%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=pQ8aWLsweh%2Fkml4q%2BxtAVbjLj6HpuonxhQe7HXgkc5k%3D&amp;reserved=0 
> 
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.isar-build.org%3A8080%2Fjob%2Fisar_am_devel%2F111%2FconsoleFull&amp;data=04%7C01%7C6549bcd2-981c-4c06-8e1b-b5c6cc3441b4%40ad011.siemens.com%7C442a9219ee3b41688f2d08d963ba04f1%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637650472859000759%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=si7qQ2PIB1nvCsNtTlAKVQ6zgpgoMwSA9z%2FObQV9Tok%3D&amp;reserved=0 
> 
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.isar-build.org%3A8080%2Fjob%2Fisar_am_devel%2F115%2FconsoleFull&amp;data=04%7C01%7C6549bcd2-981c-4c06-8e1b-b5c6cc3441b4%40ad011.siemens.com%7C442a9219ee3b41688f2d08d963ba04f1%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637650472859010754%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=KuV4mhnc8piuKaAfYImmx7VHzRykEyvTn6aCwWfnde8%3D&amp;reserved=0 
> 
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.isar-build.org%3A8080%2Fjob%2Fisar_am_devel%2F123%2FconsoleFull&amp;data=04%7C01%7C6549bcd2-981c-4c06-8e1b-b5c6cc3441b4%40ad011.siemens.com%7C442a9219ee3b41688f2d08d963ba04f1%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637650472859010754%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=1cDZUp%2BcwQ30sTqD2qYGKdMXpk9pN6EvF3ldB9TvrjM%3D&amp;reserved=0 
> 
>
Jan Kiszka Sept. 7, 2021, 1:06 p.m. UTC | #3
On 07.09.21 12:48, Gylstorff Quirin wrote:
> I will test it on our site. with the patchset applied to next(
> e274130b870b7e31532fe191b9932cde1d819b4b).
> 

Full CI with your patches applied passed our nightly run today.

Jan

> Quirin
> 
> On 8/20/21 11:07 AM, Anton Mikanovich wrote:
>> 08.07.2021 11:42, Q. Gylstorff wrote:
>>> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>>
>>> - Generate cpio images for rescue images
>>> - generate ova images for VMWare & Virtualbox
>>>
>> We've experienced floating issue during full CI testing if include
>> this patchset:
>>
>> 00:14:20 INFO: Creating image(s)...
>> 00:14:20
>> 00:14:20 ERROR: _exec_cmd: export
>> PATH=//sbin://usr/sbin://usr/bin://bin:None:$PATH;export
>> PSEUDO_PREFIX=//usr;export
>> PSEUDO_LOCALSTATEDIR=/workspace/build/isar_am_devel/110/build/tmp/work/debian-buster-amd64/isar-image-base-qemuamd64-wic-img/1.0-r0/rootfs/../pseudo;export
>> PSEUDO_PASSWD=/workspace/build/isar_am_devel/110/build/tmp/work/debian-buster-amd64/isar-image-base-qemuamd64-wic-img/1.0-r0/rootfs;export
>> PSEUDO_NOSYMLINKEXP=1;export
>> PSEUDO_IGNORE_PATHS=/tmp/tmp.nSYNbkMHKq/isar-image-base-debian-buster-qemuamd64.wic/tmp.wic.4t1qlax6/rootfs_platform.2.ext4,/;/workspace/build/isar_am_devel/110/scripts/wic_fakeroot
>> fsck.ext4 -pvfD
>> /tmp/tmp.nSYNbkMHKq/isar-image-base-debian-buster-qemuamd64.wic/tmp.wic.4t1qlax6/rootfs_platform.2.ext4
>> returned '127' instead of 0
>> 00:14:20 output: /bin/sh: 1:
>> /workspace/build/isar_am_devel/110/scripts/wic_fakeroot: not found
>> 00:14:20
>> 00:14:20 WARNING: exit code 1 from a shell command.
>>
>> It doesn't looks like something virtual images related, so not sure if
>> it was caused by this patchset.
>> But for some reason it was happened only if CPIO&OVA v14 was included.
>>
>> The failed builds are:
>> http://ci.isar-build.org:8080/job/isar_am_devel/110/consoleFull
>>
>> http://ci.isar-build.org:8080/job/isar_am_devel/111/consoleFull
>>
>> http://ci.isar-build.org:8080/job/isar_am_devel/115/consoleFull
>>
>> http://ci.isar-build.org:8080/job/isar_am_devel/123/consoleFull
>>
>>
Anton Mikanovich Sept. 7, 2021, 1:49 p.m. UTC | #4
07.09.2021 16:06, Jan Kiszka wrote:
> On 07.09.21 12:48, Gylstorff Quirin wrote:
>> I will test it on our site. with the patchset applied to next(
>> e274130b870b7e31532fe191b9932cde1d819b4b).
>>
> Full CI with your patches applied passed our nightly run today.
>
> Jan
>
As I've already mentioned, the issue is floating.
We have at least 4 failed builds in our CI, but even more without any 
issues (with v14 in both groups). And none builds with the same issue 
without v14 applied.
The last time it was seen at 2021-08-12, but I'm not able to reproduce 
the issue after rebase on next.

Anyway now we need to rebase again to remove image_undo_mounts that was 
gone after the last revert.
Anton Mikanovich Sept. 13, 2021, 9:39 a.m. UTC | #5
07.09.2021 16:06, Jan Kiszka wrote:
> On 07.09.21 12:48, Gylstorff Quirin wrote:
>> I will test it on our site. with the patchset applied to next(
>> e274130b870b7e31532fe191b9932cde1d819b4b).
>>
> Full CI with your patches applied passed our nightly run today.
>
> Jan
>
I was testing v15 on top of current next and the issue was not reproduced.
So propose to merge v15 but keep in mind this thread in case the issue 
will happen again.
Henning Schild Sept. 14, 2021, 10:54 a.m. UTC | #6
I kind of want to bet this has to do with mounting again.
generate_wic_image does quite a bit of layer mounting and umounting. If
mounting the base isar layer fails that script can probably "go
missing". And i am not sure those mounts will be cleaned up when the
task does not finish correctly.

regards,
Henning

Am Fri, 20 Aug 2021 12:07:53 +0300
schrieb Anton Mikanovich <amikan@ilbers.de>:

> 08.07.2021 11:42, Q. Gylstorff wrote:
> > From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
> >
> > - Generate cpio images for rescue images
> > - generate ova images for VMWare & Virtualbox
> >  
> We've experienced floating issue during full CI testing if include
> this patchset:
> 
> 00:14:20 INFO: Creating image(s)...
> 00:14:20
> 00:14:20 ERROR: _exec_cmd: export 
> PATH=//sbin://usr/sbin://usr/bin://bin:None:$PATH;export 
> PSEUDO_PREFIX=//usr;export 
> PSEUDO_LOCALSTATEDIR=/workspace/build/isar_am_devel/110/build/tmp/work/debian-buster-amd64/isar-image-base-qemuamd64-wic-img/1.0-r0/rootfs/../pseudo;export 
> PSEUDO_PASSWD=/workspace/build/isar_am_devel/110/build/tmp/work/debian-buster-amd64/isar-image-base-qemuamd64-wic-img/1.0-r0/rootfs;export 
> PSEUDO_NOSYMLINKEXP=1;export 
> PSEUDO_IGNORE_PATHS=/tmp/tmp.nSYNbkMHKq/isar-image-base-debian-buster-qemuamd64.wic/tmp.wic.4t1qlax6/rootfs_platform.2.ext4,/;/workspace/build/isar_am_devel/110/scripts/wic_fakeroot 
> fsck.ext4 -pvfD 
> /tmp/tmp.nSYNbkMHKq/isar-image-base-debian-buster-qemuamd64.wic/tmp.wic.4t1qlax6/rootfs_platform.2.ext4 
> returned '127' instead of 0
> 00:14:20 output: /bin/sh: 1: 
> /workspace/build/isar_am_devel/110/scripts/wic_fakeroot: not found
> 00:14:20
> 00:14:20 WARNING: exit code 1 from a shell command.
> 
> It doesn't looks like something virtual images related, so not sure
> if it was caused by this patchset.
> But for some reason it was happened only if CPIO&OVA v14 was included.
> 
> The failed builds are:
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.isar-build.org%3A8080%2Fjob%2Fisar_am_devel%2F110%2FconsoleFull&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7C11c07c50cd7c46b53c8d08d963ba0572%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637650473333123853%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=N%2Bwdk5CdmnSnxZSYawz9V9N5Doo5lBsWrU5bUodPNfY%3D&amp;reserved=0
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.isar-build.org%3A8080%2Fjob%2Fisar_am_devel%2F111%2FconsoleFull&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7C11c07c50cd7c46b53c8d08d963ba0572%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637650473333123853%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=AOlB1MufUym42Fx9MxA9i75lM%2FmgDYuH6pqYI93Ibek%3D&amp;reserved=0
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.isar-build.org%3A8080%2Fjob%2Fisar_am_devel%2F115%2FconsoleFull&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7C11c07c50cd7c46b53c8d08d963ba0572%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637650473333133842%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=W7WUqLyo0VrLMGiJMEUc%2Bpvoqzj7xutMRo2wjhmBPnE%3D&amp;reserved=0
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.isar-build.org%3A8080%2Fjob%2Fisar_am_devel%2F123%2FconsoleFull&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7C11c07c50cd7c46b53c8d08d963ba0572%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637650473333133842%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=4z1JroTZd%2Fdwf7T%2F4ptKvV30AfdEYvQqgu0p861EbSw%3D&amp;reserved=0
>
Henning Schild Sept. 16, 2021, 10:01 a.m. UTC | #7
I just read some logs where three wic images are created in the same
chroot.

The first one failed to umount tmp

> umount: /builds/.../build/tmp: target is busy.

Followed by the other two not being able to find wic_fakeroot.

I guess we will have to see how to react on mount/umount issues to make
them fail ... and not some follow-up.

And roughly looking at the code i guess we will have to grab a
buildchroot lock in do_wic_image. In fact i guess the locking of the
buildchroot should probably be implemented inside buildchroot_do_mounts.

Henning

Am Tue, 14 Sep 2021 12:54:45 +0200
schrieb Henning Schild <henning.schild@siemens.com>:

> I kind of want to bet this has to do with mounting again.
> generate_wic_image does quite a bit of layer mounting and umounting.
> If mounting the base isar layer fails that script can probably "go
> missing". And i am not sure those mounts will be cleaned up when the
> task does not finish correctly.
> 
> regards,
> Henning
> 
> Am Fri, 20 Aug 2021 12:07:53 +0300
> schrieb Anton Mikanovich <amikan@ilbers.de>:
> 
> > 08.07.2021 11:42, Q. Gylstorff wrote:  
> > > From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
> > >
> > > - Generate cpio images for rescue images
> > > - generate ova images for VMWare & Virtualbox
> > >    
> > We've experienced floating issue during full CI testing if include
> > this patchset:
> > 
> > 00:14:20 INFO: Creating image(s)...
> > 00:14:20
> > 00:14:20 ERROR: _exec_cmd: export 
> > PATH=//sbin://usr/sbin://usr/bin://bin:None:$PATH;export 
> > PSEUDO_PREFIX=//usr;export 
> > PSEUDO_LOCALSTATEDIR=/workspace/build/isar_am_devel/110/build/tmp/work/debian-buster-amd64/isar-image-base-qemuamd64-wic-img/1.0-r0/rootfs/../pseudo;export 
> > PSEUDO_PASSWD=/workspace/build/isar_am_devel/110/build/tmp/work/debian-buster-amd64/isar-image-base-qemuamd64-wic-img/1.0-r0/rootfs;export 
> > PSEUDO_NOSYMLINKEXP=1;export 
> > PSEUDO_IGNORE_PATHS=/tmp/tmp.nSYNbkMHKq/isar-image-base-debian-buster-qemuamd64.wic/tmp.wic.4t1qlax6/rootfs_platform.2.ext4,/;/workspace/build/isar_am_devel/110/scripts/wic_fakeroot 
> > fsck.ext4 -pvfD 
> > /tmp/tmp.nSYNbkMHKq/isar-image-base-debian-buster-qemuamd64.wic/tmp.wic.4t1qlax6/rootfs_platform.2.ext4 
> > returned '127' instead of 0
> > 00:14:20 output: /bin/sh: 1: 
> > /workspace/build/isar_am_devel/110/scripts/wic_fakeroot: not found
> > 00:14:20
> > 00:14:20 WARNING: exit code 1 from a shell command.
> > 
> > It doesn't looks like something virtual images related, so not sure
> > if it was caused by this patchset.
> > But for some reason it was happened only if CPIO&OVA v14 was
> > included.
> > 
> > The failed builds are:
> > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.isar-build.org%3A8080%2Fjob%2Fisar_am_devel%2F110%2FconsoleFull&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7C11c07c50cd7c46b53c8d08d963ba0572%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637650473333123853%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=N%2Bwdk5CdmnSnxZSYawz9V9N5Doo5lBsWrU5bUodPNfY%3D&amp;reserved=0
> > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.isar-build.org%3A8080%2Fjob%2Fisar_am_devel%2F111%2FconsoleFull&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7C11c07c50cd7c46b53c8d08d963ba0572%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637650473333123853%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=AOlB1MufUym42Fx9MxA9i75lM%2FmgDYuH6pqYI93Ibek%3D&amp;reserved=0
> > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.isar-build.org%3A8080%2Fjob%2Fisar_am_devel%2F115%2FconsoleFull&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7C11c07c50cd7c46b53c8d08d963ba0572%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637650473333133842%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=W7WUqLyo0VrLMGiJMEUc%2Bpvoqzj7xutMRo2wjhmBPnE%3D&amp;reserved=0
> > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.isar-build.org%3A8080%2Fjob%2Fisar_am_devel%2F123%2FconsoleFull&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7C11c07c50cd7c46b53c8d08d963ba0572%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637650473333133842%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=4z1JroTZd%2Fdwf7T%2F4ptKvV30AfdEYvQqgu0p861EbSw%3D&amp;reserved=0
> >   
>