From patchwork Sat Nov 1 11:13:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cedric Hombourger X-Patchwork-Id: 4478 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Sat, 01 Nov 2025 12:13:45 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f186.google.com (mail-lj1-f186.google.com [209.85.208.186]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 5A1BDiTH020022 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Sat, 1 Nov 2025 12:13:44 +0100 Received: by mail-lj1-f186.google.com with SMTP id 38308e7fff4ca-37777da28besf17002591fa.3 for ; Sat, 01 Nov 2025 04:13:44 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1761995617; cv=pass; d=google.com; s=arc-20240605; b=im9H2l8QTzHrCo0ebyiDfUmRUIRb1F7J5YO0O4EidPsRbu8uY2qp4EM9Rvc+JPdd95 TPP6/+MJjFaoQ4ipbR75DIDJRoxFWAghEnKm7J1QyrbvDYxomtGC0DyG5pRYjz9MUrES j4yEGrk1/72Ra6bfabLADJKOr2iTnolOfgnSot5Sw8oVf9YibTJN67hGkr+9XHv1NI+9 xnoSNSrsrE7lsFylz+19VS667YbcHZf03jUyTNkWLGUOXb1wihcTI5J9uB0Jtoa43zae +u7+rJG5QIJc+rwiUGOfdH6IdSXPAFYUXafCvE8WFfZ9a0fmjVntFLUVTYYNHGV08gg0 C4Ag== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=wtzl6idmEgEGb1ozV81Luxmmp7pXxR6O01JHEQK/i30=; fh=O7ENUgfO/TRcPd51h0ZpD6IV1EveCPrPEMrxbDjlmDM=; b=cKMFatb7K2Thep9z2Er0tQ6QVTGF7knKsLvZSVG7UTkCUqOxzuDA+qoHqM6rTYuY7C iNa+1kkhFtVI8aXyzB293X85iuf/IM4Xj9XViPuhYSxf40wTw2M/27d2fD4qgOw+eclR IE1AgA+x1v+fWXwpTTVv6ahAGswjsiY0YEpV9t+1rn+nZCp+pdv4fL3GtZl6SprABFMw ePuZpkRMCmO+g52VqI9dic05qosHckThEVu4DwmV7FdfNQg/POr1yAMkpl9Dz8nFPvTi cv+5bMqlq54n2LsiDWwDCxPHMX3XdFqCYu8kvwbVkwIiYRa1Tc57dnTfjMyjMrkQ4Dz3 /t6w==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=VjRp86mS; spf=pass (google.com: domain of fm-1212295-20251101111332b051a90686000207d8-x_dyuq@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-1212295-20251101111332b051a90686000207d8-X_dyUQ@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1761995617; x=1762600417; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=wtzl6idmEgEGb1ozV81Luxmmp7pXxR6O01JHEQK/i30=; b=eByJPF+yDmsRedUN3gQICnZz7edofdYE86RlFULdCSwPJspq21cHEI+GvYHIvaPX0J u4exbe5mskn9IUWrpeb9iNnmF93UWLkO6cRRsbaLgZXNom8Rrf2eWHQoedk661y+S24R 7Un6l5i3hMv6DlpNZAVUQ+9AiprmVjC1+yykUsNJBFUNClP2k1VCsW4KohLIFThHDV70 oG4iv7t0Phgm2alxR0tVJ86TrrjcS/XM8JFs//in9nNSBGZzfduwnTbmTi66Gb9WQkUp Wg1OUaVGbAePG7vhBJLngxItsjpdBWu0nySywxRZ1CDKtZnEoyLE40DnJI0XezM+V1yI k2pA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761995617; x=1762600417; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=wtzl6idmEgEGb1ozV81Luxmmp7pXxR6O01JHEQK/i30=; b=LTdd58Ci+OPPduvMkx7KdJsZBSmO6deZsIJdEp5ME1mwD4NJqsJsAc8rT7HO9NOuMB 22yvRZK7cQ9wwqXiog6yl9llK9FX9gjb93bNGJ1bAkA0XdBJjBODG8f5ku5rWw3iEGYM RSTc3J5dlP3wnft/2XLxcwkO6fND9J2JAXvolZOfsmOFnDuQQ2lFt6cu4IUgs4q4RmuJ J0DVP5XEiPJkeE/w68PA1/CjsgHrckQJ1pNEDSBe9UTl1sVjkT40CJZLson24LzcV5Yh pd0/AyQoGaPSzEZ+tC5Oq/Fucyu84F6gWkTJ/VPrOKHJCsmLpyukP2wJz+n21YqTwGbU 9peQ== X-Forwarded-Encrypted: i=2; AJvYcCVH+djBvddKW8tm5F38HAEV997z2q3Z5jb25nETfm6RMePlpSHXyOEnjTd56A+jgdne8UG2jBI=@isar-build.org X-Gm-Message-State: AOJu0YzB/PryhBNWWZDhY7a+kZgiBVnrnThNEzix8jX6sGBmOehNTZot u0HYuH0LieXDvfkEBwTpfup5RnJyyuMij031Hi+FcfkR3Y497L9/Imjo X-Google-Smtp-Source: AGHT+IEq4Wevx1YbYdaUidh318qT4JQhWeI2U/LCy/VzOtRx99rNezosC5SMA7mF88lSF0bOKYeWRg== X-Received: by 2002:a2e:82cf:0:b0:36e:93a3:979d with SMTP id 38308e7fff4ca-37a18d91a51mr15600521fa.19.1761995617248; Sat, 01 Nov 2025 04:13:37 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h="Ae8XA+bR5ScddZlDfNTrGhfD9Ne/wUz/lOE1Ik8bwpiLA4yaXQ==" Received: by 2002:a2e:9042:0:b0:378:cfe9:cbdd with SMTP id 38308e7fff4ca-37a10a41997ls3910471fa.1.-pod-prod-07-eu; Sat, 01 Nov 2025 04:13:33 -0700 (PDT) X-Received: by 2002:a05:651c:555:b0:378:dd6e:104b with SMTP id 38308e7fff4ca-37a18d869d6mr23433941fa.1.1761995613588; Sat, 01 Nov 2025 04:13:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1761995613; cv=none; d=google.com; s=arc-20240605; b=ErUaNHdboNO5fDLr+7Bk3RhQDXhvepuiOu/TcOzOCoSePzHzyM4h50tUvDs0kmgpl2 WOvWa1I7VK/h+cmiu71JWNHBU0Y+4s2DRRJfRjhbTfYXTZWHOPIZEt3d1UWK3KgiCaYG F7s0psyxdmyVjZrWQLFOT8+wHCWaAVBMhNYOZKYlcBXn8wjMFclICrSk++TRGgaboObc xo5rodtGCXQHQgDEeOug0QgFvORfdlgtuPJ2vGzU01nqREEvDUn0JzUecphO6gGvS4Eq fyIzznQazS0gon2DhYnBTvNDfvw+eR0DuEn/Y91TLbWxoup50PYx0RPIP0tPWRXJBcNz 8l/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=1vrnl5t5lJNonguDbK+uxWM2/HX5EfttiGXoRAAnClc=; fh=1z37pEVhqwMLlkT4FDzOga7XBsM6Rzv6sXOq0pipxqo=; b=KJ/VWyn81+XVrQT5zBsDqOiRDBpkApVmN+jrwPFCQsHmvpROu67t7lpPiJdXiadqDM 7RdMbHBZt2UpdcqICeYMAafUqdEC6KxLFnleMlTP1aljCUfs8h6Jqw5oYqnst3er72cr +2ybrJV3BdfwMfC0h/RJFCG0SSQR+ijJz71+EVrtVcNzHHnhKJPPQpOVRr0U97vzas+D L7R9LSS5Dz5+CHOBcuRDgV43n2Y4Lyb+U/nQGFeunQGMhdjj1qs52uJ/pD5d1YhTjt/g xNoy0c4bmXy3AEx6aZLlWEeZqXiM+oS5SOegNAYRcNXeg7M0rj2oxBMzmo0XketqdRYP ZQyQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=VjRp86mS; spf=pass (google.com: domain of fm-1212295-20251101111332b051a90686000207d8-x_dyuq@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-1212295-20251101111332b051a90686000207d8-X_dyUQ@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-227.siemens.flowmailer.net (mta-65-227.siemens.flowmailer.net. [185.136.65.227]) by gmr-mx.google.com with ESMTPS id 38308e7fff4ca-37a1c067edesi1040421fa.7.2025.11.01.04.13.33 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 01 Nov 2025 04:13:33 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1212295-20251101111332b051a90686000207d8-x_dyuq@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) client-ip=185.136.65.227; Received: by mta-65-227.siemens.flowmailer.net with ESMTPSA id 20251101111332b051a90686000207d8 for ; Sat, 01 Nov 2025 12:13:32 +0100 X-Patchwork-Original-From: "'Cedric Hombourger' via isar-users" From: Cedric Hombourger To: isar-users@googlegroups.com Cc: Cedric Hombourger Subject: [PATCH 1/4] CI: support negative build tests Date: Sat, 1 Nov 2025 12:13:14 +0100 Message-ID: <20251101111326.600581-2-cedric.hombourger@siemens.com> In-Reply-To: <20251101111326.600581-1-cedric.hombourger@siemens.com> References: <20251101111326.600581-1-cedric.hombourger@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1212295:519-21489:flowmailer X-Original-Sender: cedric.hombourger@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=VjRp86mS; spf=pass (google.com: domain of fm-1212295-20251101111332b051a90686000207d8-x_dyuq@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-1212295-20251101111332b051a90686000207d8-X_dyUQ@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Cedric Hombourger Reply-To: Cedric Hombourger Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= It appears that our build tests are only check for successful builds. We also want to make sure builds fail when they are expected to. This adds a should_fail boolean to bitbake() and perform_build_test() Signed-off-by: Cedric Hombourger --- testsuite/cibase.py | 4 ++-- testsuite/cibuilder.py | 12 ++++++++---- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/testsuite/cibase.py b/testsuite/cibase.py index 6ffedcea..0a006a21 100755 --- a/testsuite/cibase.py +++ b/testsuite/cibase.py @@ -14,12 +14,12 @@ from avocado.utils import process class CIBaseTest(CIBuilder): - def perform_build_test(self, targets, **kwargs): + def perform_build_test(self, targets, should_fail=False, **kwargs): self.configure(**kwargs) self.log.info("Starting build...") - self.bitbake(targets, **kwargs) + self.bitbake(targets, should_fail=should_fail, **kwargs) def perform_wic_partition_test(self, targets, wic_deploy_parts, **kwargs): self.configure(wic_deploy_parts=wic_deploy_parts, **kwargs) diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py index cd5e70ba..366f6a1f 100755 --- a/testsuite/cibuilder.py +++ b/testsuite/cibuilder.py @@ -281,7 +281,8 @@ class CIBuilder(Test): if os.path.exists(self.build_dir + '/' + src): shutil.move(self.build_dir + '/' + src, self.build_dir + '/' + dst) - def bitbake(self, target, bitbake_cmd=None, sig_handler=None, **kwargs): + def bitbake(self, target, bitbake_cmd=None, should_fail=False, + sig_handler=None, **kwargs): self.check_init() self.log.info("===================================================") self.log.info(f"Building {str(target)}") @@ -318,13 +319,16 @@ class CIBuilder(Test): continue if fd == p1.stdout.fileno(): self.log.info(p1.stdout.readline().rstrip()) - if fd == p1.stderr.fileno(): + if fd == p1.stderr.fileno() and should_fail is False: app_log.error(p1.stderr.readline().rstrip()) if p1.poll() is not None: break p1.wait() - if p1.returncode: - self.fail("Bitbake failed") + if should_fail is False: + if p1.returncode: + self.fail("Bitbake failed") + elif p1.returncode == 0: + self.fail("Bitbake suceeded but was expected to fail!") def backupfile(self, path): self.check_init()