From patchwork Tue Nov 29 17:58:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 2249 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 29 Nov 2022 18:58:26 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-il1-f185.google.com (mail-il1-f185.google.com [209.85.166.185]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 2ATHwOsT003024 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 29 Nov 2022 18:58:25 +0100 Received: by mail-il1-f185.google.com with SMTP id k3-20020a92c243000000b0030201475a6bsf12730410ilo.9; Tue, 29 Nov 2022 09:58:25 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1669744698; cv=pass; d=google.com; s=arc-20160816; b=JuSBo4Dt8sHaFW741rM5Dr/vwCQuFkGiLCPgaASvBb927RlnDCxOrKV8b2sZkSHoyy 3BJnr2dmU5y3BJ4JI8yxWU1CMVJWYaw3yEFnwz4qQnXgTitkl2ZnRL6bTaWAnxvSV4oj 2EN+UgkiQ39KnigCqW6gF2f/XWdYnAm3aEfdM6q7Lb1uuH2kC4l9698PnfaOc1Sk32wV bmm5zbMSNs4yN5OXcnr+tOAH2crAp2UievcWfKu+8biEnDzqtYnDaIwgGk0NdcK2gseE 04stYsE/VjdlNe1+9uYV9R+gXJs+1NQFSCJ6OGI3QSf65Y1fP1AWJvaudHv0ndLwlBxf 4Mpg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=MB00UmlrSLWCR7jBG5gPJ1BaL36G2Djx8k7EL/KftOc=; b=pO09pmIGHfJ8uyvKUGxNAmFIPm5ALR2mDeOn6+hky1p9ejK7KRn1OfnOkuZt+qBWG7 0LRvMiKlvwXOFwuInrtddt6Nj7+KhYD3pGkFPhygLo2Dee9U1cC27wcJR8atQ0KIDNy6 S0ud9Aw/6VBiBTAqTzXXiba63QPLBv7xGkrN1LkDX+2lFPojdi8myeJp95fXcpzSgfGo rMEDzdi8HgiXm+H8pnscCNGtgwhvlyj4a6y4RYdomEUv3XelkPEmG0xezi6TVYaMcOqh Xx0x40Vv5rDiod0NR4ByWHbzX6R4mUHquQQrY0CsXptHfaEKzJbA6PpzqJM4oA8TGwx7 eLdA== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=MB00UmlrSLWCR7jBG5gPJ1BaL36G2Djx8k7EL/KftOc=; b=cuWXTzxKhu7JAjiMlBpic9NYrvAxgglMxHqmau+wPZKQHKB1uhrVcc6e0wVxY02X9w fVxgLfYaFA5s5LAUiEfyxeA19C7tT826lErSAeKiAgRW7Ka9UWuUQULPvQclKIjjeKno kIxXkOyRVopVpwyylBFONGrBO2CtWGwaQ5HOBCA5YPX2ziDLziJmVkL8LGiSMZ5KyErg 9caHqdndbBsnmV/SujUx6gvnHPFXJu0yMuz3q2ykT7yyySm3/X0uT3GHZwvHwUMe84ic L+MAz7fkersva0/Bi9TJNHXfL7bS4Xa11GtS2MviqYKS3jklp3ZWwF9rAaLGQbQbiQ0M rDPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=MB00UmlrSLWCR7jBG5gPJ1BaL36G2Djx8k7EL/KftOc=; b=tDJicL52u4vkhijX/uaZbs45SvOFcEZBqQenzcL3vsWMbucG+NDdIsR0HNaoX073kY FnsFpAzcEft4rjt2yuPSUZesei0wK8E2KpMCu+q3p/lppEhOAui4RGtnm2yn2Z/Fo/cq T9q/OehEXgcsDv2pZv8u4okTfuRCRISLlWNyJsU8hgMPRGLuMTevKRyfpp3NTcmVf15A bhJcTCPyME2VEg9bYRg1J8ZUkzA+lebfRdC0+N7o+jpz0EFbnlf5aSZ6U3KDXQBo3gg2 9CgVJ8HSMoVvsZbQ7PLKQhX5jzQ+FImdl/Ef6pw6npUPgjAJveGTTlWTvOYfeEKWfdB9 UeKQ== Sender: isar-users@googlegroups.com X-Gm-Message-State: ANoB5pkttlR3cQIvvFFelPybbnrQdGTZzZcRVfMmZIzK73aMq+/xj5md iciH4kM4u6H5f9+ezjVHp0M= X-Google-Smtp-Source: AA0mqf6vJYIhVLLomw3+dU2KQLNQkY7a5Z0o6inAN793sweG7HeGyWBPZ5ZEYtW+PjCPgZQdpjXnww== X-Received: by 2002:a05:6602:220c:b0:6df:3e12:b640 with SMTP id n12-20020a056602220c00b006df3e12b640mr11151494ion.124.1669744698387; Tue, 29 Nov 2022 09:58:18 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a02:22c1:0:b0:373:f0cf:a81f with SMTP id o184-20020a0222c1000000b00373f0cfa81fls1405823jao.9.-pod-prod-gmail; Tue, 29 Nov 2022 09:58:17 -0800 (PST) X-Received: by 2002:a02:a399:0:b0:363:a2f8:5223 with SMTP id y25-20020a02a399000000b00363a2f85223mr26273272jak.282.1669744697885; Tue, 29 Nov 2022 09:58:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669744697; cv=none; d=google.com; s=arc-20160816; b=Y90kV6FzeBz5xsUm21J1umNf4Uo8uAmw4+8yWfN+0xRKh0V3OC6B/Gul3qLgUPOCRk vUa6K2j60HvmgjKCe+omyoNDWBjlFQWYrqN0alkNSegiM3HsUz7OpZ/FXSwLW3UdkdRN vZ9xLnWVB1XSnGxLn/qA0cP5Pc6tnNiCDDHK6Tv10Zobc/M2giZa2xmEgMYNycRz2xqC TQsDRIpwT2Oj/kiZNewX8O46xc7SaS3i+LGzXpuv3W2VKbwf/EA1bA2P2Il9fATXHBWs /1jxQ/4BHQXjIdLkxIlIdHBYqS1sQDSo989nxWg6Oibaw/3hR6LAenjFfJoHMPsx26WM OCGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=7xMWg08YlphV+7dpSc8PUAWS6X98LOfeBQXRnL/6MQk=; b=F8xk/ft6FKRNttIb3tKhtMDVau1Pvo901BLVVzknfXVQTeDPEMT9M3EexNSNIzsbUm /Ldvmhs+RaMaoLv4Lk6St/SO1xkXXSt8K5AlQOcdig86TCVpFsqDOEbgAZaDVrIVgmWa wZjVuujYeggKOUdLrZxngjeKmp+h27vqrpY17D5ov0lzgZDY7uRUV8udhMB4e+PMsPZv irD8edy4DSItrSHQqzYp8htLpSygNW1ocKvqCk96EtndZIDDMwgbiCu/5IDy6MAyqtEA UoLiqt6+muAQYvtDwa0r7P9qx6UauTrglZch/hy9Djuk7vpLNCdT43sUmPYylVxvJP8P LEfQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id b24-20020a056602001800b00684c9b5bc7asi627805ioa.1.2022.11.29.09.58.17 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 29 Nov 2022 09:58:17 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from baighyz.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 2ATHwEbR003006 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 29 Nov 2022 18:58:15 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH 2/2] testsuite: check Isar image for reproducability. Date: Tue, 29 Nov 2022 18:58:14 +0100 Message-Id: <20221129175814.16355-3-ubely@ilbers.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221129175814.16355-1-ubely@ilbers.de> References: <20221129175814.16355-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This test uses 'diffoscope'[1] tool to check the difference between two consecutive builds and copy the differeneces in html format. [1] https://diffoscope.org/ Signed-off-by: Uladzimir Bely --- testsuite/cibase.py | 44 ++++++++++++++++++++++++++++++++++++++++++++ testsuite/citest.py | 18 ++++++++++++++++++ 2 files changed, 62 insertions(+) diff --git a/testsuite/cibase.py b/testsuite/cibase.py index 6239b4de..8a6d4c6e 100755 --- a/testsuite/cibase.py +++ b/testsuite/cibase.py @@ -2,6 +2,7 @@ import glob import os +import subprocess import re import tempfile import time @@ -55,6 +56,49 @@ class CIBaseTest(CIBuilder): # Try to build with changed configuration with no cleanup self.bitbake(targets, **kwargs) + def perform_reproducable_test(self, targets, **kwargs): + def get_bitbake_var(output, var): + ret = '' + for line in output.splitlines(): + if line.startswith(var + '='): + ret = line.split('"')[1] + return ret + + self.configure(**kwargs) + + self.log.info('Starting first build...') + self.bitbake(targets, **kwargs) + + self.move_in_build_dir('tmp', 'tmp1') + self.delete_from_build_dir('sstate-cache') + self.delete_from_build_dir('ccache') + + self.log.info('Starting second build...') + self.bitbake(targets, **kwargs) + self.bitbake(targets, **kwargs) + + self.move_in_build_dir('tmp', 'tmp2') + self.delete_from_build_dir('sstate-cache') + self.delete_from_build_dir('ccache') + + for target in targets: + output = subprocess.check_output(['bitbake', '-e', target]).decode() + + machine=get_bitbake_var(output, 'MACHINE') + distro=get_bitbake_var(output, 'DISTRO') + + cmdline = ['diffoscope'] + cmdline.extend(['--html-dir', 'diffoscope_out']) + cmdline.extend(['tmp1/deploy/images/' + machine + '/isar-image-base-' + distro + '-' + machine + '.tar.gz']) + cmdline.extend(['tmp2/deploy/images/' + machine + '/isar-image-base-' + distro + '-' + machine + '.tar.gz']) + + print(cmdline) + + p1 = subprocess.call(cmdline) + + if p1.returncode: + self.fail('Image is not reproducable') + def perform_ccache_test(self, targets, **kwargs): def ccache_stats(dir, field): # Look ccache source's 'src/core/Statistic.hpp' for field meanings diff --git a/testsuite/citest.py b/testsuite/citest.py index 2dc78015..aa63e848 100755 --- a/testsuite/citest.py +++ b/testsuite/citest.py @@ -48,6 +48,24 @@ class ReproTest(CIBaseTest): finally: self.move_in_build_dir('tmp', 'tmp_repro_unsigned') +class ReproducableTest(CIBaseTest): + + """ + Test cached base repository + + :avocado: tags=reproducable + """ + def test_reproducable(self): + targets = [ + 'mc:qemuamd64-bullseye:isar-image-base' + ] + + self.init() + try: + self.perform_reproducable_test(targets) + except: + self.cancel('KFAIL') + class CcacheTest(CIBaseTest): """