From patchwork Wed Jul 26 06:14:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "MOESSBAUER, Felix" X-Patchwork-Id: 2985 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Wed, 26 Jul 2023 08:15:29 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f64.google.com (mail-wr1-f64.google.com [209.85.221.64]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 36Q6FSBC021228 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 26 Jul 2023 08:15:28 +0200 Received: by mail-wr1-f64.google.com with SMTP id ffacd0b85a97d-3176ace3f58sf785914f8f.0 for ; Tue, 25 Jul 2023 23:15:28 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1690352123; cv=pass; d=google.com; s=arc-20160816; b=nlBw/4XTtGqJJzE6i2bkYnF+son2vEBVDw8DlmDYPVT9z4v+hA452qUgQBnsaubXKX Rn17EFWbjfcQtmnkn7UrgMcBHF99EAiH0Exiqs3CKXUhIUYuB+Ngop8NoszYAurYUCqx eI4DsZLIOVxamH8iI4MlhzPG+ElyoMhzSzyYsfgWYXqARUiNgq5hvdofJDfadohGcuNd iwkK5trGmzlj6L7WtYxDCHKcsD4oeSjle/00vRVmN2EQgI8P04K1HrCXGoXe07d4g6xd St/CZCR0EWpdVdF63Pjv4QD6tdHbZa/AtopWJMrL5j6p6251s3f7AZC5CoD6JRGsBGe4 DPBA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=knSdZ1E/HlqAJdLcFLv7Al2UGigiu8AWaLo6+p0L3mk=; fh=vELPoCEDH1a+uFwvXJ1LTTAooXVqv7NJCdnkYbEPMbo=; b=pQVpPoq4PS7eX+vKF7CEVpgOTMLiyp/HOYqw0cS9IfHjRFg+mSMiygd4sw7rdw7tKU /2YzU/X1uP1xKmkdazoHxdhn1lSvl1su6JK//8SyNANhltLXcAbHd7dJYOyq9r1bOLFe ZaPibWlpTLVEc51n2eJenqi2z6/kygPh5Us90s13c6qdLHVK1dCRIdTH0o49POuHtmAE ktPfnfT+T8lwJd88fCb8PW8txtCUenzZVP0rNfBx4FVqC803ckAp1sp+/eVJZWM9Spsk kjaK7Q/jlmLnPpLKiquKmEF74o9w+CGx7EOgKg8aMYYfMKkgmyCr37ZH5XORTjd0pcqF J3Sg== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=ePzJ82YB; spf=pass (google.com: domain of fm-72506-202307260615204c8e238e95951742a6-lnyxeb@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-72506-202307260615204c8e238e95951742a6-LnyxeB@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20221208; t=1690352123; x=1690956923; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=knSdZ1E/HlqAJdLcFLv7Al2UGigiu8AWaLo6+p0L3mk=; b=OjHpGMdofxuoG1MqqxxEAfkwkY3Uj9wH0edVTkhHZSeUufO0iDBlD9nzSnm0tVmJO1 FrzrvTMQ71+PRTYAZofVqaoqh/VTwTgdcAO9MWx/A3sjGP8vpZMIetkl7N/4vnP6fGCS gqK3z47VxwLkrIcRk2ZHSP1TgB6hT3M1t1bwasTGkOQicpqILHGxuXMALX5cfj/Iybp/ oFpSXcyfmTCnaJ0pRHDURAqDBs9W5iqJpQr5O/G8SKB2Ok0WIMqHZoKN++OkX4O6NCol wt7O/prKo6XRGRSsyS52hWHRCSVmid1oQeKruJxLVr3l2Nz+jsA2qT9MxQAG4ZToSsyv wd9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690352123; x=1690956923; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=knSdZ1E/HlqAJdLcFLv7Al2UGigiu8AWaLo6+p0L3mk=; b=hBcOkJZyZgNuT+qoO1a/VhbkWQdNYT5qh1S4Vfua1oxlV3U+mBiuu5Y6vdSE/wOkCp t3csAe+wHbIHXlxYj7HRPZJQvB1e3jhIKAGjR3PdbC3RXU9ziJPXAP3OA3sII9sYbWGP JCSL106/pY3YAKpZPikh9qSPuGUwtVCXNl9nwOm8EcfRG6V7K/KppreYGRArZnyaYtkY j9ncTKapl/CGxXL5IZbjnf2xZ7+VkpXA5dWfEUZTtn90LUPQKrYiehgal3XVhXO9JkT3 bDxS0LPrf0BpT2yrhHnzaFrH5wXYGFd79y5/tKcXswRzOevkci4zcn5riaRNpt6HHGJx MH/g== X-Gm-Message-State: ABy/qLb6ghcTNf/b7qs6v7yf4+mISt61G9PsgD27M+k3jeHc7XPI893R et/wOhfJL/uqTUJZsfi2Lkk= X-Google-Smtp-Source: APBJJlFQtrpryW3vxeIbB5ZujIPvM6B4N8YQ17QU1TVW3712kl/xSTXeG9zdVz0pN1KK0ct0EeyHyQ== X-Received: by 2002:a05:600c:2219:b0:3fa:99d6:47a4 with SMTP id z25-20020a05600c221900b003fa99d647a4mr540494wml.22.1690352122561; Tue, 25 Jul 2023 23:15:22 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:5113:b0:3fa:8731:d752 with SMTP id o19-20020a05600c511300b003fa8731d752ls575425wms.0.-pod-prod-08-eu; Tue, 25 Jul 2023 23:15:20 -0700 (PDT) X-Received: by 2002:adf:fecf:0:b0:317:643a:4fa7 with SMTP id q15-20020adffecf000000b00317643a4fa7mr668190wrs.26.1690352120704; Tue, 25 Jul 2023 23:15:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690352120; cv=none; d=google.com; s=arc-20160816; b=lkDLdq85ueFh49NBTSgVvMnc9uGcOeB+vUIEbsDVKhQk6JKxV2mWgqP9GR38Wh6GPS tEvoVmFlenosi9HU7O4sjz/mbEFFqQEldXad5CzggsRnWCoWVsJ2ZjBO7y4Ncu7o5xjz ZNLX8lNjn5jZ3XHJp3HHY8g4d2D96Xp1QssRkwJbFwcgZeiJ2eVBGXE/H59c7I1IlTp4 1pJLENhGJwc2eE+oTQBvVDDIuh8VQdwISfL7O7ILL6WOdloNi4DhIDD9YI2sOMqbSMbQ flkiPi86NuuhVXzAUyKGgL3kaJoH57tCsJiG0Q3BO6hCMElf/oyDsvl0QF8yhG8SslKk aiZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=8QtjhtISr9qSbtTl060N51267L84nMVhcVL9qFNeZwo=; fh=pJ48im6kvLu3Nw+SYlsSQKXN/ec/H+0X8C6odOd+3MU=; b=RUNO89AkWmNgVUd5UVewcaxgJtpP2vXzqDa5BmREOzJF5/uF4d09oWgoqsNd81Y1R7 T7gNPt2kyqTVrJ+G9FoqzxPRXeySB+7OQiIdT1qzlByI/SbGeDQVCKZBjjfH3qX17PzO ef8mnsLniJJLHUAEqyKdbG0kj45bpn2nHsqy6UQ7M04kMXO0CdeIbZQgWMlj0YsHucoU Wfs4sfWcIaQwBH8/Qu7k6Bzsq/NrQEUQnuV1ado/XeBJGT2nD1AWyd4BTC3qhC1Tax+K bWXCkLXBVTjZ//Aq+n+XvPcq2Da+ZEd/vK6M3rzxW8z2Wg3GsNsLg7W7z2l7oAR/s5nd VeIw== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=ePzJ82YB; spf=pass (google.com: domain of fm-72506-202307260615204c8e238e95951742a6-lnyxeb@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-72506-202307260615204c8e238e95951742a6-LnyxeB@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-226.siemens.flowmailer.net (mta-65-226.siemens.flowmailer.net. [185.136.65.226]) by gmr-mx.google.com with ESMTPS id bn15-20020a056000060f00b00317685aa000si253559wrb.4.2023.07.25.23.15.20 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Jul 2023 23:15:20 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-72506-202307260615204c8e238e95951742a6-lnyxeb@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) client-ip=185.136.65.226; Received: by mta-65-226.siemens.flowmailer.net with ESMTPSA id 202307260615204c8e238e95951742a6 for ; Wed, 26 Jul 2023 08:15:20 +0200 X-Patchwork-Original-From: "'Felix Moessbauer' via isar-users" From: "MOESSBAUER, Felix" To: isar-users@googlegroups.com Cc: jan.kiszka@siemens.com, Felix Moessbauer Subject: [PATCH 1/1] deduplicate apt sources.list based on filenames Date: Wed, 26 Jul 2023 06:14:57 +0000 Message-Id: <20230726061457.2297344-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-72506:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=ePzJ82YB; spf=pass (google.com: domain of fm-72506-202307260615204c8e238e95951742a6-lnyxeb@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-72506-202307260615204c8e238e95951742a6-LnyxeB@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= When combining layers, it can happen that the same DISTRO_APT_SOURCES entry is added multiple times. This creates duplicate entries in the bootstrap.list, which is considered invalid by apt. To solve this, we deduplicate the values in the [HOST_]DISTRO_APT_SOURCES variable. In addition, this also avoids adding the same SRC_URI entry twice. Note, that this only solves the trivial case that the duplicate entries come from the same file. However in case duplicate entries are added from multiple files with different names, the :remove syntax can be used to remove these for the affected targets. Signed-off-by: Felix Moessbauer --- meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index b94ae0bd..8af73a9b 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -59,8 +59,8 @@ python () { filename = os.path.relpath(fetcher.localpath(key), topdir) d.appendVar("THIRD_PARTY_APT_KEYFILES", " ${TOPDIR}/%s" % filename) - distro_apt_sources = d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_SOURCES") or "" - for file in distro_apt_sources.split(): + distro_apt_sources = get_aptsources_list(d) + for file in distro_apt_sources: d.appendVar("SRC_URI", " file://%s" % file) distro_apt_preferences = d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_PREFERENCES") or "" @@ -149,10 +149,11 @@ def aggregate_aptsources_list(d, file_list, file_out): def get_aptsources_list(d): import errno + from collections import OrderedDict apt_sources_var = d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_SOURCES" - list = (d.getVar(apt_sources_var) or "").split() + apt_sources_list = list(OrderedDict.fromkeys((d.getVar(apt_sources_var) or "").split())) ret = [] - for p in list: + for p in apt_sources_list: try: f = bb.parse.resolve_file(p, d) ret.append(f) @@ -251,9 +252,7 @@ python do_apt_config_prepare() { apt_sources_out = d.getVar("APTSRCS") apt_sources_init_out = d.getVar("APTSRCS_INIT") - apt_sources_list = ( - d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_SOURCES") or "" - ).split() + apt_sources_list = get_aptsources_list(d) aggregate_files(d, apt_sources_list, apt_sources_init_out) aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)