[v4,03/13] kas: Add mirror selection

Message ID 20230922091003.20763-4-ubely@ilbers.de
State Superseded, archived
Headers show
Series Support building Isar using kas | expand

Commit Message

Uladzimir Bely Sept. 22, 2023, 9:09 a.m. UTC
This allows to select better (faster) mirror for Debian and Ubuntu
distributions.

Isar DISTRO_APT_PREMIRRORS is used under the hood.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 Kconfig                    |  1 +
 kas/opt/Kconfig            | 42 ++++++++++++++++++++++++++++++++++++++
 kas/opt/mirror-debian.yaml | 10 +++++++++
 kas/opt/mirror-ubuntu.yaml | 10 +++++++++
 4 files changed, 63 insertions(+)
 create mode 100644 kas/opt/Kconfig
 create mode 100644 kas/opt/mirror-debian.yaml
 create mode 100644 kas/opt/mirror-ubuntu.yaml

Comments

Jan Kiszka Sept. 25, 2023, 11:34 a.m. UTC | #1
On 22.09.23 11:09, Uladzimir Bely wrote:
> This allows to select better (faster) mirror for Debian and Ubuntu
> distributions.
> 
> Isar DISTRO_APT_PREMIRRORS is used under the hood.
> 
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  Kconfig                    |  1 +
>  kas/opt/Kconfig            | 42 ++++++++++++++++++++++++++++++++++++++
>  kas/opt/mirror-debian.yaml | 10 +++++++++
>  kas/opt/mirror-ubuntu.yaml | 10 +++++++++
>  4 files changed, 63 insertions(+)
>  create mode 100644 kas/opt/Kconfig
>  create mode 100644 kas/opt/mirror-debian.yaml
>  create mode 100644 kas/opt/mirror-ubuntu.yaml
> 
> diff --git a/Kconfig b/Kconfig
> index 591fe765..a62fbf6c 100644
> --- a/Kconfig
> +++ b/Kconfig
> @@ -16,3 +16,4 @@ config KAS_BUILD_SYSTEM
>  
>  source "kas/distro/Kconfig"
>  source "kas/machine/Kconfig"
> +source "kas/opt/Kconfig"
> diff --git a/kas/opt/Kconfig b/kas/opt/Kconfig
> new file mode 100644
> index 00000000..06a9e3e5
> --- /dev/null
> +++ b/kas/opt/Kconfig
> @@ -0,0 +1,42 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2023 ilbers GmbH
> +#
> +# Authors:
> +#  Uladzimir Bely <ubely@ilbers.de>
> +
> +menu "Mirror selection"
> +
> +config MIRROR_DEBIAN
> +	bool "Use specific Debian mirror"
> +	depends on DEBIAN_BUSTER || DEBIAN_BULLSEYE || DEBIAN_BOOKWORM
> +	help
> +	  Select better local Debian mirror instead of deb.debian.org
> +
> +config MIRROR_DEBIAN_URL
> +	string "URL of Debian mirror"
> +	default "http://deb.debian.org"
> +	depends on MIRROR_DEBIAN
> +
> +config KAS_INCLUDE_MIRROR_DEBIAN
> +	string
> +	default "kas/opt/mirror-debian.yaml"
> +	depends on MIRROR_DEBIAN
> +
> +
> +config MIRROR_UBUNTU
> +	bool "Use specific Ubuntu mirror"
> +	depends on UBUNTU_FOCAL || UBUNTU_JAMMY
> +	help
> +	  Select better local Ubuntu mirror instead of archive.ubuntu.com
> +
> +config MIRROR_UBUNTU_URL
> +	string "URL of Ubuntu mirror"
> +	default "http://archive.ubuntu.com"
> +	depends on MIRROR_UBUNTU
> +
> +config KAS_INCLUDE_MIRROR_UBUNTU
> +	string
> +	default "kas/opt/mirror-ubuntu.yaml"
> +	depends on MIRROR_UBUNTU
> +
> +endmenu

This menu looks strangely empty when selecting sid[-ports]. Maybe make
it !depend on that or depend on supported distros?

Jan

> diff --git a/kas/opt/mirror-debian.yaml b/kas/opt/mirror-debian.yaml
> new file mode 100644
> index 00000000..4dd4fc17
> --- /dev/null
> +++ b/kas/opt/mirror-debian.yaml
> @@ -0,0 +1,10 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2023 ilbers GmbH
> +
> +header:
> +  version: 14
> +
> +local_conf_header:
> +  mirror-debian: |
> +    DISTRO_APT_PREMIRRORS += "\
> +      http://deb\.debian\.org ${MIRROR_DEBIAN_URL}\n"
> diff --git a/kas/opt/mirror-ubuntu.yaml b/kas/opt/mirror-ubuntu.yaml
> new file mode 100644
> index 00000000..3774b079
> --- /dev/null
> +++ b/kas/opt/mirror-ubuntu.yaml
> @@ -0,0 +1,10 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2023 ilbers GmbH
> +
> +header:
> +  version: 14
> +
> +local_conf_header:
> +  mirror-ubuntu: |
> +    DISTRO_APT_PREMIRRORS += "\
> +      http://archive\.ubuntu\.com ${MIRROR_UBUNTU_URL}\n"
Uladzimir Bely Sept. 25, 2023, 2:38 p.m. UTC | #2
On Mon, 2023-09-25 at 13:34 +0200, Jan Kiszka wrote:
> On 22.09.23 11:09, Uladzimir Bely wrote:
> > This allows to select better (faster) mirror for Debian and Ubuntu
> > distributions.
> > 
> > Isar DISTRO_APT_PREMIRRORS is used under the hood.
> > 
> > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > ---
> >  Kconfig                    |  1 +
> >  kas/opt/Kconfig            | 42
> > ++++++++++++++++++++++++++++++++++++++
> >  kas/opt/mirror-debian.yaml | 10 +++++++++
> >  kas/opt/mirror-ubuntu.yaml | 10 +++++++++
> >  4 files changed, 63 insertions(+)
> >  create mode 100644 kas/opt/Kconfig
> >  create mode 100644 kas/opt/mirror-debian.yaml
> >  create mode 100644 kas/opt/mirror-ubuntu.yaml
> > 
> > diff --git a/Kconfig b/Kconfig
> > index 591fe765..a62fbf6c 100644
> > --- a/Kconfig
> > +++ b/Kconfig
> > @@ -16,3 +16,4 @@ config KAS_BUILD_SYSTEM
> >  
> >  source "kas/distro/Kconfig"
> >  source "kas/machine/Kconfig"
> > +source "kas/opt/Kconfig"
> > diff --git a/kas/opt/Kconfig b/kas/opt/Kconfig
> > new file mode 100644
> > index 00000000..06a9e3e5
> > --- /dev/null
> > +++ b/kas/opt/Kconfig
> > @@ -0,0 +1,42 @@
> > +# This software is a part of ISAR.
> > +# Copyright (C) 2023 ilbers GmbH
> > +#
> > +# Authors:
> > +#  Uladzimir Bely <ubely@ilbers.de>
> > +
> > +menu "Mirror selection"
> > +
> > +config MIRROR_DEBIAN
> > +       bool "Use specific Debian mirror"
> > +       depends on DEBIAN_BUSTER || DEBIAN_BULLSEYE ||
> > DEBIAN_BOOKWORM
> > +       help
> > +         Select better local Debian mirror instead of
> > deb.debian.org
> > +
> > +config MIRROR_DEBIAN_URL
> > +       string "URL of Debian mirror"
> > +       default "http://deb.debian.org"
> > +       depends on MIRROR_DEBIAN
> > +
> > +config KAS_INCLUDE_MIRROR_DEBIAN
> > +       string
> > +       default "kas/opt/mirror-debian.yaml"
> > +       depends on MIRROR_DEBIAN
> > +
> > +
> > +config MIRROR_UBUNTU
> > +       bool "Use specific Ubuntu mirror"
> > +       depends on UBUNTU_FOCAL || UBUNTU_JAMMY
> > +       help
> > +         Select better local Ubuntu mirror instead of
> > archive.ubuntu.com
> > +
> > +config MIRROR_UBUNTU_URL
> > +       string "URL of Ubuntu mirror"
> > +       default "http://archive.ubuntu.com"
> > +       depends on MIRROR_UBUNTU
> > +
> > +config KAS_INCLUDE_MIRROR_UBUNTU
> > +       string
> > +       default "kas/opt/mirror-ubuntu.yaml"
> > +       depends on MIRROR_UBUNTU
> > +
> > +endmenu
> 
> This menu looks strangely empty when selecting sid[-ports]. Maybe
> make
> it !depend on that or depend on supported distros?
> 

Do you mean, hide complete menu if sid-ports/raspbian selected?

I also got an idea - not about mirrors, but in general... Maybe,
instead of always thinking on dependencies between distro and machine -
we could simply replace it with multiconfig selection?

E.g., in one menu we could simply list all possible multiconfigs. And,
even select several of them for the build simulaneously.

> Jan
> 
> > diff --git a/kas/opt/mirror-debian.yaml b/kas/opt/mirror-
> > debian.yaml
> > new file mode 100644
> > index 00000000..4dd4fc17
> > --- /dev/null
> > +++ b/kas/opt/mirror-debian.yaml
> > @@ -0,0 +1,10 @@
> > +# This software is a part of ISAR.
> > +# Copyright (C) 2023 ilbers GmbH
> > +
> > +header:
> > +  version: 14
> > +
> > +local_conf_header:
> > +  mirror-debian: |
> > +    DISTRO_APT_PREMIRRORS += "\
> > +      http://deb\.debian\.org ${MIRROR_DEBIAN_URL}\n"
> > diff --git a/kas/opt/mirror-ubuntu.yaml b/kas/opt/mirror-
> > ubuntu.yaml
> > new file mode 100644
> > index 00000000..3774b079
> > --- /dev/null
> > +++ b/kas/opt/mirror-ubuntu.yaml
> > @@ -0,0 +1,10 @@
> > +# This software is a part of ISAR.
> > +# Copyright (C) 2023 ilbers GmbH
> > +
> > +header:
> > +  version: 14
> > +
> > +local_conf_header:
> > +  mirror-ubuntu: |
> > +    DISTRO_APT_PREMIRRORS += "\
> > +      http://archive\.ubuntu\.com ${MIRROR_UBUNTU_URL}\n"
>
Jan Kiszka Sept. 25, 2023, 2:49 p.m. UTC | #3
On 25.09.23 16:38, Uladzimir Bely wrote:
> On Mon, 2023-09-25 at 13:34 +0200, Jan Kiszka wrote:
>> On 22.09.23 11:09, Uladzimir Bely wrote:
>>> This allows to select better (faster) mirror for Debian and Ubuntu
>>> distributions.
>>>
>>> Isar DISTRO_APT_PREMIRRORS is used under the hood.
>>>
>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
>>> ---
>>>  Kconfig                    |  1 +
>>>  kas/opt/Kconfig            | 42
>>> ++++++++++++++++++++++++++++++++++++++
>>>  kas/opt/mirror-debian.yaml | 10 +++++++++
>>>  kas/opt/mirror-ubuntu.yaml | 10 +++++++++
>>>  4 files changed, 63 insertions(+)
>>>  create mode 100644 kas/opt/Kconfig
>>>  create mode 100644 kas/opt/mirror-debian.yaml
>>>  create mode 100644 kas/opt/mirror-ubuntu.yaml
>>>
>>> diff --git a/Kconfig b/Kconfig
>>> index 591fe765..a62fbf6c 100644
>>> --- a/Kconfig
>>> +++ b/Kconfig
>>> @@ -16,3 +16,4 @@ config KAS_BUILD_SYSTEM
>>>  
>>>  source "kas/distro/Kconfig"
>>>  source "kas/machine/Kconfig"
>>> +source "kas/opt/Kconfig"
>>> diff --git a/kas/opt/Kconfig b/kas/opt/Kconfig
>>> new file mode 100644
>>> index 00000000..06a9e3e5
>>> --- /dev/null
>>> +++ b/kas/opt/Kconfig
>>> @@ -0,0 +1,42 @@
>>> +# This software is a part of ISAR.
>>> +# Copyright (C) 2023 ilbers GmbH
>>> +#
>>> +# Authors:
>>> +#  Uladzimir Bely <ubely@ilbers.de>
>>> +
>>> +menu "Mirror selection"
>>> +
>>> +config MIRROR_DEBIAN
>>> +       bool "Use specific Debian mirror"
>>> +       depends on DEBIAN_BUSTER || DEBIAN_BULLSEYE ||
>>> DEBIAN_BOOKWORM
>>> +       help
>>> +         Select better local Debian mirror instead of
>>> deb.debian.org
>>> +
>>> +config MIRROR_DEBIAN_URL
>>> +       string "URL of Debian mirror"
>>> +       default "http://deb.debian.org"
>>> +       depends on MIRROR_DEBIAN
>>> +
>>> +config KAS_INCLUDE_MIRROR_DEBIAN
>>> +       string
>>> +       default "kas/opt/mirror-debian.yaml"
>>> +       depends on MIRROR_DEBIAN
>>> +
>>> +
>>> +config MIRROR_UBUNTU
>>> +       bool "Use specific Ubuntu mirror"
>>> +       depends on UBUNTU_FOCAL || UBUNTU_JAMMY
>>> +       help
>>> +         Select better local Ubuntu mirror instead of
>>> archive.ubuntu.com
>>> +
>>> +config MIRROR_UBUNTU_URL
>>> +       string "URL of Ubuntu mirror"
>>> +       default "http://archive.ubuntu.com"
>>> +       depends on MIRROR_UBUNTU
>>> +
>>> +config KAS_INCLUDE_MIRROR_UBUNTU
>>> +       string
>>> +       default "kas/opt/mirror-ubuntu.yaml"
>>> +       depends on MIRROR_UBUNTU
>>> +
>>> +endmenu
>>
>> This menu looks strangely empty when selecting sid[-ports]. Maybe
>> make
>> it !depend on that or depend on supported distros?
>>
> 
> Do you mean, hide complete menu if sid-ports/raspbian selected?

Yep.

> 
> I also got an idea - not about mirrors, but in general... Maybe,
> instead of always thinking on dependencies between distro and machine -
> we could simply replace it with multiconfig selection?
> 
> E.g., in one menu we could simply list all possible multiconfigs. And,
> even select several of them for the build simulaneously.

Also possible. Only similar case I'm aware of (for whatever reason):
https://github.com/siemens/jailhouse-images/blob/master/Kconfig

Jan
Uladzimir Bely Sept. 26, 2023, 7:59 a.m. UTC | #4
On Mon, 2023-09-25 at 16:49 +0200, Jan Kiszka wrote:
> On 25.09.23 16:38, Uladzimir Bely wrote:
> > On Mon, 2023-09-25 at 13:34 +0200, Jan Kiszka wrote:
> > > On 22.09.23 11:09, Uladzimir Bely wrote:
> > > > This allows to select better (faster) mirror for Debian and
> > > > Ubuntu
> > > > distributions.
> > > > 
> > > > Isar DISTRO_APT_PREMIRRORS is used under the hood.
> > > > 
> > > > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > > > ---
> > > >  Kconfig                    |  1 +
> > > >  kas/opt/Kconfig            | 42
> > > > ++++++++++++++++++++++++++++++++++++++
> > > >  kas/opt/mirror-debian.yaml | 10 +++++++++
> > > >  kas/opt/mirror-ubuntu.yaml | 10 +++++++++
> > > >  4 files changed, 63 insertions(+)
> > > >  create mode 100644 kas/opt/Kconfig
> > > >  create mode 100644 kas/opt/mirror-debian.yaml
> > > >  create mode 100644 kas/opt/mirror-ubuntu.yaml
> > > > 
> > > > diff --git a/Kconfig b/Kconfig
> > > > index 591fe765..a62fbf6c 100644
> > > > --- a/Kconfig
> > > > +++ b/Kconfig
> > > > @@ -16,3 +16,4 @@ config KAS_BUILD_SYSTEM
> > > >  
> > > >  source "kas/distro/Kconfig"
> > > >  source "kas/machine/Kconfig"
> > > > +source "kas/opt/Kconfig"
> > > > diff --git a/kas/opt/Kconfig b/kas/opt/Kconfig
> > > > new file mode 100644
> > > > index 00000000..06a9e3e5
> > > > --- /dev/null
> > > > +++ b/kas/opt/Kconfig
> > > > @@ -0,0 +1,42 @@
> > > > +# This software is a part of ISAR.
> > > > +# Copyright (C) 2023 ilbers GmbH
> > > > +#
> > > > +# Authors:
> > > > +#  Uladzimir Bely <ubely@ilbers.de>
> > > > +
> > > > +menu "Mirror selection"
> > > > +
> > > > +config MIRROR_DEBIAN
> > > > +       bool "Use specific Debian mirror"
> > > > +       depends on DEBIAN_BUSTER || DEBIAN_BULLSEYE ||
> > > > DEBIAN_BOOKWORM
> > > > +       help
> > > > +         Select better local Debian mirror instead of
> > > > deb.debian.org
> > > > +
> > > > +config MIRROR_DEBIAN_URL
> > > > +       string "URL of Debian mirror"
> > > > +       default "http://deb.debian.org"
> > > > +       depends on MIRROR_DEBIAN
> > > > +
> > > > +config KAS_INCLUDE_MIRROR_DEBIAN
> > > > +       string
> > > > +       default "kas/opt/mirror-debian.yaml"
> > > > +       depends on MIRROR_DEBIAN
> > > > +
> > > > +
> > > > +config MIRROR_UBUNTU
> > > > +       bool "Use specific Ubuntu mirror"
> > > > +       depends on UBUNTU_FOCAL || UBUNTU_JAMMY
> > > > +       help
> > > > +         Select better local Ubuntu mirror instead of
> > > > archive.ubuntu.com
> > > > +
> > > > +config MIRROR_UBUNTU_URL
> > > > +       string "URL of Ubuntu mirror"
> > > > +       default "http://archive.ubuntu.com"
> > > > +       depends on MIRROR_UBUNTU
> > > > +
> > > > +config KAS_INCLUDE_MIRROR_UBUNTU
> > > > +       string
> > > > +       default "kas/opt/mirror-ubuntu.yaml"
> > > > +       depends on MIRROR_UBUNTU
> > > > +
> > > > +endmenu
> > > 
> > > This menu looks strangely empty when selecting sid[-ports]. Maybe
> > > make
> > > it !depend on that or depend on supported distros?
> > > 
> > 
> > Do you mean, hide complete menu if sid-ports/raspbian selected?
> 
> Yep.
> 
> > 
> > I also got an idea - not about mirrors, but in general... Maybe,
> > instead of always thinking on dependencies between distro and
> > machine -
> > we could simply replace it with multiconfig selection?
> > 
> > E.g., in one menu we could simply list all possible multiconfigs.
> > And,
> > even select several of them for the build simulaneously.
> 
> Also possible. Only similar case I'm aware of (for whatever reason):
> https://github.com/siemens/jailhouse-images/blob/master/Kconfig
> 
> Jan
> 

Send v5 where "mirrors" menu is hidden if non-debian/ubuntu distro
selected.

Regarding multiconfig support.

I've acutally prepared some work-in-progress solution. It allows to
select several "multiconfig" items additionally to what user selected
in "distro" and "machine" menus.

But it currently does not enable required options depending on
distro/machine. E.g., if user selected "ubuntu" as distro, only URLS
related for ubuntu are shown in "mirrors" menu, even if some "debian"
multiconfigs are selected.

Another moment is that supporting multiconfigs makes "distro" and
"machine" menus a bit redundant. We could simply remove them and even
get some benefits from this:
- we should not care about implementing distro/machine dependency logic
in Kconfig - we simply use those multiconfigs we support in Isar
- user can see all possible configurations in one place and they are
not confused by missing hardware targets in the menu (if wrong distro
chosen for now).

So, we have several options:

1. Leave and merge it as is and don't add multiconfigs for now.
2. Add multiconfigs and keep distro/machine and all this dependency
logic in menu.
3. Add multiconfigs as a replacement of distro/machine menus.
 
For me, the third looks a bit more reasonable.

Patch

diff --git a/Kconfig b/Kconfig
index 591fe765..a62fbf6c 100644
--- a/Kconfig
+++ b/Kconfig
@@ -16,3 +16,4 @@  config KAS_BUILD_SYSTEM
 
 source "kas/distro/Kconfig"
 source "kas/machine/Kconfig"
+source "kas/opt/Kconfig"
diff --git a/kas/opt/Kconfig b/kas/opt/Kconfig
new file mode 100644
index 00000000..06a9e3e5
--- /dev/null
+++ b/kas/opt/Kconfig
@@ -0,0 +1,42 @@ 
+# This software is a part of ISAR.
+# Copyright (C) 2023 ilbers GmbH
+#
+# Authors:
+#  Uladzimir Bely <ubely@ilbers.de>
+
+menu "Mirror selection"
+
+config MIRROR_DEBIAN
+	bool "Use specific Debian mirror"
+	depends on DEBIAN_BUSTER || DEBIAN_BULLSEYE || DEBIAN_BOOKWORM
+	help
+	  Select better local Debian mirror instead of deb.debian.org
+
+config MIRROR_DEBIAN_URL
+	string "URL of Debian mirror"
+	default "http://deb.debian.org"
+	depends on MIRROR_DEBIAN
+
+config KAS_INCLUDE_MIRROR_DEBIAN
+	string
+	default "kas/opt/mirror-debian.yaml"
+	depends on MIRROR_DEBIAN
+
+
+config MIRROR_UBUNTU
+	bool "Use specific Ubuntu mirror"
+	depends on UBUNTU_FOCAL || UBUNTU_JAMMY
+	help
+	  Select better local Ubuntu mirror instead of archive.ubuntu.com
+
+config MIRROR_UBUNTU_URL
+	string "URL of Ubuntu mirror"
+	default "http://archive.ubuntu.com"
+	depends on MIRROR_UBUNTU
+
+config KAS_INCLUDE_MIRROR_UBUNTU
+	string
+	default "kas/opt/mirror-ubuntu.yaml"
+	depends on MIRROR_UBUNTU
+
+endmenu
diff --git a/kas/opt/mirror-debian.yaml b/kas/opt/mirror-debian.yaml
new file mode 100644
index 00000000..4dd4fc17
--- /dev/null
+++ b/kas/opt/mirror-debian.yaml
@@ -0,0 +1,10 @@ 
+# This software is a part of ISAR.
+# Copyright (C) 2023 ilbers GmbH
+
+header:
+  version: 14
+
+local_conf_header:
+  mirror-debian: |
+    DISTRO_APT_PREMIRRORS += "\
+      http://deb\.debian\.org ${MIRROR_DEBIAN_URL}\n"
diff --git a/kas/opt/mirror-ubuntu.yaml b/kas/opt/mirror-ubuntu.yaml
new file mode 100644
index 00000000..3774b079
--- /dev/null
+++ b/kas/opt/mirror-ubuntu.yaml
@@ -0,0 +1,10 @@ 
+# This software is a part of ISAR.
+# Copyright (C) 2023 ilbers GmbH
+
+header:
+  version: 14
+
+local_conf_header:
+  mirror-ubuntu: |
+    DISTRO_APT_PREMIRRORS += "\
+      http://archive\.ubuntu\.com ${MIRROR_UBUNTU_URL}\n"