| Message ID | 20241106124108.220323-1-felix.moessbauer@siemens.com |
|---|---|
| State | Accepted, archived |
| Headers | show |
| Series | [1/1] improve apt fetching from rate-limited snapshot mirrors | expand |
On 06.11.24 13:41, Felix Moessbauer wrote: > The snapshot mirrors apply strict rate limiting, making it impossible to > use them with a default apt configuration. On the other hand, apt does > not obey the retry-after http response header, but uses a exponential > backoff mechanism (retry after 1,2,4,...) seconds. By that, the > currently used 3 retries are basically useless as apt aborts way before > the rate-limit expires. > > We change that to retry at most 10 times with a futher upper limit of 10 > minutes (both only when using ISAR_USE_APT_SNAPSHOT). This is a > compromise between not stalling failed builds but also making them more > reliable. > > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> > --- > doc/user_manual.md | 4 +++- > meta/classes/rootfs.bbclass | 5 ++++- > meta/conf/bitbake.conf | 2 ++ > 3 files changed, 9 insertions(+), 2 deletions(-) > > diff --git a/doc/user_manual.md b/doc/user_manual.md > index 284847c8..4b07f903 100644 > --- a/doc/user_manual.md > +++ b/doc/user_manual.md > @@ -433,7 +433,9 @@ Some other variables include: > - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host. > - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES. > - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of the live mirror. Optional. > - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. Optional. > + - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. Optional. > + - `ISAR_APT_RETRIES` - Number of apt fetching retries before giving up. Optional > + - `ISAR_APT_DELAY_MAX` - Maximum time in seconds apt performs retries. Optional > - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported distros. > - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Unix timestamp of the apt snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not overwritten. (Consider `ISAR_APT_SNAPSHOT_DATE` for a more user friendly format) > - `ISAR_APT_SNAPSHOT_DATE` - Timestamp in upstream format (e.g. `20240702T082400Z`) of the apt snapshot. Overrides `ISAR_APT_SNAPSHOT_TIMESTAMP` if set. Otherwise, will be automatically derived from `ISAR_APT_SNAPSHOT_TIMESTAMP` > diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass > index c7011508..03e80018 100644 > --- a/meta/classes/rootfs.bbclass > +++ b/meta/classes/rootfs.bbclass > @@ -148,7 +148,10 @@ rootfs_configure_apt() { > > mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d' > { > - echo 'Acquire::Retries "3";' > + echo 'Acquire::Retries "${ISAR_APT_RETRIES}";' > + if [ -n "${ISAR_APT_DELAY_MAX}" ]; then > + echo 'Acquire::Retries::Delay::Maximum "${ISAR_APT_DELAY_MAX}";' > + fi > if [ -n "${ISAR_APT_DL_LIMIT}" ]; then > echo 'Acquire::http::Dl-Limit "${ISAR_APT_DL_LIMIT}";' > echo 'Acquire::https::Dl-Limit "${ISAR_APT_DL_LIMIT}";' > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 13966a62..5fe50787 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -148,6 +148,8 @@ export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}" > SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251" > # Debian snapshots > ISAR_USE_APT_SNAPSHOT ??= "0" > +ISAR_APT_RETRIES ??= "${@'10' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else '3'}" > +ISAR_APT_DELAY_MAX ??= "${@'600' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''}" > ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}" > > # Default parallelism and resource usage for xz Seems to help, just unbroke the riscv build of isar-cip-core with that which works against snapshot.d.o so far. jan
On Wed, 2024-11-06 at 13:41 +0100, Felix Moessbauer wrote: > The snapshot mirrors apply strict rate limiting, making it impossible > to > use them with a default apt configuration. Hi, can we please merge this soon-ish, if there are no objections? We already have multiple downstream layers that carry this patch to be able to build against snapshots again. At the same time I'm working on improve the performance of snapshot.d.o , together with the upstream maintainers. Best regards, Felix > On the other hand, apt does > not obey the retry-after http response header, but uses a exponential > backoff mechanism (retry after 1,2,4,...) seconds. By that, the > currently used 3 retries are basically useless as apt aborts way > before > the rate-limit expires. > > We change that to retry at most 10 times with a futher upper limit of > 10 > minutes (both only when using ISAR_USE_APT_SNAPSHOT). This is a > compromise between not stalling failed builds but also making them > more > reliable. > > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> > --- > doc/user_manual.md | 4 +++- > meta/classes/rootfs.bbclass | 5 ++++- > meta/conf/bitbake.conf | 2 ++ > 3 files changed, 9 insertions(+), 2 deletions(-) > > diff --git a/doc/user_manual.md b/doc/user_manual.md > index 284847c8..4b07f903 100644 > --- a/doc/user_manual.md > +++ b/doc/user_manual.md > @@ -433,7 +433,9 @@ Some other variables include: > - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to > DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt > bootstrap repo for the host. > - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the > default URI in the format `ftp.debian.org my.preferred.mirror`. This > variable is optional. PREMIRRORS will be used only for the build. The > final images will have the sources list as mentioned in > DISTRO_APT_SOURCES. > - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of > the live mirror. Optional. > - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. > Optional. > + - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. > Optional. > + - `ISAR_APT_RETRIES` - Number of apt fetching retries before giving > up. Optional > + - `ISAR_APT_DELAY_MAX` - Maximum time in seconds apt performs > retries. Optional > - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to > `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported > distros. > - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Unix timestamp of the apt > snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not > overwritten. (Consider `ISAR_APT_SNAPSHOT_DATE` for a more user > friendly format) > - `ISAR_APT_SNAPSHOT_DATE` - Timestamp in upstream format (e.g. > `20240702T082400Z`) of the apt snapshot. Overrides > `ISAR_APT_SNAPSHOT_TIMESTAMP` if set. Otherwise, will be > automatically derived from `ISAR_APT_SNAPSHOT_TIMESTAMP` > diff --git a/meta/classes/rootfs.bbclass > b/meta/classes/rootfs.bbclass > index c7011508..03e80018 100644 > --- a/meta/classes/rootfs.bbclass > +++ b/meta/classes/rootfs.bbclass > @@ -148,7 +148,10 @@ rootfs_configure_apt() { > > mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d' > { > - echo 'Acquire::Retries "3";' > + echo 'Acquire::Retries "${ISAR_APT_RETRIES}";' > + if [ -n "${ISAR_APT_DELAY_MAX}" ]; then > + echo 'Acquire::Retries::Delay::Maximum > "${ISAR_APT_DELAY_MAX}";' > + fi > if [ -n "${ISAR_APT_DL_LIMIT}" ]; then > echo 'Acquire::http::Dl-Limit "${ISAR_APT_DL_LIMIT}";' > echo 'Acquire::https::Dl-Limit "${ISAR_APT_DL_LIMIT}";' > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 13966a62..5fe50787 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -148,6 +148,8 @@ export SOURCE_DATE_EPOCH ?= > "${@get_source_date_epoch_value(d)}" > SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251" > # Debian snapshots > ISAR_USE_APT_SNAPSHOT ??= "0" > +ISAR_APT_RETRIES ??= "${@'10' if > bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else '3'}" > +ISAR_APT_DELAY_MAX ??= "${@'600' if > bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''}" > ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}" > > # Default parallelism and resource usage for xz
On Wed, 2024-11-06 at 13:41 +0100, 'Felix Moessbauer' via isar-users wrote: > The snapshot mirrors apply strict rate limiting, making it impossible > to > use them with a default apt configuration. On the other hand, apt > does > not obey the retry-after http response header, but uses a exponential > backoff mechanism (retry after 1,2,4,...) seconds. By that, the > currently used 3 retries are basically useless as apt aborts way > before > the rate-limit expires. > > We change that to retry at most 10 times with a futher upper limit of > 10 > minutes (both only when using ISAR_USE_APT_SNAPSHOT). This is a > compromise between not stalling failed builds but also making them > more > reliable. > > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> > --- > doc/user_manual.md | 4 +++- > meta/classes/rootfs.bbclass | 5 ++++- > meta/conf/bitbake.conf | 2 ++ > 3 files changed, 9 insertions(+), 2 deletions(-) > > diff --git a/doc/user_manual.md b/doc/user_manual.md > index 284847c8..4b07f903 100644 > --- a/doc/user_manual.md > +++ b/doc/user_manual.md > @@ -433,7 +433,9 @@ Some other variables include: > - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to > DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt > bootstrap repo for the host. > - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the > default URI in the format `ftp.debian.org my.preferred.mirror`. This > variable is optional. PREMIRRORS will be used only for the build. The > final images will have the sources list as mentioned in > DISTRO_APT_SOURCES. > - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of > the live mirror. Optional. > - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. > Optional. > + - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. > Optional. > + - `ISAR_APT_RETRIES` - Number of apt fetching retries before giving > up. Optional > + - `ISAR_APT_DELAY_MAX` - Maximum time in seconds apt performs > retries. Optional > - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to > `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported > distros. > - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Unix timestamp of the apt > snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not > overwritten. (Consider `ISAR_APT_SNAPSHOT_DATE` for a more user > friendly format) > - `ISAR_APT_SNAPSHOT_DATE` - Timestamp in upstream format (e.g. > `20240702T082400Z`) of the apt snapshot. Overrides > `ISAR_APT_SNAPSHOT_TIMESTAMP` if set. Otherwise, will be > automatically derived from `ISAR_APT_SNAPSHOT_TIMESTAMP` > diff --git a/meta/classes/rootfs.bbclass > b/meta/classes/rootfs.bbclass > index c7011508..03e80018 100644 > --- a/meta/classes/rootfs.bbclass > +++ b/meta/classes/rootfs.bbclass > @@ -148,7 +148,10 @@ rootfs_configure_apt() { > > mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d' > { > - echo 'Acquire::Retries "3";' > + echo 'Acquire::Retries "${ISAR_APT_RETRIES}";' > + if [ -n "${ISAR_APT_DELAY_MAX}" ]; then > + echo 'Acquire::Retries::Delay::Maximum > "${ISAR_APT_DELAY_MAX}";' > + fi > if [ -n "${ISAR_APT_DL_LIMIT}" ]; then > echo 'Acquire::http::Dl-Limit "${ISAR_APT_DL_LIMIT}";' > echo 'Acquire::https::Dl-Limit "${ISAR_APT_DL_LIMIT}";' > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 13966a62..5fe50787 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -148,6 +148,8 @@ export SOURCE_DATE_EPOCH ?= > "${@get_source_date_epoch_value(d)}" > SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251" > # Debian snapshots > ISAR_USE_APT_SNAPSHOT ??= "0" > +ISAR_APT_RETRIES ??= "${@'10' if > bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else '3'}" > +ISAR_APT_DELAY_MAX ??= "${@'600' if > bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''}" > ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}" > > # Default parallelism and resource usage for xz > -- > 2.39.5 > Applied to next, thanks.
diff --git a/doc/user_manual.md b/doc/user_manual.md index 284847c8..4b07f903 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -433,7 +433,9 @@ Some other variables include: - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host. - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES. - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of the live mirror. Optional. - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. Optional. + - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. Optional. + - `ISAR_APT_RETRIES` - Number of apt fetching retries before giving up. Optional + - `ISAR_APT_DELAY_MAX` - Maximum time in seconds apt performs retries. Optional - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported distros. - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Unix timestamp of the apt snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not overwritten. (Consider `ISAR_APT_SNAPSHOT_DATE` for a more user friendly format) - `ISAR_APT_SNAPSHOT_DATE` - Timestamp in upstream format (e.g. `20240702T082400Z`) of the apt snapshot. Overrides `ISAR_APT_SNAPSHOT_TIMESTAMP` if set. Otherwise, will be automatically derived from `ISAR_APT_SNAPSHOT_TIMESTAMP` diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index c7011508..03e80018 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -148,7 +148,10 @@ rootfs_configure_apt() { mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d' { - echo 'Acquire::Retries "3";' + echo 'Acquire::Retries "${ISAR_APT_RETRIES}";' + if [ -n "${ISAR_APT_DELAY_MAX}" ]; then + echo 'Acquire::Retries::Delay::Maximum "${ISAR_APT_DELAY_MAX}";' + fi if [ -n "${ISAR_APT_DL_LIMIT}" ]; then echo 'Acquire::http::Dl-Limit "${ISAR_APT_DL_LIMIT}";' echo 'Acquire::https::Dl-Limit "${ISAR_APT_DL_LIMIT}";' diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 13966a62..5fe50787 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -148,6 +148,8 @@ export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}" SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251" # Debian snapshots ISAR_USE_APT_SNAPSHOT ??= "0" +ISAR_APT_RETRIES ??= "${@'10' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else '3'}" +ISAR_APT_DELAY_MAX ??= "${@'600' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''}" ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}" # Default parallelism and resource usage for xz
The snapshot mirrors apply strict rate limiting, making it impossible to use them with a default apt configuration. On the other hand, apt does not obey the retry-after http response header, but uses a exponential backoff mechanism (retry after 1,2,4,...) seconds. By that, the currently used 3 retries are basically useless as apt aborts way before the rate-limit expires. We change that to retry at most 10 times with a futher upper limit of 10 minutes (both only when using ISAR_USE_APT_SNAPSHOT). This is a compromise between not stalling failed builds but also making them more reliable. Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> --- doc/user_manual.md | 4 +++- meta/classes/rootfs.bbclass | 5 ++++- meta/conf/bitbake.conf | 2 ++ 3 files changed, 9 insertions(+), 2 deletions(-)