| Message ID | 20240923142704.389438-1-cedric.hombourger@siemens.com |
|---|---|
| State | Accepted, archived |
| Headers | show |
| Series | image: copy .list files from SRC_URI to the image | expand |
On 23.09.24 16:27, 'Cedric Hombourger' via isar-users wrote: > There are use-cases where the .list files used by Isar / apt during the > build would not be the same as the initial .list files presented to the > end-user of the image (e.g. build from base-apt or from a local mirror). > If any .list files are listed in the SRC_URI of an image recipe, they > will be copied to /etc/apt/sources.list.d/ verbatim and in lieu of the > bootstrap sources that were used by Isar. How well does this interact with the list swappings we do already, e.g. for using snapshots during build but then upstream on the target? Jan > > Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com> > --- > doc/user_manual.md | 14 ++++++++++++++ > meta/classes/image.bbclass | 18 ++++++++++++++++++ > 2 files changed, 32 insertions(+) > > diff --git a/doc/user_manual.md b/doc/user_manual.md > index 847f4b07..89c3a293 100644 > --- a/doc/user_manual.md > +++ b/doc/user_manual.md > @@ -769,6 +769,20 @@ inherit image > > ``` > > +If the resulting image should not ship apt sources used during the build but custom ones (e.g. for end-users to point > +to an external or simply different server when they "apt-get update", custom list files may be listed in `SRC_URI`: > +Isar will copy them to `/etc/apt/sources.list.d/` and omit bootstrap sources. Possible use-cases: > + > + * image built from base-apt (which is by definition local to the build host) > + > + * image built from an internal mirror, not reachable by devices running the produced image > + > + * ship template list files for the end-user to edit (e.g. letting him uncomment `deb` or `deb-src` entries) > + > +It should be noted that Isar will not validate or even load supplied list files: they are simply copied verbatim to > +the root file-system just before creating an image out of it (loading sources from the network would make the build > +non-reproducible). > + > ### Additional Notes > > The distribution selected via the `DISTRO` variable may need to run a post-configuration script after the root file-system > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index c29d9e26..0c162aa3 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -392,6 +392,18 @@ python do_deploy() { > } > addtask deploy before do_build after do_image > > +def apt_list_files(d): > + lists = [] > + sources = d.getVar("SRC_URI").split() > + for s in sources: > + _, _, local, _, _, parm = bb.fetch.decodeurl(s) > + base, ext = os.path.splitext(os.path.basename(local)) > + if ext and ext in (".list"): > + lists.append(local) > + return lists > + > +IMAGE_LISTS = "${@ ' '.join(apt_list_files(d)) }" > + > do_rootfs_finalize() { > sudo -s <<'EOSUDO' > set -e > @@ -429,6 +441,12 @@ do_rootfs_finalize() { > mv "${ROOTFSDIR}/etc/apt/sources-list" \ > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > fi > + if [ -n "${IMAGE_LISTS}" ]; then > + rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > + for l in ${IMAGE_LISTS}; do > + cp "${WORKDIR}"/${l} "${ROOTFSDIR}/etc/apt/sources.list.d/" > + done > + fi > > rm -f "${ROOTFSDIR}/run/blkid/blkid.tab" > rm -f "${ROOTFSDIR}/run/blkid/blkid.tab.old"
On Wed, 2024-09-25 at 18:04 +0200, Jan Kiszka wrote: > On 23.09.24 16:27, 'Cedric Hombourger' via isar-users wrote: > > There are use-cases where the .list files used by Isar / apt during > > the > > build would not be the same as the initial .list files presented to > > the > > end-user of the image (e.g. build from base-apt or from a local > > mirror). > > If any .list files are listed in the SRC_URI of an image recipe, > > they > > will be copied to /etc/apt/sources.list.d/ verbatim and in lieu of > > the > > bootstrap sources that were used by Isar. > > How well does this interact with the list swappings we do already, > e.g. > for using snapshots during build but then upstream on the target? Rechecked to be sure. Lines above my changes are responsible for the swap (/etc/apt/sources-list => /etc/apt/sources.list.d/bootstrap.list) head of /etc/apt/sources.list.d/bootstrap.list before do_rootfs_finalize() deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20240925T180156Z/ bookworm main contrib non-free-firmware non-free head of /etc/apt/sources-list before do_rootfs_finalize() deb http://deb.debian.org/debian bookworm main contrib non-free- firmware non-free After do_rootfs_finalize(), sources-list is gone and bootstrap.list is no lomger pointing to the selected snapshot With my changes and with user-supplied .list files in SRC_URI bootstrap.list is removed and only user-supplied .list files are present in /etc/apt/sources.list.d/ Hope this clarifies Cedric > > Jan > > > > > Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com> > > --- > > doc/user_manual.md | 14 ++++++++++++++ > > meta/classes/image.bbclass | 18 ++++++++++++++++++ > > 2 files changed, 32 insertions(+) > > > > diff --git a/doc/user_manual.md b/doc/user_manual.md > > index 847f4b07..89c3a293 100644 > > --- a/doc/user_manual.md > > +++ b/doc/user_manual.md > > @@ -769,6 +769,20 @@ inherit image > > > > ``` > > > > +If the resulting image should not ship apt sources used during the > > build but custom ones (e.g. for end-users to point > > +to an external or simply different server when they "apt-get > > update", custom list files may be listed in `SRC_URI`: > > +Isar will copy them to `/etc/apt/sources.list.d/` and omit > > bootstrap sources. Possible use-cases: > > + > > + * image built from base-apt (which is by definition local to the > > build host) > > + > > + * image built from an internal mirror, not reachable by devices > > running the produced image > > + > > + * ship template list files for the end-user to edit (e.g. letting > > him uncomment `deb` or `deb-src` entries) > > + > > +It should be noted that Isar will not validate or even load > > supplied list files: they are simply copied verbatim to > > +the root file-system just before creating an image out of it > > (loading sources from the network would make the build > > +non-reproducible). > > + > > ### Additional Notes > > > > The distribution selected via the `DISTRO` variable may need to > > run a post-configuration script after the root file-system > > diff --git a/meta/classes/image.bbclass > > b/meta/classes/image.bbclass > > index c29d9e26..0c162aa3 100644 > > --- a/meta/classes/image.bbclass > > +++ b/meta/classes/image.bbclass > > @@ -392,6 +392,18 @@ python do_deploy() { > > } > > addtask deploy before do_build after do_image > > > > +def apt_list_files(d): > > + lists = [] > > + sources = d.getVar("SRC_URI").split() > > + for s in sources: > > + _, _, local, _, _, parm = bb.fetch.decodeurl(s) > > + base, ext = os.path.splitext(os.path.basename(local)) > > + if ext and ext in (".list"): > > + lists.append(local) > > + return lists > > + > > +IMAGE_LISTS = "${@ ' '.join(apt_list_files(d)) }" > > + > > do_rootfs_finalize() { > > sudo -s <<'EOSUDO' > > set -e > > @@ -429,6 +441,12 @@ do_rootfs_finalize() { > > mv "${ROOTFSDIR}/etc/apt/sources-list" \ > > > > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > > fi > > + if [ -n "${IMAGE_LISTS}" ]; then > > + rm -f > > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > > + for l in ${IMAGE_LISTS}; do > > + cp "${WORKDIR}"/${l} > > "${ROOTFSDIR}/etc/apt/sources.list.d/" > > + done > > + fi > > > > rm -f "${ROOTFSDIR}/run/blkid/blkid.tab" > > rm -f "${ROOTFSDIR}/run/blkid/blkid.tab.old" >
On Mon, 2024-09-23 at 16:27 +0200, 'Cedric Hombourger' via isar-users wrote: > There are use-cases where the .list files used by Isar / apt during > the > build would not be the same as the initial .list files presented to > the > end-user of the image (e.g. build from base-apt or from a local > mirror). > If any .list files are listed in the SRC_URI of an image recipe, they > will be copied to /etc/apt/sources.list.d/ verbatim and in lieu of > the > bootstrap sources that were used by Isar. > > Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com> > --- > doc/user_manual.md | 14 ++++++++++++++ > meta/classes/image.bbclass | 18 ++++++++++++++++++ > 2 files changed, 32 insertions(+) > > diff --git a/doc/user_manual.md b/doc/user_manual.md > index 847f4b07..89c3a293 100644 > --- a/doc/user_manual.md > +++ b/doc/user_manual.md > @@ -769,6 +769,20 @@ inherit image > > ``` > > +If the resulting image should not ship apt sources used during the > build but custom ones (e.g. for end-users to point > +to an external or simply different server when they "apt-get > update", custom list files may be listed in `SRC_URI`: > +Isar will copy them to `/etc/apt/sources.list.d/` and omit bootstrap > sources. Possible use-cases: > + > + * image built from base-apt (which is by definition local to the > build host) > + > + * image built from an internal mirror, not reachable by devices > running the produced image > + > + * ship template list files for the end-user to edit (e.g. letting > him uncomment `deb` or `deb-src` entries) > + > +It should be noted that Isar will not validate or even load supplied > list files: they are simply copied verbatim to > +the root file-system just before creating an image out of it > (loading sources from the network would make the build > +non-reproducible). > + > ### Additional Notes > > The distribution selected via the `DISTRO` variable may need to run > a post-configuration script after the root file-system > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index c29d9e26..0c162aa3 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -392,6 +392,18 @@ python do_deploy() { > } > addtask deploy before do_build after do_image > > +def apt_list_files(d): > + lists = [] > + sources = d.getVar("SRC_URI").split() > + for s in sources: > + _, _, local, _, _, parm = bb.fetch.decodeurl(s) > + base, ext = os.path.splitext(os.path.basename(local)) > + if ext and ext in (".list"): > + lists.append(local) > + return lists > + > +IMAGE_LISTS = "${@ ' '.join(apt_list_files(d)) }" > + > do_rootfs_finalize() { > sudo -s <<'EOSUDO' > set -e > @@ -429,6 +441,12 @@ do_rootfs_finalize() { > mv "${ROOTFSDIR}/etc/apt/sources-list" \ > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > fi > + if [ -n "${IMAGE_LISTS}" ]; then > + rm -f > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > + for l in ${IMAGE_LISTS}; do > + cp "${WORKDIR}"/${l} > "${ROOTFSDIR}/etc/apt/sources.list.d/" > + done > + fi > > rm -f "${ROOTFSDIR}/run/blkid/blkid.tab" > rm -f "${ROOTFSDIR}/run/blkid/blkid.tab.old" > -- > 2.39.2 > Hello all. If there are no objection and futher discussion, we are ready to merge the patch.
On Mon, 2024-09-23 at 16:27 +0200, 'Cedric Hombourger' via isar-users wrote: > There are use-cases where the .list files used by Isar / apt during > the > build would not be the same as the initial .list files presented to > the > end-user of the image (e.g. build from base-apt or from a local > mirror). > If any .list files are listed in the SRC_URI of an image recipe, they > will be copied to /etc/apt/sources.list.d/ verbatim and in lieu of > the > bootstrap sources that were used by Isar. > > Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com> > --- > doc/user_manual.md | 14 ++++++++++++++ > meta/classes/image.bbclass | 18 ++++++++++++++++++ > 2 files changed, 32 insertions(+) > > diff --git a/doc/user_manual.md b/doc/user_manual.md > index 847f4b07..89c3a293 100644 > --- a/doc/user_manual.md > +++ b/doc/user_manual.md > @@ -769,6 +769,20 @@ inherit image > > ``` > > +If the resulting image should not ship apt sources used during the > build but custom ones (e.g. for end-users to point > +to an external or simply different server when they "apt-get > update", custom list files may be listed in `SRC_URI`: > +Isar will copy them to `/etc/apt/sources.list.d/` and omit bootstrap > sources. Possible use-cases: > + > + * image built from base-apt (which is by definition local to the > build host) > + > + * image built from an internal mirror, not reachable by devices > running the produced image > + > + * ship template list files for the end-user to edit (e.g. letting > him uncomment `deb` or `deb-src` entries) > + > +It should be noted that Isar will not validate or even load supplied > list files: they are simply copied verbatim to > +the root file-system just before creating an image out of it > (loading sources from the network would make the build > +non-reproducible). > + > ### Additional Notes > > The distribution selected via the `DISTRO` variable may need to run > a post-configuration script after the root file-system > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index c29d9e26..0c162aa3 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -392,6 +392,18 @@ python do_deploy() { > } > addtask deploy before do_build after do_image > > +def apt_list_files(d): > + lists = [] > + sources = d.getVar("SRC_URI").split() > + for s in sources: > + _, _, local, _, _, parm = bb.fetch.decodeurl(s) > + base, ext = os.path.splitext(os.path.basename(local)) > + if ext and ext in (".list"): > + lists.append(local) > + return lists > + > +IMAGE_LISTS = "${@ ' '.join(apt_list_files(d)) }" > + > do_rootfs_finalize() { > sudo -s <<'EOSUDO' > set -e > @@ -429,6 +441,12 @@ do_rootfs_finalize() { > mv "${ROOTFSDIR}/etc/apt/sources-list" \ > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > fi > + if [ -n "${IMAGE_LISTS}" ]; then > + rm -f > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > + for l in ${IMAGE_LISTS}; do > + cp "${WORKDIR}"/${l} > "${ROOTFSDIR}/etc/apt/sources.list.d/" > + done > + fi > > rm -f "${ROOTFSDIR}/run/blkid/blkid.tab" > rm -f "${ROOTFSDIR}/run/blkid/blkid.tab.old" > -- > 2.39.2 > Applied to next, thanks.
diff --git a/doc/user_manual.md b/doc/user_manual.md index 847f4b07..89c3a293 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -769,6 +769,20 @@ inherit image ``` +If the resulting image should not ship apt sources used during the build but custom ones (e.g. for end-users to point +to an external or simply different server when they "apt-get update", custom list files may be listed in `SRC_URI`: +Isar will copy them to `/etc/apt/sources.list.d/` and omit bootstrap sources. Possible use-cases: + + * image built from base-apt (which is by definition local to the build host) + + * image built from an internal mirror, not reachable by devices running the produced image + + * ship template list files for the end-user to edit (e.g. letting him uncomment `deb` or `deb-src` entries) + +It should be noted that Isar will not validate or even load supplied list files: they are simply copied verbatim to +the root file-system just before creating an image out of it (loading sources from the network would make the build +non-reproducible). + ### Additional Notes The distribution selected via the `DISTRO` variable may need to run a post-configuration script after the root file-system diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index c29d9e26..0c162aa3 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -392,6 +392,18 @@ python do_deploy() { } addtask deploy before do_build after do_image +def apt_list_files(d): + lists = [] + sources = d.getVar("SRC_URI").split() + for s in sources: + _, _, local, _, _, parm = bb.fetch.decodeurl(s) + base, ext = os.path.splitext(os.path.basename(local)) + if ext and ext in (".list"): + lists.append(local) + return lists + +IMAGE_LISTS = "${@ ' '.join(apt_list_files(d)) }" + do_rootfs_finalize() { sudo -s <<'EOSUDO' set -e @@ -429,6 +441,12 @@ do_rootfs_finalize() { mv "${ROOTFSDIR}/etc/apt/sources-list" \ "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" fi + if [ -n "${IMAGE_LISTS}" ]; then + rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" + for l in ${IMAGE_LISTS}; do + cp "${WORKDIR}"/${l} "${ROOTFSDIR}/etc/apt/sources.list.d/" + done + fi rm -f "${ROOTFSDIR}/run/blkid/blkid.tab" rm -f "${ROOTFSDIR}/run/blkid/blkid.tab.old"
There are use-cases where the .list files used by Isar / apt during the build would not be the same as the initial .list files presented to the end-user of the image (e.g. build from base-apt or from a local mirror). If any .list files are listed in the SRC_URI of an image recipe, they will be copied to /etc/apt/sources.list.d/ verbatim and in lieu of the bootstrap sources that were used by Isar. Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com> --- doc/user_manual.md | 14 ++++++++++++++ meta/classes/image.bbclass | 18 ++++++++++++++++++ 2 files changed, 32 insertions(+)