| Message ID | 20200930060259.12422-1-Vijaikumar_Kanagarajan@mentor.com |
|---|---|
| Headers | show |
| Series | Deb-src caching | expand |
Any review comment on this series? Thanks, Vijai Kumar K On Wednesday, September 30, 2020 at 11:33:54 AM UTC+5:30 vijaikumar_...@mentor.com wrote: > Changes in v6: > - Rebase against latest next > - Address review comments from Henning > Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3 > > Changes in v5: > The major changes in this series are as below, > - (P2) Handle cases where HOST_DISTRO!=DISTRO. This avoids mixing > of debs from different distro which inturn helps the src caching logic. > - (P5) Avoid downloading package from other builds. > - (P8) Add a reprepro based sanity test to check if the repo contains > the sources for all the debs. > - (P9) Introduce a new variable BASE_REPO_FEATURES which provides means > to enable or disable various base-apt features. (cache-deb-src for now) > - Some fixes in (P12 & P13) > Also, addressed some review comments from Henning. > Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc5 > > Changes in v4: > - Use <source package>=<version> format instead of just using <packagename> > to download the right version of source package. > > Changes in v3: > - Take care of non-existent downloads/deb-src directory. > > Changes in v2: > - Introduced additional patch to cache deb src > - Rebased on top of henning/staging4 tree > > Vijai Kumar K (13): > rootfs: Make rootfs finalize a separate task > deb-dl-dir: Cache host distro debs separately > meta: cache deb srcs as part of postprocessing > deb-dl-dir: Make debsrc_download faster > deb-dl-dir: Download files only belonging to the current image > deb-dl-dir: Factor out the mounting part > deb-dl-dir: Fix skipping of removed files > repository: Add a sanity test to check missing sources > base-apt: Introduce BASE_REPO_FEATURES > repository: Fix failures due to missing section > scripts/ci_build.sh: Enable deb-src caching > rootfs: Fix possible overwrite of existing resolv.conf > rootfs: Handle failures when postprocess is rerun > > meta-isar/conf/local.conf.sample | 4 ++ > meta/classes/deb-dl-dir.bbclass | 61 +++++++++++++++++- > meta/classes/dpkg-gbp.bbclass | 8 ++- > meta/classes/dpkg.bbclass | 8 ++- > meta/classes/image-tools-extension.bbclass | 4 +- > meta/classes/image.bbclass | 43 ++++++++++++- > meta/classes/repository.bbclass | 13 +++- > meta/classes/rootfs.bbclass | 62 +++++++------------ > .../isar-bootstrap/isar-bootstrap.inc | 12 +++- > meta/recipes-devtools/base-apt/base-apt.bb | 3 + > scripts/ci_build.sh | 1 + > 11 files changed, 166 insertions(+), 53 deletions(-) > > -- > 2.17.1 > >
On 23.11.20 16:08, vijaikumar....@gmail.com wrote: > > Any review comment on this series? > Do we have this anchored in CI via some tests already? Jan > Thanks, > Vijai Kumar K > > On Wednesday, September 30, 2020 at 11:33:54 AM UTC+5:30 > vijaikumar_...@mentor.com wrote: > > Changes in v6: > - Rebase against latest next > - Address review comments from Henning > Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3 > <https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3> > > Changes in v5: > The major changes in this series are as below, > - (P2) Handle cases where HOST_DISTRO!=DISTRO. This avoids mixing > of debs from different distro which inturn helps the src caching logic. > - (P5) Avoid downloading package from other builds. > - (P8) Add a reprepro based sanity test to check if the repo contains > the sources for all the debs. > - (P9) Introduce a new variable BASE_REPO_FEATURES which provides means > to enable or disable various base-apt features. (cache-deb-src for now) > - Some fixes in (P12 & P13) > Also, addressed some review comments from Henning. > Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc5 > <https://github.com/vj-kumar/isar/tree/vijai/debsrc5> > > Changes in v4: > - Use <source package>=<version> format instead of just using > <packagename> > to download the right version of source package. > > Changes in v3: > - Take care of non-existent downloads/deb-src directory. > > Changes in v2: > - Introduced additional patch to cache deb src > - Rebased on top of henning/staging4 tree > > Vijai Kumar K (13): > rootfs: Make rootfs finalize a separate task > deb-dl-dir: Cache host distro debs separately > meta: cache deb srcs as part of postprocessing > deb-dl-dir: Make debsrc_download faster > deb-dl-dir: Download files only belonging to the current image > deb-dl-dir: Factor out the mounting part > deb-dl-dir: Fix skipping of removed files > repository: Add a sanity test to check missing sources > base-apt: Introduce BASE_REPO_FEATURES > repository: Fix failures due to missing section > scripts/ci_build.sh: Enable deb-src caching > rootfs: Fix possible overwrite of existing resolv.conf > rootfs: Handle failures when postprocess is rerun > > meta-isar/conf/local.conf.sample | 4 ++ > meta/classes/deb-dl-dir.bbclass | 61 +++++++++++++++++- > meta/classes/dpkg-gbp.bbclass | 8 ++- > meta/classes/dpkg.bbclass | 8 ++- > meta/classes/image-tools-extension.bbclass | 4 +- > meta/classes/image.bbclass | 43 ++++++++++++- > meta/classes/repository.bbclass | 13 +++- > meta/classes/rootfs.bbclass | 62 +++++++------------ > .../isar-bootstrap/isar-bootstrap.inc | 12 +++- > meta/recipes-devtools/base-apt/base-apt.bb <http://base-apt.bb> | 3 + > scripts/ci_build.sh | 1 + > 11 files changed, 166 insertions(+), 53 deletions(-) > > -- > 2.17.1 > > -- > You received this message because you are subscribed to the Google > Groups "isar-users" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to isar-users+unsubscribe@googlegroups.com > <mailto:isar-users+unsubscribe@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/isar-users/bf7f91cf-250b-4807-9ad4-110704f89cd3n%40googlegroups.com > <https://groups.google.com/d/msgid/isar-users/bf7f91cf-250b-4807-9ad4-110704f89cd3n%40googlegroups.com?utm_medium=email&utm_source=footer>.
On Monday, November 23, 2020 at 9:08:37 PM UTC+5:30 Jan Kiszka wrote: > On 23.11.20 16:08, vijaikumar....@gmail.com wrote: > > > > Any review comment on this series? > > > > Do we have this anchored in CI via some tests already? > Hi Jan, Yes. We have it enabled in CI[1]. [1]https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/k0zKlMVhBAAJ Thanks, Vijai Kumar K > Jan > > > Thanks, > > Vijai Kumar K > > > > On Wednesday, September 30, 2020 at 11:33:54 AM UTC+5:30 > > vijaikumar_...@mentor.com wrote: > > > > Changes in v6: > > - Rebase against latest next > > - Address review comments from Henning > > Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3 > > <https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3> > > > > Changes in v5: > > The major changes in this series are as below, > > - (P2) Handle cases where HOST_DISTRO!=DISTRO. This avoids mixing > > of debs from different distro which inturn helps the src caching logic. > > - (P5) Avoid downloading package from other builds. > > - (P8) Add a reprepro based sanity test to check if the repo contains > > the sources for all the debs. > > - (P9) Introduce a new variable BASE_REPO_FEATURES which provides means > > to enable or disable various base-apt features. (cache-deb-src for now) > > - Some fixes in (P12 & P13) > > Also, addressed some review comments from Henning. > > Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc5 > > <https://github.com/vj-kumar/isar/tree/vijai/debsrc5> > > > > Changes in v4: > > - Use <source package>=<version> format instead of just using > > <packagename> > > to download the right version of source package. > > > > Changes in v3: > > - Take care of non-existent downloads/deb-src directory. > > > > Changes in v2: > > - Introduced additional patch to cache deb src > > - Rebased on top of henning/staging4 tree > > > > Vijai Kumar K (13): > > rootfs: Make rootfs finalize a separate task > > deb-dl-dir: Cache host distro debs separately > > meta: cache deb srcs as part of postprocessing > > deb-dl-dir: Make debsrc_download faster > > deb-dl-dir: Download files only belonging to the current image > > deb-dl-dir: Factor out the mounting part > > deb-dl-dir: Fix skipping of removed files > > repository: Add a sanity test to check missing sources > > base-apt: Introduce BASE_REPO_FEATURES > > repository: Fix failures due to missing section > > scripts/ci_build.sh: Enable deb-src caching > > rootfs: Fix possible overwrite of existing resolv.conf > > rootfs: Handle failures when postprocess is rerun > > > > meta-isar/conf/local.conf.sample | 4 ++ > > meta/classes/deb-dl-dir.bbclass | 61 +++++++++++++++++- > > meta/classes/dpkg-gbp.bbclass | 8 ++- > > meta/classes/dpkg.bbclass | 8 ++- > > meta/classes/image-tools-extension.bbclass | 4 +- > > meta/classes/image.bbclass | 43 ++++++++++++- > > meta/classes/repository.bbclass | 13 +++- > > meta/classes/rootfs.bbclass | 62 +++++++------------ > > .../isar-bootstrap/isar-bootstrap.inc | 12 +++- > > meta/recipes-devtools/base-apt/base-apt.bb <http://base-apt.bb> | 3 + > > scripts/ci_build.sh | 1 + > > 11 files changed, 166 insertions(+), 53 deletions(-) > > > > -- > > 2.17.1 > > > > -- > > You received this message because you are subscribed to the Google > > Groups "isar-users" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to isar-users+...@googlegroups.com > > <mailto:isar-users+...@googlegroups.com>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/isar-users/bf7f91cf-250b-4807-9ad4-110704f89cd3n%40googlegroups.com > > < > https://groups.google.com/d/msgid/isar-users/bf7f91cf-250b-4807-9ad4-110704f89cd3n%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > -- > Siemens AG, T RDA IOT > Corporate Competence Center Embedded Linux >
On 23.11.20 16:47, vijaikumar....@gmail.com wrote: > > > On Monday, November 23, 2020 at 9:08:37 PM UTC+5:30 Jan Kiszka wrote: > > On 23.11.20 16:08, vijaikumar....@gmail.com wrote: > > > > Any review comment on this series? > > > > Do we have this anchored in CI via some tests already? > > > Hi Jan, > > Yes. We have it enabled in CI[1]. > > [1]https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/k0zKlMVhBAAJ > OK, it's executed, and my one runs in our nightly builds didn't fail. But is caching effectively checked this way? Jan
On Tuesday, November 24, 2020 at 11:11:52 AM UTC+5:30 Jan Kiszka wrote: > On 23.11.20 16:47, vijaikumar....@gmail.com wrote: > > > > > > On Monday, November 23, 2020 at 9:08:37 PM UTC+5:30 Jan Kiszka wrote: > > > > On 23.11.20 16:08, vijaikumar....@gmail.com wrote: > > > > > > Any review comment on this series? > > > > > > > Do we have this anchored in CI via some tests already? > > > > > > Hi Jan, > > > > Yes. We have it enabled in CI[1]. > > > > [1]https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/k0zKlMVhBAAJ > > > > OK, it's executed, and my one runs in our nightly builds didn't fail. > But is caching effectively checked this way? > Yes. There is a sanity test[1] after base-apt creation which doesnot throw a warning. It should when some sources are missing. [1] https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/LeacMMNhBAAJ Thanks, Vijai Kumar K Jan > > -- > Siemens AG, T RDA IOT > Corporate Competence Center Embedded Linux >
Hello Vijai Kumar, On Wed, Sep 30, 2020 at 11:32:46AM +0530, Vijai Kumar K wrote: > Changes in v6: > - Rebase against latest next > - Address review comments from Henning > Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3 The code looks good to me. There is a couple of small issues, I'll reply to the respective patches. How do I use this? I've bitbake mc:qemuamd64-buster:isar-image-base, see downloads/deb-src/debian-buster, which contains only hello. Is this intended? With kind regards, Baurzhan.
On 24.11.20 20:45, Baurzhan Ismagulov wrote: > Hello Vijai Kumar, > > On Wed, Sep 30, 2020 at 11:32:46AM +0530, Vijai Kumar K wrote: >> Changes in v6: >> - Rebase against latest next >> - Address review comments from Henning >> Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3 > > The code looks good to me. There is a couple of small issues, I'll reply to the > respective patches. > > How do I use this? I've bitbake mc:qemuamd64-buster:isar-image-base, see > downloads/deb-src/debian-buster, which contains only hello. Is this intended? I would expect that everything a build pulled from deb-src goes into the cache. IIRC, only hello is rebuilt from sources in these runs. I wonder, though, and that might be a topic for some user manual extension, how this series can also be used to fetch all sources corresponding to all deployed binaries (compliance use case). Or does this require extra logic? Jan
On Wednesday, November 25, 2020 at 1:15:30 AM UTC+5:30 i...@radix50.net wrote: > Hello Vijai Kumar, > > On Wed, Sep 30, 2020 at 11:32:46AM +0530, Vijai Kumar K wrote: > > Changes in v6: > > - Rebase against latest next > > - Address review comments from Henning > > Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3 > > The code looks good to me. There is a couple of small issues, I'll reply > to the > respective patches. > > How do I use this? I've bitbake mc:qemuamd64-buster:isar-image-base, see > downloads/deb-src/debian-buster, which contains only hello. Is this > intended? > Hi Baurzhan, You might want to set BASE_REPO_FEATURES ?= "cache-deb-src" in local conf. By default its disabled. Not everyone want to cache the deb sources. It takes a lot of time. [1] https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/_ofw8MFhBAAJ Thanks, Vijai Kumar K > With kind regards, > Baurzhan. >
On 25.11.20 07:04, vijaikumar....@gmail.com wrote: > > > On Wednesday, November 25, 2020 at 1:15:30 AM UTC+5:30 i...@radix50.net > wrote: > > Hello Vijai Kumar, > > On Wed, Sep 30, 2020 at 11:32:46AM +0530, Vijai Kumar K wrote: > > Changes in v6: > > - Rebase against latest next > > - Address review comments from Henning > > Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3 > <https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3> > > The code looks good to me. There is a couple of small issues, I'll > reply to the > respective patches. > > How do I use this? I've bitbake mc:qemuamd64-buster:isar-image-base, > see > downloads/deb-src/debian-buster, which contains only hello. Is this > intended? > > > Hi Baurzhan, > > You might want to set BASE_REPO_FEATURES ?= "cache-deb-src" in local > conf. By default its disabled. Not everyone > want to cache the deb sources. It takes a lot of time. > > [1] https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/_ofw8MFhBAAJ > Please add a documentation patch to your series to explain all use cases and knobs of it. See also my other reply. Jan
On Wednesday, November 25, 2020 at 12:18:40 PM UTC+5:30 Jan Kiszka wrote: > On 25.11.20 07:04, vijaikumar....@gmail.com wrote: > > > > > > On Wednesday, November 25, 2020 at 1:15:30 AM UTC+5:30 i...@radix50.net > > wrote: > > > > Hello Vijai Kumar, > > > > On Wed, Sep 30, 2020 at 11:32:46AM +0530, Vijai Kumar K wrote: > > > Changes in v6: > > > - Rebase against latest next > > > - Address review comments from Henning > > > Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3 > > <https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3> > > > > The code looks good to me. There is a couple of small issues, I'll > > reply to the > > respective patches. > > > > How do I use this? I've bitbake mc:qemuamd64-buster:isar-image-base, > > see > > downloads/deb-src/debian-buster, which contains only hello. Is this > > intended? > > > > > > Hi Baurzhan, > > > > You might want to set BASE_REPO_FEATURES ?= "cache-deb-src" in local > > conf. By default its disabled. Not everyone > > want to cache the deb sources. It takes a lot of time. > > > > [1] https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/_ofw8MFhBAAJ > > > > Please add a documentation patch to your series to explain all use cases > and knobs of it. See also my other reply. > > Jan > Yes Jan. I am currently adding that in usermanual. Will send the patch soon. Thanks, Vijai Kumar K > > -- > Siemens AG, T RDA IOT > Corporate Competence Center Embedded Linux >
On Tue, Nov 24, 2020 at 10:04:24PM -0800, vijaikumar....@gmail.com wrote: > You might want to set BASE_REPO_FEATURES ?= "cache-deb-src" in local conf. > By default its disabled. Not everyone > want to cache the deb sources. It takes a lot of time. Thanks Vijai Kumar, will play with v8. It's good to have this in the manual. What about the issues below? Should I include them myself? p1: On Tue, Nov 24, 2020 at 08:47:11PM +0100, Baurzhan Ismagulov wrote: > On Wed, Sep 30, 2020 at 11:32:47AM +0530, Vijai Kumar K wrote: > > diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass > > index afec1cb..eae14d5 100644 > > --- a/meta/classes/rootfs.bbclass > > +++ b/meta/classes/rootfs.bbclass > > @@ -201,45 +201,6 @@ rootfs_generate_manifest () { > > ${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest > > } > > > > -ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs', 'rootfs_postprocess_finalize', '', d)}" > > Suggest also removing the 'finalize-rootfs' comment before ROOTFS_FEATURES ?= > "". p3, p9: On Tue, Nov 24, 2020 at 08:49:19PM +0100, Baurzhan Ismagulov wrote: > On Wed, Sep 30, 2020 at 11:32:49AM +0530, Vijai Kumar K wrote: > > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > > index 2391529..6ca5759 100644 > > --- a/meta/classes/image.bbclass > > +++ b/meta/classes/image.bbclass > > @@ -63,7 +63,7 @@ image_do_mounts() { > > } > > > > ROOTFSDIR = "${IMAGE_ROOTFS}" > > -ROOTFS_FEATURES += "clean-package-cache generate-manifest" > > +ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src" > > Should it be "cache-deb-src"? The same for p9. With kind regards, Baurzhan.
On Thursday, November 26, 2020 at 3:51:34 PM UTC+5:30 i...@radix50.net wrote: > On Tue, Nov 24, 2020 at 10:04:24PM -0800, vijaikumar....@gmail.com wrote: > > You might want to set BASE_REPO_FEATURES ?= "cache-deb-src" in local > conf. > > By default its disabled. Not everyone > > want to cache the deb sources. It takes a lot of time. > > Thanks Vijai Kumar, will play with v8. It's good to have this in the > manual. > > What about the issues below? Should I include them myself? > Oops. Sorry about that. I somehow missed to see these comments. I will address these in v9. Thanks, Vijai Kumar K > > > p1: > > On Tue, Nov 24, 2020 at 08:47:11PM +0100, Baurzhan Ismagulov wrote: > > On Wed, Sep 30, 2020 at 11:32:47AM +0530, Vijai Kumar K wrote: > > > diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass > > > index afec1cb..eae14d5 100644 > > > --- a/meta/classes/rootfs.bbclass > > > +++ b/meta/classes/rootfs.bbclass > > > @@ -201,45 +201,6 @@ rootfs_generate_manifest () { > > > ${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest > > > } > > > > > > -ROOTFS_POSTPROCESS_COMMAND += > "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs', > 'rootfs_postprocess_finalize', '', d)}" > > > > Suggest also removing the 'finalize-rootfs' comment before > ROOTFS_FEATURES ?= > > "". > > > p3, p9: > > On Tue, Nov 24, 2020 at 08:49:19PM +0100, Baurzhan Ismagulov wrote: > > On Wed, Sep 30, 2020 at 11:32:49AM +0530, Vijai Kumar K wrote: > > > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > > > index 2391529..6ca5759 100644 > > > --- a/meta/classes/image.bbclass > > > +++ b/meta/classes/image.bbclass > > > @@ -63,7 +63,7 @@ image_do_mounts() { > > > } > > > > > > ROOTFSDIR = "${IMAGE_ROOTFS}" > > > -ROOTFS_FEATURES += "clean-package-cache generate-manifest" > > > +ROOTFS_FEATURES += "clean-package-cache generate-manifest > cach-deb-src" > > > > Should it be "cache-deb-src"? The same for p9. > > > With kind regards, > Baurzhan. >