[RFC,1/2] dpkg-base: Handle custom source directory in do_apt_fetch

With the current do_apt_fetch implementation, it is not possible to use
a custom source directory(${S}).

apt-get source by default extracts the contents of the debian source
into folder with name <pkg>_<version>.

Add provision for specifying a custom source directory.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 meta/classes/dpkg-base.bbclass | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

Am Sat, 20 Feb 2021 01:27:18 +0530
schrieb Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>:

> With the current do_apt_fetch implementation, it is not possible to
> use a custom source directory(${S}).
> 
> apt-get source by default extracts the contents of the debian source
> into folder with name <pkg>_<version>.
> 
> Add provision for specifying a custom source directory.

I think this one is indeed worth being discussed seperately.

Could you go into detail why a custom S is required in the first place?
My guess is that we might not know PV and need to wait for that
apt-unpack to finish so we can find it in the changelog.

> Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
> ---
>  meta/classes/dpkg-base.bbclass | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/dpkg-base.bbclass
> b/meta/classes/dpkg-base.bbclass index 5c7bddc..1b94b76 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -70,7 +70,12 @@ do_apt_fetch() {
>          sudo -E chroot --userspec=$( id -u ):$( id -g )
> ${BUILDCHROOT_DIR} \ sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" &&
> cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only
> --only-source source "$2"' my_script "${DISTRO}" "${uri}" sudo -E
> chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
> -            sh -c 'cp /downloads/deb-src/"$1"/"$2"/* ${PP} && cd
> ${PP} && apt-get -y --only-source source "$2"' my_script "${DISTRO}"
> "${uri}"
> +            sh -c ' \
> +                dscfile="$(apt-get -y -qq --print-uris source "${2}"
> | cut -d " " -f2 | grep -E "*.dsc")"
> +                cd ${PP}
> +                cp /downloads/deb-src/"${1}"/"${2}"/* ${PP}
> +                dpkg-source -x "${dscfile}" "${PPS}"' \
> +                    my_script "${DISTRO}" "${uri}"

You removed the && chaining, please try a "false" in there, i guess
that sh might need a -e now

I wonder if we can just symlink debians choice to PPS, or mv, or even
"cp -a"
And make that a new task we call apt_unpack

Henning

>      done
>  
>      dpkg_undo_mounts

On Sat, Feb 20, 2021 at 1:47 PM Henning Schild
<henning.schild@siemens.com> wrote:
>
> Am Sat, 20 Feb 2021 01:27:18 +0530
> schrieb Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>:
>
> > With the current do_apt_fetch implementation, it is not possible to
> > use a custom source directory(${S}).
> >
> > apt-get source by default extracts the contents of the debian source
> > into folder with name <pkg>_<version>.
> >
> > Add provision for specifying a custom source directory.
>
> I think this one is indeed worth being discussed seperately.
>
> Could you go into detail why a custom S is required in the first place?
> My guess is that we might not know PV and need to wait for that
> apt-unpack to finish so we can find it in the changelog.

Exactly. In some cases we might not know PV, we might need to wait
for apt to see what version it is fetching. Change the APT sources and or
preferences and you might get a completely new version of the package.
We cannot effectively know ${S} for manipulating the source code from
the recipe.

>
> > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
> > ---
> >  meta/classes/dpkg-base.bbclass | 7 ++++++-
> >  1 file changed, 6 insertions(+), 1 deletion(-)
> >
> > diff --git a/meta/classes/dpkg-base.bbclass
> > b/meta/classes/dpkg-base.bbclass index 5c7bddc..1b94b76 100644
> > --- a/meta/classes/dpkg-base.bbclass
> > +++ b/meta/classes/dpkg-base.bbclass
> > @@ -70,7 +70,12 @@ do_apt_fetch() {
> >          sudo -E chroot --userspec=$( id -u ):$( id -g )
> > ${BUILDCHROOT_DIR} \ sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" &&
> > cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only
> > --only-source source "$2"' my_script "${DISTRO}" "${uri}" sudo -E
> > chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
> > -            sh -c 'cp /downloads/deb-src/"$1"/"$2"/* ${PP} && cd
> > ${PP} && apt-get -y --only-source source "$2"' my_script "${DISTRO}"
> > "${uri}"
> > +            sh -c ' \
> > +                dscfile="$(apt-get -y -qq --print-uris source "${2}"
> > | cut -d " " -f2 | grep -E "*.dsc")"
> > +                cd ${PP}
> > +                cp /downloads/deb-src/"${1}"/"${2}"/* ${PP}
> > +                dpkg-source -x "${dscfile}" "${PPS}"' \
> > +                    my_script "${DISTRO}" "${uri}"
>
> You removed the && chaining, please try a "false" in there, i guess
> that sh might need a -e now

Yes. Will add that in v2.

>
> I wonder if we can just symlink debians choice to PPS, or mv, or even
> "cp -a"

Wondering whether it brings in any added advantage?

> And make that a new task we call apt_unpack

I thought about this too. We could definitely have something like apt_unpack
that makes things clear. When I first started looking around, I didn't
expect the
unpacking to happen in fetch.

Thanks,
Vijai Kumar K

>
> Henning
>
> >      done
> >
> >      dpkg_undo_mounts
>
> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20210220090734.60a591c2%40md1za8fc.ad001.siemens.net.

Can we please get this one stand-alone on the fast-path? Just ran into
exactly the problem and now have to work around.

regards,
Henning

Am Mon, 22 Feb 2021 12:41:40 +0530
schrieb vijai kumar <vijaikumar.kanagarajan@gmail.com>:

> On Sat, Feb 20, 2021 at 1:47 PM Henning Schild
> <henning.schild@siemens.com> wrote:
> >
> > Am Sat, 20 Feb 2021 01:27:18 +0530
> > schrieb Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>:
> >  
> > > With the current do_apt_fetch implementation, it is not possible
> > > to use a custom source directory(${S}).
> > >
> > > apt-get source by default extracts the contents of the debian
> > > source into folder with name <pkg>_<version>.
> > >
> > > Add provision for specifying a custom source directory.  
> >
> > I think this one is indeed worth being discussed seperately.
> >
> > Could you go into detail why a custom S is required in the first
> > place? My guess is that we might not know PV and need to wait for
> > that apt-unpack to finish so we can find it in the changelog.  
> 
> Exactly. In some cases we might not know PV, we might need to wait
> for apt to see what version it is fetching. Change the APT sources
> and or preferences and you might get a completely new version of the
> package. We cannot effectively know ${S} for manipulating the source
> code from the recipe.
> 
> >  
> > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
> > > ---
> > >  meta/classes/dpkg-base.bbclass | 7 ++++++-
> > >  1 file changed, 6 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/meta/classes/dpkg-base.bbclass
> > > b/meta/classes/dpkg-base.bbclass index 5c7bddc..1b94b76 100644
> > > --- a/meta/classes/dpkg-base.bbclass
> > > +++ b/meta/classes/dpkg-base.bbclass
> > > @@ -70,7 +70,12 @@ do_apt_fetch() {
> > >          sudo -E chroot --userspec=$( id -u ):$( id -g )
> > > ${BUILDCHROOT_DIR} \ sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2"
> > > && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only
> > > --only-source source "$2"' my_script "${DISTRO}" "${uri}" sudo -E
> > > chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
> > > -            sh -c 'cp /downloads/deb-src/"$1"/"$2"/* ${PP} && cd
> > > ${PP} && apt-get -y --only-source source "$2"' my_script
> > > "${DISTRO}" "${uri}"
> > > +            sh -c ' \
> > > +                dscfile="$(apt-get -y -qq --print-uris source
> > > "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
> > > +                cd ${PP}
> > > +                cp /downloads/deb-src/"${1}"/"${2}"/* ${PP}
> > > +                dpkg-source -x "${dscfile}" "${PPS}"' \
> > > +                    my_script "${DISTRO}" "${uri}"  
> >
> > You removed the && chaining, please try a "false" in there, i guess
> > that sh might need a -e now  
> 
> Yes. Will add that in v2.
> 
> >
> > I wonder if we can just symlink debians choice to PPS, or mv, or
> > even "cp -a"  
> 
> Wondering whether it brings in any added advantage?
> 
> > And make that a new task we call apt_unpack  
> 
> I thought about this too. We could definitely have something like
> apt_unpack that makes things clear. When I first started looking
> around, I didn't expect the
> unpacking to happen in fetch.
> 
> Thanks,
> Vijai Kumar K
> 
> >
> > Henning
> >  
> > >      done
> > >
> > >      dpkg_undo_mounts  
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "isar-users" group. To unsubscribe from this group and stop
> > receiving emails from it, send an email to
> > isar-users+unsubscribe@googlegroups.com. To view this discussion on
> > the web visit
> > https://groups.google.com/d/msgid/isar-users/20210220090734.60a591c2%40md1za8fc.ad001.siemens.net.
> >  
>

-----Original Message-----
From: Henning Schild [mailto:henning.schild@siemens.com] 
Sent: 03 March 2021 18:19
To: vijai kumar <vijaikumar.kanagarajan@gmail.com>
Cc: Kanagarajan, Vijaikumar <Vijaikumar_Kanagarajan@mentor.com>; isar-users <isar-users@googlegroups.com>; ibr@radix10.net; Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
Subject: Re: [RFC PATCH 1/2] dpkg-base: Handle custom source directory in do_apt_fetch

Can we please get this one stand-alone on the fast-path? Just ran into exactly the problem and now have to work around.


Sure Henning. Will take this out and send a v2 tonight.

regards,
Henning

Am Mon, 22 Feb 2021 12:41:40 +0530
schrieb vijai kumar <vijaikumar.kanagarajan@gmail.com>:

> On Sat, Feb 20, 2021 at 1:47 PM Henning Schild 
> <henning.schild@siemens.com> wrote:
> >
> > Am Sat, 20 Feb 2021 01:27:18 +0530
> > schrieb Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>:
> >  
> > > With the current do_apt_fetch implementation, it is not possible 
> > > to use a custom source directory(${S}).
> > >
> > > apt-get source by default extracts the contents of the debian 
> > > source into folder with name <pkg>_<version>.
> > >
> > > Add provision for specifying a custom source directory.  
> >
> > I think this one is indeed worth being discussed seperately.
> >
> > Could you go into detail why a custom S is required in the first 
> > place? My guess is that we might not know PV and need to wait for 
> > that apt-unpack to finish so we can find it in the changelog.
> 
> Exactly. In some cases we might not know PV, we might need to wait for 
> apt to see what version it is fetching. Change the APT sources and or 
> preferences and you might get a completely new version of the package. 
> We cannot effectively know ${S} for manipulating the source code from 
> the recipe.
> 
> >  
> > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
> > > ---
> > >  meta/classes/dpkg-base.bbclass | 7 ++++++-
> > >  1 file changed, 6 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/meta/classes/dpkg-base.bbclass 
> > > b/meta/classes/dpkg-base.bbclass index 5c7bddc..1b94b76 100644
> > > --- a/meta/classes/dpkg-base.bbclass
> > > +++ b/meta/classes/dpkg-base.bbclass
> > > @@ -70,7 +70,12 @@ do_apt_fetch() {
> > >          sudo -E chroot --userspec=$( id -u ):$( id -g ) 
> > > ${BUILDCHROOT_DIR} \ sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2"
> > > && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only 
> > > --only-source source "$2"' my_script "${DISTRO}" "${uri}" sudo -E 
> > > chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
> > > -            sh -c 'cp /downloads/deb-src/"$1"/"$2"/* ${PP} && cd
> > > ${PP} && apt-get -y --only-source source "$2"' my_script 
> > > "${DISTRO}" "${uri}"
> > > +            sh -c ' \
> > > +                dscfile="$(apt-get -y -qq --print-uris source
> > > "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
> > > +                cd ${PP}
> > > +                cp /downloads/deb-src/"${1}"/"${2}"/* ${PP}
> > > +                dpkg-source -x "${dscfile}" "${PPS}"' \
> > > +                    my_script "${DISTRO}" "${uri}"  
> >
> > You removed the && chaining, please try a "false" in there, i guess 
> > that sh might need a -e now
> 
> Yes. Will add that in v2.
> 
> >
> > I wonder if we can just symlink debians choice to PPS, or mv, or 
> > even "cp -a"
> 
> Wondering whether it brings in any added advantage?
> 
> > And make that a new task we call apt_unpack
> 
> I thought about this too. We could definitely have something like 
> apt_unpack that makes things clear. When I first started looking 
> around, I didn't expect the unpacking to happen in fetch.
> 
> Thanks,
> Vijai Kumar K
> 
> >
> > Henning
> >  
> > >      done
> > >
> > >      dpkg_undo_mounts
> >
> > --
> > You received this message because you are subscribed to the Google 
> > Groups "isar-users" group. To unsubscribe from this group and stop 
> > receiving emails from it, send an email to
> > isar-users+unsubscribe@googlegroups.com. To view this discussion on
> > the web visit
> > https://groups.google.com/d/msgid/isar-users/20210220090734.60a591c2%40md1za8fc.ad001.siemens.net.
> >  
>

Sweet, this is what i just placed in a recipe

+## TODO hack, upstream will soon have a patch
+do_rename_to_s() {
+  debs=$( find ${WORKDIR} -type d -iname "${P}*" | head -1 )
+  rm -rf "${S}"
+  cp -a "$debs" "${S}"
+}
+addtask rename_to_s after do_apt_fetch

symlink will not work, since that is mounted into buildchroot

Henning

Am Wed, 3 Mar 2021 13:49:26 +0000
schrieb "Kanagarajan, Vijaikumar" <Vijaikumar_Kanagarajan@mentor.com>:

> -----Original Message-----
> From: Henning Schild [mailto:henning.schild@siemens.com] 
> Sent: 03 March 2021 18:19
> To: vijai kumar <vijaikumar.kanagarajan@gmail.com>
> Cc: Kanagarajan, Vijaikumar <Vijaikumar_Kanagarajan@mentor.com>;
> isar-users <isar-users@googlegroups.com>; ibr@radix10.net; Silvano
> Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com> Subject: Re:
> [RFC PATCH 1/2] dpkg-base: Handle custom source directory in
> do_apt_fetch
> 
> Can we please get this one stand-alone on the fast-path? Just ran
> into exactly the problem and now have to work around.
> 
> 
> Sure Henning. Will take this out and send a v2 tonight.
> 
> regards,
> Henning
> 
> Am Mon, 22 Feb 2021 12:41:40 +0530
> schrieb vijai kumar <vijaikumar.kanagarajan@gmail.com>:
> 
> > On Sat, Feb 20, 2021 at 1:47 PM Henning Schild 
> > <henning.schild@siemens.com> wrote:  
> > >
> > > Am Sat, 20 Feb 2021 01:27:18 +0530
> > > schrieb Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>:
> > >    
> > > > With the current do_apt_fetch implementation, it is not
> > > > possible to use a custom source directory(${S}).
> > > >
> > > > apt-get source by default extracts the contents of the debian 
> > > > source into folder with name <pkg>_<version>.
> > > >
> > > > Add provision for specifying a custom source directory.    
> > >
> > > I think this one is indeed worth being discussed seperately.
> > >
> > > Could you go into detail why a custom S is required in the first 
> > > place? My guess is that we might not know PV and need to wait for 
> > > that apt-unpack to finish so we can find it in the changelog.  
> > 
> > Exactly. In some cases we might not know PV, we might need to wait
> > for apt to see what version it is fetching. Change the APT sources
> > and or preferences and you might get a completely new version of
> > the package. We cannot effectively know ${S} for manipulating the
> > source code from the recipe.
> >   
> > >    
> > > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
> > > > ---
> > > >  meta/classes/dpkg-base.bbclass | 7 ++++++-
> > > >  1 file changed, 6 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/meta/classes/dpkg-base.bbclass 
> > > > b/meta/classes/dpkg-base.bbclass index 5c7bddc..1b94b76 100644
> > > > --- a/meta/classes/dpkg-base.bbclass
> > > > +++ b/meta/classes/dpkg-base.bbclass
> > > > @@ -70,7 +70,12 @@ do_apt_fetch() {
> > > >          sudo -E chroot --userspec=$( id -u ):$( id -g ) 
> > > > ${BUILDCHROOT_DIR} \ sh -c 'mkdir -p
> > > > /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2"
> > > > && apt-get -y --download-only --only-source source "$2"'
> > > > my_script "${DISTRO}" "${uri}" sudo -E chroot --userspec=$( id
> > > > -u ):$( id -g ) ${BUILDCHROOT_DIR} \
> > > > -            sh -c 'cp /downloads/deb-src/"$1"/"$2"/* ${PP} &&
> > > > cd ${PP} && apt-get -y --only-source source "$2"' my_script 
> > > > "${DISTRO}" "${uri}"
> > > > +            sh -c ' \
> > > > +                dscfile="$(apt-get -y -qq --print-uris source
> > > > "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
> > > > +                cd ${PP}
> > > > +                cp /downloads/deb-src/"${1}"/"${2}"/* ${PP}
> > > > +                dpkg-source -x "${dscfile}" "${PPS}"' \
> > > > +                    my_script "${DISTRO}" "${uri}"    
> > >
> > > You removed the && chaining, please try a "false" in there, i
> > > guess that sh might need a -e now  
> > 
> > Yes. Will add that in v2.
> >   
> > >
> > > I wonder if we can just symlink debians choice to PPS, or mv, or 
> > > even "cp -a"  
> > 
> > Wondering whether it brings in any added advantage?
> >   
> > > And make that a new task we call apt_unpack  
> > 
> > I thought about this too. We could definitely have something like 
> > apt_unpack that makes things clear. When I first started looking 
> > around, I didn't expect the unpacking to happen in fetch.
> > 
> > Thanks,
> > Vijai Kumar K
> >   
> > >
> > > Henning
> > >    
> > > >      done
> > > >
> > > >      dpkg_undo_mounts  
> > >
> > > --
> > > You received this message because you are subscribed to the
> > > Google Groups "isar-users" group. To unsubscribe from this group
> > > and stop receiving emails from it, send an email to
> > > isar-users+unsubscribe@googlegroups.com. To view this discussion
> > > on the web visit
> > > https://groups.google.com/d/msgid/isar-users/20210220090734.60a591c2%40md1za8fc.ad001.siemens.net.
> > >    
> >   
>