| Message ID | 20200902185624.15044-4-Vijaikumar_Kanagarajan@mentor.com |
|---|---|
| State | Superseded, archived |
| Headers | show |
| Series | WIC update | expand |
On Thu, 3 Sep 2020 00:26:17 +0530 Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> wrote: > The latest WIC doesnot use fakeroot if > pseudo_dir(IMAGE_ROOTFS/../pseudo) is non-existent. This results in > wic_fakeroot not being used. Create a dummy pseudo_dir to satisfy wic. > > WIC hardcodes the python(/usr/bin/python3-native/python3) to be used > while running bmaptool. The above path is OE specific and hence breaks > build in ISAR. Create a symbolic link in the above location to satisfy > wic. > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > --- > meta/classes/wic-img.bbclass | 14 ++++++++++++-- > 1 file changed, 12 insertions(+), 2 deletions(-) > > diff --git a/meta/classes/wic-img.bbclass > b/meta/classes/wic-img.bbclass index afab394..a2c9627 100644 > --- a/meta/classes/wic-img.bbclass > +++ b/meta/classes/wic-img.bbclass > @@ -143,6 +143,7 @@ EOSUDO > export FAKEROOTCMD=${FAKEROOTCMD} > export BUILDDIR=${BUILDDIR} > export MTOOLS_SKIP_CHECK=1 > + mkdir -p ${IMAGE_ROOTFS}/../pseudo > > # create the temp dir in the buildchroot to ensure uniqueness > WICTMP=$(cd ${BUILDCHROOT_DIR}; mktemp -d -p tmp) > @@ -157,22 +158,31 @@ EOSUDO > IMAGE_FULLNAME="$6" > IMAGE_BASENAME="$7" > shift 7 > - > + # The python path is hard-coded as > /usr/bin/python3-native/python3 in wic. Handle that. > + mkdir -p /usr/bin/python3-native/ > + if [ $(head -1 $(which bmaptool) | grep python3) ];then > + ln -s /usr/bin/python3 /usr/bin/python3-native/python3 > + else > + ln -s /usr/bin/python2 /usr/bin/python3-native/python3 > + fi Can this be moved to wic_fakeroot? That script was designed to hold all the weird hacks ... I guess you need to "ln -sf" to make that whole thing executable multiple times. Can you explain why we need the case where python2 is linked as python3? > export PATH="$ISARROOT/bitbake/bin:$PATH" > "$ISARROOT"/scripts/wic create "$WKS_FULL_PATH" \ > --vars "$STAGING_DIR/$MACHINE/imgdata/" \ > -o "/$WICTMP/${IMAGE_FULLNAME}.wic/" \ > --bmap \ > - -e "$IMAGE_BASENAME" $@' \ > + -e "$IMAGE_BASENAME" $@ > + rm -rf /usr/bin/python3-native' \ This looks like the rm is in the middle of wic, should probably come after. > my_script "${ISARROOT}" "${WKS_FULL_PATH}" > "${STAGING_DIR}" \ "${MACHINE}" "${WICTMP}" "${IMAGE_FULLNAME}" > "${IMAGE_BASENAME}" \ ${WIC_CREATE_EXTRA_ARGS} > + > sudo chown -R $(stat -c "%U" ${ISARROOT}) ${ISARROOT}/meta > ${ISARROOT}/meta-isar ${ISARROOT}/scripts || true WIC_DIRECT=$(ls -t > -1 ${BUILDCHROOT_DIR}/$WICTMP/${IMAGE_FULLNAME}.wic/*.direct | head > -1) sudo chown -R $(id -u):$(id -g) ${BUILDCHROOT_DIR}/${WICTMP} mv > -f ${WIC_DIRECT} ${WIC_IMAGE_FILE} mv -f ${WIC_DIRECT}.bmap > ${WIC_IMAGE_FILE}.bmap rm -rf ${BUILDCHROOT_DIR}/${WICTMP} > + rm -rf ${IMAGE_ROOTFS}/../pseudo > } > > do_wic_image[file-checksums] += "${WKS_FILE_CHECKSUM}"
On Saturday, September 5, 2020 at 2:00:59 PM UTC+5:30 Henning Schild wrote: > On Thu, 3 Sep 2020 00:26:17 +0530 > Vijai Kumar K <Vijaikumar_...@mentor.com> wrote: > > > The latest WIC doesnot use fakeroot if > > pseudo_dir(IMAGE_ROOTFS/../pseudo) is non-existent. This results in > > wic_fakeroot not being used. Create a dummy pseudo_dir to satisfy wic. > > > > WIC hardcodes the python(/usr/bin/python3-native/python3) to be used > > while running bmaptool. The above path is OE specific and hence breaks > > build in ISAR. Create a symbolic link in the above location to satisfy > > wic. > > > > Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com> > > --- > > meta/classes/wic-img.bbclass | 14 ++++++++++++-- > > 1 file changed, 12 insertions(+), 2 deletions(-) > > > > diff --git a/meta/classes/wic-img.bbclass > > b/meta/classes/wic-img.bbclass index afab394..a2c9627 100644 > > --- a/meta/classes/wic-img.bbclass > > +++ b/meta/classes/wic-img.bbclass > > @@ -143,6 +143,7 @@ EOSUDO > > export FAKEROOTCMD=${FAKEROOTCMD} > > export BUILDDIR=${BUILDDIR} > > export MTOOLS_SKIP_CHECK=1 > > + mkdir -p ${IMAGE_ROOTFS}/../pseudo > > > > # create the temp dir in the buildchroot to ensure uniqueness > > WICTMP=$(cd ${BUILDCHROOT_DIR}; mktemp -d -p tmp) > > @@ -157,22 +158,31 @@ EOSUDO > > IMAGE_FULLNAME="$6" > > IMAGE_BASENAME="$7" > > shift 7 > > - > > + # The python path is hard-coded as > > /usr/bin/python3-native/python3 in wic. Handle that. > > + mkdir -p /usr/bin/python3-native/ > > + if [ $(head -1 $(which bmaptool) | grep python3) ];then > > + ln -s /usr/bin/python3 /usr/bin/python3-native/python3 > > + else > > + ln -s /usr/bin/python2 /usr/bin/python3-native/python3 > > + fi > > Can this be moved to wic_fakeroot? That script was designed to hold all > the weird hacks ... > Unfortunately no. This invocation doesnot use fakeroot at all. Please see [1] & [2]. pseudo argument not used for exec_native_cmd which will in-turn not use fakeroot. [1] https://github.com/openembedded/openembedded-core/blob/404292b570a78895a1c7900eeb319e36e31dec20/scripts/lib/wic/plugins/imager/direct.py#L210 [2] https://github.com/openembedded/openembedded-core/blob/404292b570a78895a1c7900eeb319e36e31dec20/scripts/lib/wic/misc.py#L138 > I guess you need to "ln -sf" to make that whole thing executable > multiple times. > Sorry I did not get this completely. -f could be good to have to overwrite. We are remove this after wic runs so i didnot use -f here. > > Can you explain why we need the case where python2 is linked as python3? > I thought I captured it in commit message but looks like I have missed it. Because of stretch. bmaptools in stretch is python2 based and in buster and subsequent releases it is python3 based. OE-core in [1] has hardcoded to python3 since they have moved bmaptools to the latest one which uses python3. We would still need to workaround [1] for stretch too. Will capture this missed detail in commit message in V3. [1] https://github.com/openembedded/openembedded-core/blob/404292b570a78895a1c7900eeb319e36e31dec20/scripts/lib/wic/plugins/imager/direct.py#L210 > > export PATH="$ISARROOT/bitbake/bin:$PATH" > > "$ISARROOT"/scripts/wic create "$WKS_FULL_PATH" \ > > --vars "$STAGING_DIR/$MACHINE/imgdata/" \ > > -o "/$WICTMP/${IMAGE_FULLNAME}.wic/" \ > > --bmap \ > > - -e "$IMAGE_BASENAME" $@' \ > > + -e "$IMAGE_BASENAME" $@ > > + rm -rf /usr/bin/python3-native' \ > > This looks like the rm is in the middle of wic, should probably come > after. > It is at the end of my_script. Should be okay but bit difficult to read though. > > my_script "${ISARROOT}" "${WKS_FULL_PATH}" > > "${STAGING_DIR}" \ "${MACHINE}" "${WICTMP}" "${IMAGE_FULLNAME}" > > "${IMAGE_BASENAME}" \ ${WIC_CREATE_EXTRA_ARGS} > > + > > sudo chown -R $(stat -c "%U" ${ISARROOT}) ${ISARROOT}/meta > > ${ISARROOT}/meta-isar ${ISARROOT}/scripts || true WIC_DIRECT=$(ls -t > > -1 ${BUILDCHROOT_DIR}/$WICTMP/${IMAGE_FULLNAME}.wic/*.direct | head > > -1) sudo chown -R $(id -u):$(id -g) ${BUILDCHROOT_DIR}/${WICTMP} mv > > -f ${WIC_DIRECT} ${WIC_IMAGE_FILE} mv -f ${WIC_DIRECT}.bmap > > ${WIC_IMAGE_FILE}.bmap rm -rf ${BUILDCHROOT_DIR}/${WICTMP} > > + rm -rf ${IMAGE_ROOTFS}/../pseudo > > } > > > > do_wic_image[file-checksums] += "${WKS_FILE_CHECKSUM}" > >
On Sat, 5 Sep 2020 09:24:08 -0700 (PDT) "vijaikumar....@gmail.com" <vijaikumar.kanagarajan@gmail.com> wrote: > On Saturday, September 5, 2020 at 2:00:59 PM UTC+5:30 Henning Schild > wrote: > > > On Thu, 3 Sep 2020 00:26:17 +0530 > > Vijai Kumar K <Vijaikumar_...@mentor.com> wrote: > > > > > The latest WIC doesnot use fakeroot if > > > pseudo_dir(IMAGE_ROOTFS/../pseudo) is non-existent. This results > > > in wic_fakeroot not being used. Create a dummy pseudo_dir to > > > satisfy wic. > > > > > > WIC hardcodes the python(/usr/bin/python3-native/python3) to be > > > used while running bmaptool. The above path is OE specific and > > > hence breaks build in ISAR. Create a symbolic link in the above > > > location to satisfy wic. > > > > > > Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com> > > > --- > > > meta/classes/wic-img.bbclass | 14 ++++++++++++-- > > > 1 file changed, 12 insertions(+), 2 deletions(-) > > > > > > diff --git a/meta/classes/wic-img.bbclass > > > b/meta/classes/wic-img.bbclass index afab394..a2c9627 100644 > > > --- a/meta/classes/wic-img.bbclass > > > +++ b/meta/classes/wic-img.bbclass > > > @@ -143,6 +143,7 @@ EOSUDO > > > export FAKEROOTCMD=${FAKEROOTCMD} > > > export BUILDDIR=${BUILDDIR} > > > export MTOOLS_SKIP_CHECK=1 > > > + mkdir -p ${IMAGE_ROOTFS}/../pseudo > > > > > > # create the temp dir in the buildchroot to ensure uniqueness > > > WICTMP=$(cd ${BUILDCHROOT_DIR}; mktemp -d -p tmp) > > > @@ -157,22 +158,31 @@ EOSUDO > > > IMAGE_FULLNAME="$6" > > > IMAGE_BASENAME="$7" > > > shift 7 > > > - > > > + # The python path is hard-coded as > > > /usr/bin/python3-native/python3 in wic. Handle that. > > > + mkdir -p /usr/bin/python3-native/ > > > + if [ $(head -1 $(which bmaptool) | grep python3) ];then > > > + ln -s /usr/bin/python3 /usr/bin/python3-native/python3 > > > + else > > > + ln -s /usr/bin/python2 /usr/bin/python3-native/python3 > > > + fi > > > > Can this be moved to wic_fakeroot? That script was designed to hold > > all the weird hacks ... > > > > Unfortunately no. This invocation doesnot use fakeroot at all. Please > see [1] & [2]. pseudo argument not used for exec_native_cmd > which will in-turn not use fakeroot. > > [1] > https://github.com/openembedded/openembedded-core/blob/404292b570a78895a1c7900eeb319e36e31dec20/scripts/lib/wic/plugins/imager/direct.py#L210 > [2] > https://github.com/openembedded/openembedded-core/blob/404292b570a78895a1c7900eeb319e36e31dec20/scripts/lib/wic/misc.py#L138 > > > > > I guess you need to "ln -sf" to make that whole thing executable > > multiple times. > > > > Sorry I did not get this completely. -f could be good to have to > overwrite. We are remove this > after wic runs so i didnot use -f here. if wic fails for whatever reason ... i.e. missing or broken wks file ... a partial rebuild after the bugfix would run "ln -s" again while "rm" has never been done. Did not try, but it is in general a good idea to write any task so that it can be partially re-executed. you could also look into do_wic_image[cleandirs] In fact those symlinks could probably be placed there using a package-recipe and installed as a wic imager_depends. Just an idea, might be overkill. Henning > > > > > Can you explain why we need the case where python2 is linked as > > python3? > > I thought I captured it in commit message but looks like I have > missed it. Because of stretch. bmaptools in stretch is python2 based > and in buster and subsequent releases it is python3 based. > OE-core in [1] has hardcoded to python3 since they have moved > bmaptools to the latest one > which uses python3. We would still need to workaround [1] for stretch > too. > > Will capture this missed detail in commit message in V3. > > [1] > https://github.com/openembedded/openembedded-core/blob/404292b570a78895a1c7900eeb319e36e31dec20/scripts/lib/wic/plugins/imager/direct.py#L210 > > > > > export PATH="$ISARROOT/bitbake/bin:$PATH" > > > "$ISARROOT"/scripts/wic create "$WKS_FULL_PATH" \ > > > --vars "$STAGING_DIR/$MACHINE/imgdata/" \ > > > -o "/$WICTMP/${IMAGE_FULLNAME}.wic/" \ > > > --bmap \ > > > - -e "$IMAGE_BASENAME" $@' \ > > > + -e "$IMAGE_BASENAME" $@ > > > + rm -rf /usr/bin/python3-native' \ > > > > This looks like the rm is in the middle of wic, should probably > > come after. > > > > It is at the end of my_script. Should be okay but bit difficult to > read though. > > > > > my_script "${ISARROOT}" "${WKS_FULL_PATH}" > > > "${STAGING_DIR}" \ "${MACHINE}" "${WICTMP}" "${IMAGE_FULLNAME}" > > > "${IMAGE_BASENAME}" \ ${WIC_CREATE_EXTRA_ARGS} > > > + > > > sudo chown -R $(stat -c "%U" ${ISARROOT}) ${ISARROOT}/meta > > > ${ISARROOT}/meta-isar ${ISARROOT}/scripts || true WIC_DIRECT=$(ls > > > -t -1 ${BUILDCHROOT_DIR}/$WICTMP/${IMAGE_FULLNAME}.wic/*.direct | > > > head -1) sudo chown -R $(id -u):$(id -g) > > > ${BUILDCHROOT_DIR}/${WICTMP} mv -f ${WIC_DIRECT} > > > ${WIC_IMAGE_FILE} mv -f ${WIC_DIRECT}.bmap ${WIC_IMAGE_FILE}.bmap > > > rm -rf ${BUILDCHROOT_DIR}/${WICTMP} > > > + rm -rf ${IMAGE_ROOTFS}/../pseudo > > > } > > > > > > do_wic_image[file-checksums] += "${WKS_FILE_CHECKSUM}" > > > > >
diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass index afab394..a2c9627 100644 --- a/meta/classes/wic-img.bbclass +++ b/meta/classes/wic-img.bbclass @@ -143,6 +143,7 @@ EOSUDO export FAKEROOTCMD=${FAKEROOTCMD} export BUILDDIR=${BUILDDIR} export MTOOLS_SKIP_CHECK=1 + mkdir -p ${IMAGE_ROOTFS}/../pseudo # create the temp dir in the buildchroot to ensure uniqueness WICTMP=$(cd ${BUILDCHROOT_DIR}; mktemp -d -p tmp) @@ -157,22 +158,31 @@ EOSUDO IMAGE_FULLNAME="$6" IMAGE_BASENAME="$7" shift 7 - + # The python path is hard-coded as /usr/bin/python3-native/python3 in wic. Handle that. + mkdir -p /usr/bin/python3-native/ + if [ $(head -1 $(which bmaptool) | grep python3) ];then + ln -s /usr/bin/python3 /usr/bin/python3-native/python3 + else + ln -s /usr/bin/python2 /usr/bin/python3-native/python3 + fi export PATH="$ISARROOT/bitbake/bin:$PATH" "$ISARROOT"/scripts/wic create "$WKS_FULL_PATH" \ --vars "$STAGING_DIR/$MACHINE/imgdata/" \ -o "/$WICTMP/${IMAGE_FULLNAME}.wic/" \ --bmap \ - -e "$IMAGE_BASENAME" $@' \ + -e "$IMAGE_BASENAME" $@ + rm -rf /usr/bin/python3-native' \ my_script "${ISARROOT}" "${WKS_FULL_PATH}" "${STAGING_DIR}" \ "${MACHINE}" "${WICTMP}" "${IMAGE_FULLNAME}" "${IMAGE_BASENAME}" \ ${WIC_CREATE_EXTRA_ARGS} + sudo chown -R $(stat -c "%U" ${ISARROOT}) ${ISARROOT}/meta ${ISARROOT}/meta-isar ${ISARROOT}/scripts || true WIC_DIRECT=$(ls -t -1 ${BUILDCHROOT_DIR}/$WICTMP/${IMAGE_FULLNAME}.wic/*.direct | head -1) sudo chown -R $(id -u):$(id -g) ${BUILDCHROOT_DIR}/${WICTMP} mv -f ${WIC_DIRECT} ${WIC_IMAGE_FILE} mv -f ${WIC_DIRECT}.bmap ${WIC_IMAGE_FILE}.bmap rm -rf ${BUILDCHROOT_DIR}/${WICTMP} + rm -rf ${IMAGE_ROOTFS}/../pseudo } do_wic_image[file-checksums] += "${WKS_FILE_CHECKSUM}"
The latest WIC doesnot use fakeroot if pseudo_dir(IMAGE_ROOTFS/../pseudo) is non-existent. This results in wic_fakeroot not being used. Create a dummy pseudo_dir to satisfy wic. WIC hardcodes the python(/usr/bin/python3-native/python3) to be used while running bmaptool. The above path is OE specific and hence breaks build in ISAR. Create a symbolic link in the above location to satisfy wic. Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> --- meta/classes/wic-img.bbclass | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)