| Message ID | 20220127201845.19319-2-henning.schild@siemens.com |
|---|---|
| State | Superseded, archived |
| Headers | show |
| Series | sstate bug fix | expand |
Schild, Henning (T CED SES-DE), 27. Januar 2022 21:19: > Subject: [PATCH 1/1] sttate: change rootfs task to cache Typo in commit msg -----^ > This patch moves sstate caching from the task "do_rootfs" to > "do_rootfs_install". Which is earlier in the task chain. > > There are two "bugs" that we fix by doing so, while hopefully not > giving up on too much caching speed. > - rootfs_postprocess: > - image_postprocess_mark: potentially outdated BUILD_ID and friends > - rootfs_generate_manifest: manifest file gone missing in DEPLOY_DIR > > Since rootfs_postprocess is kind of an open field it might also fix > issues in ROOTFS_POSTPROCESS_COMMAND functions in layers. Any function > deriving stuff (like a manifest) or changing stuff (like the mark) could > potentially benefit from "not caching". > > Signed-off-by: Henning Schild <henning.schild@siemens.com> > --- > meta/classes/rootfs.bbclass | 25 ++++++++++++------------- > 1 file changed, 12 insertions(+), 13 deletions(-) > > diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass > index 1fc487896f4d..e368244ce1ca 100644 > --- a/meta/classes/rootfs.bbclass > +++ b/meta/classes/rootfs.bbclass > @@ -157,6 +157,7 @@ rootfs_install_pkgs_install() { > do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}" > do_rootfs_install[vardeps] += "${ROOTFS_CONFIGURE_COMMAND} > ${ROOTFS_INSTALL_COMMAND}" > do_rootfs_install[depends] = "isar-bootstrap-${@'target' if > d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build" > +do_rootfs_install[depends] += " base-apt:do_cache isar-apt:do_cache_config" > do_rootfs_install[recrdeptask] = "do_deploy_deb" > python do_rootfs_install() { > configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) or > "").split() > @@ -270,29 +271,27 @@ python do_rootfs() { > } > addtask rootfs before do_build > > -do_rootfs[depends] = "base-apt:do_cache isar-apt:do_cache_config" You removed that line. I don't remember all of the details, but there were some cases where this explicit dependency was needed when restoring from cache to an empty tmp/ dir. Might need some testing... Adriaan > - > -SSTATETASKS += "do_rootfs" > +SSTATETASKS += "do_rootfs_install" > ROOTFS_SSTATE = "${WORKDIR}/rootfs-sstate" > -do_rootfs[dirs] += "${ROOTFS_SSTATE} ${WORKDIR}/mnt/rootfs" > -do_rootfs[cleandirs] += "${ROOTFS_SSTATE}" > -do_rootfs[sstate-plaindirs] = "${ROOTFS_SSTATE}" > -do_rootfs[sstate-interceptfuncs] = "rootfs_sstate_prepare" > +do_rootfs_install[dirs] += "${ROOTFS_SSTATE} ${WORKDIR}/mnt/rootfs" > +do_rootfs_install[cleandirs] += "${ROOTFS_SSTATE}" > +do_rootfs_install[sstate-plaindirs] = "${ROOTFS_SSTATE}" > +do_rootfs_install[sstate-interceptfuncs] = "rootfs_install_sstate_prepare" > > # the buildchroot is owned by root, so we need some sudoing to pack and > unpack > -rootfs_sstate_prepare() { > +rootfs_install_sstate_prepare() { > sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro > sudo tar -C ${WORKDIR}/mnt -cpf ${ROOTFS_SSTATE}/rootfs.tar --one-file- > system rootfs > sudo umount ${WORKDIR}/mnt/rootfs > } > -do_rootfs_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock" > +do_rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock" > > -rootfs_sstate_finalize() { > +rootfs_install_sstate_finalize() { > sudo tar -C ${WORKDIR} -xpf ${ROOTFS_SSTATE}/rootfs.tar > } > > -python do_rootfs_setscene() { > +python do_rootfs_install_setscene() { > sstate_setscene(d) > - bb.build.exec_func('rootfs_sstate_finalize', d) > + bb.build.exec_func('rootfs_install_sstate_finalize', d) > } > -addtask do_rootfs_setscene > +addtask do_rootfs_install_setscene > -- > 2.34.1
Am Fri, 28 Jan 2022 07:57:31 +0100 schrieb "Schmidt, Adriaan (T CED SES-DE)" <adriaan.schmidt@siemens.com>: > Schild, Henning (T CED SES-DE), 27. Januar 2022 21:19: > > > Subject: [PATCH 1/1] sttate: change rootfs task to cache > > Typo in commit msg -----^ Thanks! > > This patch moves sstate caching from the task "do_rootfs" to > > "do_rootfs_install". Which is earlier in the task chain. > > > > There are two "bugs" that we fix by doing so, while hopefully not > > giving up on too much caching speed. > > - rootfs_postprocess: > > - image_postprocess_mark: potentially outdated BUILD_ID and > > friends > > - rootfs_generate_manifest: manifest file gone missing in > > DEPLOY_DIR > > > > Since rootfs_postprocess is kind of an open field it might also fix > > issues in ROOTFS_POSTPROCESS_COMMAND functions in layers. Any > > function deriving stuff (like a manifest) or changing stuff (like > > the mark) could potentially benefit from "not caching". > > > > Signed-off-by: Henning Schild <henning.schild@siemens.com> > > --- > > meta/classes/rootfs.bbclass | 25 ++++++++++++------------- > > 1 file changed, 12 insertions(+), 13 deletions(-) > > > > diff --git a/meta/classes/rootfs.bbclass > > b/meta/classes/rootfs.bbclass index 1fc487896f4d..e368244ce1ca > > 100644 --- a/meta/classes/rootfs.bbclass > > +++ b/meta/classes/rootfs.bbclass > > @@ -157,6 +157,7 @@ rootfs_install_pkgs_install() { > > do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}" > > do_rootfs_install[vardeps] += "${ROOTFS_CONFIGURE_COMMAND} > > ${ROOTFS_INSTALL_COMMAND}" > > do_rootfs_install[depends] = "isar-bootstrap-${@'target' if > > d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else > > 'host'}:do_build" +do_rootfs_install[depends] += " > > base-apt:do_cache isar-apt:do_cache_config" > > do_rootfs_install[recrdeptask] = "do_deploy_deb" python > > do_rootfs_install() { configure_cmds = > > (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) or "").split() > > @@ -270,29 +271,27 @@ python do_rootfs() { > > } > > addtask rootfs before do_build > > > > -do_rootfs[depends] = "base-apt:do_cache isar-apt:do_cache_config" > > You removed that line. > I don't remember all of the details, but there were some cases where > this explicit dependency was needed when restoring from cache to > an empty tmp/ dir. Might need some testing... It got moved up under a preexisting do_rootfs_install[depends]. In fact we might want to move all do_rootfs_install[] assignments there for readability. But hey, i think i will move it back down so we have the actual function []s and the sstate specific []s. Henning > Adriaan > > > - > > -SSTATETASKS += "do_rootfs" > > +SSTATETASKS += "do_rootfs_install" > > ROOTFS_SSTATE = "${WORKDIR}/rootfs-sstate" > > -do_rootfs[dirs] += "${ROOTFS_SSTATE} ${WORKDIR}/mnt/rootfs" > > -do_rootfs[cleandirs] += "${ROOTFS_SSTATE}" > > -do_rootfs[sstate-plaindirs] = "${ROOTFS_SSTATE}" > > -do_rootfs[sstate-interceptfuncs] = "rootfs_sstate_prepare" > > +do_rootfs_install[dirs] += "${ROOTFS_SSTATE} ${WORKDIR}/mnt/rootfs" > > +do_rootfs_install[cleandirs] += "${ROOTFS_SSTATE}" > > +do_rootfs_install[sstate-plaindirs] = "${ROOTFS_SSTATE}" > > +do_rootfs_install[sstate-interceptfuncs] = > > "rootfs_install_sstate_prepare" > > > > # the buildchroot is owned by root, so we need some sudoing to > > pack and unpack > > -rootfs_sstate_prepare() { > > +rootfs_install_sstate_prepare() { > > sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro > > sudo tar -C ${WORKDIR}/mnt -cpf ${ROOTFS_SSTATE}/rootfs.tar > > --one-file- system rootfs > > sudo umount ${WORKDIR}/mnt/rootfs > > } > > -do_rootfs_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock" > > +do_rootfs_install_sstate_prepare[lockfiles] = > > "${REPO_ISAR_DIR}/isar.lock" > > > > -rootfs_sstate_finalize() { > > +rootfs_install_sstate_finalize() { > > sudo tar -C ${WORKDIR} -xpf ${ROOTFS_SSTATE}/rootfs.tar > > } > > > > -python do_rootfs_setscene() { > > +python do_rootfs_install_setscene() { > > sstate_setscene(d) > > - bb.build.exec_func('rootfs_sstate_finalize', d) > > + bb.build.exec_func('rootfs_install_sstate_finalize', d) > > } > > -addtask do_rootfs_setscene > > +addtask do_rootfs_install_setscene > > -- > > 2.34.1 >
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 1fc487896f4d..e368244ce1ca 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -157,6 +157,7 @@ rootfs_install_pkgs_install() { do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}" do_rootfs_install[vardeps] += "${ROOTFS_CONFIGURE_COMMAND} ${ROOTFS_INSTALL_COMMAND}" do_rootfs_install[depends] = "isar-bootstrap-${@'target' if d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build" +do_rootfs_install[depends] += " base-apt:do_cache isar-apt:do_cache_config" do_rootfs_install[recrdeptask] = "do_deploy_deb" python do_rootfs_install() { configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) or "").split() @@ -270,29 +271,27 @@ python do_rootfs() { } addtask rootfs before do_build -do_rootfs[depends] = "base-apt:do_cache isar-apt:do_cache_config" - -SSTATETASKS += "do_rootfs" +SSTATETASKS += "do_rootfs_install" ROOTFS_SSTATE = "${WORKDIR}/rootfs-sstate" -do_rootfs[dirs] += "${ROOTFS_SSTATE} ${WORKDIR}/mnt/rootfs" -do_rootfs[cleandirs] += "${ROOTFS_SSTATE}" -do_rootfs[sstate-plaindirs] = "${ROOTFS_SSTATE}" -do_rootfs[sstate-interceptfuncs] = "rootfs_sstate_prepare" +do_rootfs_install[dirs] += "${ROOTFS_SSTATE} ${WORKDIR}/mnt/rootfs" +do_rootfs_install[cleandirs] += "${ROOTFS_SSTATE}" +do_rootfs_install[sstate-plaindirs] = "${ROOTFS_SSTATE}" +do_rootfs_install[sstate-interceptfuncs] = "rootfs_install_sstate_prepare" # the buildchroot is owned by root, so we need some sudoing to pack and unpack -rootfs_sstate_prepare() { +rootfs_install_sstate_prepare() { sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro sudo tar -C ${WORKDIR}/mnt -cpf ${ROOTFS_SSTATE}/rootfs.tar --one-file-system rootfs sudo umount ${WORKDIR}/mnt/rootfs } -do_rootfs_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock" +do_rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock" -rootfs_sstate_finalize() { +rootfs_install_sstate_finalize() { sudo tar -C ${WORKDIR} -xpf ${ROOTFS_SSTATE}/rootfs.tar } -python do_rootfs_setscene() { +python do_rootfs_install_setscene() { sstate_setscene(d) - bb.build.exec_func('rootfs_sstate_finalize', d) + bb.build.exec_func('rootfs_install_sstate_finalize', d) } -addtask do_rootfs_setscene +addtask do_rootfs_install_setscene
This patch moves sstate caching from the task "do_rootfs" to "do_rootfs_install". Which is earlier in the task chain. There are two "bugs" that we fix by doing so, while hopefully not giving up on too much caching speed. - rootfs_postprocess: - image_postprocess_mark: potentially outdated BUILD_ID and friends - rootfs_generate_manifest: manifest file gone missing in DEPLOY_DIR Since rootfs_postprocess is kind of an open field it might also fix issues in ROOTFS_POSTPROCESS_COMMAND functions in layers. Any function deriving stuff (like a manifest) or changing stuff (like the mark) could potentially benefit from "not caching". Signed-off-by: Henning Schild <henning.schild@siemens.com> --- meta/classes/rootfs.bbclass | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-)