mbox series

[PATCHv3,0/3] sstate bug fix

Message ID 20220204164124.10396-1-henning.schild@siemens.com
Headers show
Series sstate bug fix | expand

Message

Henning Schild Feb. 4, 2022, 6:41 a.m. UTC
changes since v2:
 - add p3

changes since v1:
 - add p2
 - fix typo in p1
 - move code around in p1, later touched again in p2

This is just a single patch but i think it fixes bugs and should be
considered before a release ... i usually do not do cover letters for
single patches. So here i still do it to point out that it might be
important.

Henning Schild (3):
  sstate: change rootfs task to cache
  sstate: fix task order and deps
  buildchroot: make function buildchroot_install_files idempotent

 meta/classes/rootfs.bbclass                   | 26 +++++++++----------
 .../buildchroot/buildchroot.inc               |  2 +-
 2 files changed, 14 insertions(+), 14 deletions(-)

Comments

Schmidt, Adriaan Feb. 15, 2022, 12:08 a.m. UTC | #1
The root problem here is that the sstate code does not "see" when/which files are put into DEPLOY_DIR_IMAGE as a side-effect of tasks.

Unfortunately, we just found out that this problem is not limited to rootfs recipes. There are examples of dpkg recipes [1] placing artifacts into the deploy dir, which also breaks with sstate caching.

One option I can see is:
  * "forbid" (by convention/documentation) recipes from manually placing files in DEPLOY_DIR_IMAGE
  * instead, recipes need to set a variable (e.g., FILES_TO_DEPLOY)
  * these files will be deployed by isar classes
  * sstate will cache and restore these files

Or we go the OE way:
  * no-one writes directly to DEPLOY_DIR_IMAGE, but instead to IMGDEPLOYDIR, which is below WORKDIR
  * isar classes transfer the deployed artifacts to DEPLOY_DIR_IMAGE (in OE this transfer is actually done implicitly by sstate code)

Q:
  * is DEPLOY_DIR_IMAGE the only location with that problem? Or should we assume that recipes have side-effects writing to arbitrary locations?

Adriaan

[1] https://gitlab.com/cip-project/cip-core/isar-cip-core/-/blob/master/recipes-bsp/efibootguard/efibootguard_0.9-git%2Bisar.bb#L42

henning.schild@siemens.com, 4. Februar 2022 17:41:
> 
> changes since v2:
>  - add p3
> 
> changes since v1:
>  - add p2
>  - fix typo in p1
>  - move code around in p1, later touched again in p2
> 
> This is just a single patch but i think it fixes bugs and should be
> considered before a release ... i usually do not do cover letters for
> single patches. So here i still do it to point out that it might be
> important.
> 
> Henning Schild (3):
>   sstate: change rootfs task to cache
>   sstate: fix task order and deps
>   buildchroot: make function buildchroot_install_files idempotent
> 
>  meta/classes/rootfs.bbclass                   | 26 +++++++++----------
>  .../buildchroot/buildchroot.inc               |  2 +-
>  2 files changed, 14 insertions(+), 14 deletions(-)
> 
> --
> 2.34.1
> 
> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-users/20220204164124.10396-1-
> henning.schild%40siemens.com.
Anton Mikanovich Feb. 16, 2022, 4:08 a.m. UTC | #2
15.02.2022 13:08, Schmidt, Adriaan wrote:
> The root problem here is that the sstate code does not "see" when/which files are put into DEPLOY_DIR_IMAGE as a side-effect of tasks.
>
> Unfortunately, we just found out that this problem is not limited to rootfs recipes. There are examples of dpkg recipes [1] placing artifacts into the deploy dir, which also breaks with sstate caching.
>
> One option I can see is:
>    * "forbid" (by convention/documentation) recipes from manually placing files in DEPLOY_DIR_IMAGE
>    * instead, recipes need to set a variable (e.g., FILES_TO_DEPLOY)
>    * these files will be deployed by isar classes
>    * sstate will cache and restore these files
>
> Or we go the OE way:
>    * no-one writes directly to DEPLOY_DIR_IMAGE, but instead to IMGDEPLOYDIR, which is below WORKDIR
>    * isar classes transfer the deployed artifacts to DEPLOY_DIR_IMAGE (in OE this transfer is actually done implicitly by sstate code)
>
> Q:
>    * is DEPLOY_DIR_IMAGE the only location with that problem? Or should we assume that recipes have side-effects writing to arbitrary locations?
>
> Adriaan
>
> [1] https://gitlab.com/cip-project/cip-core/isar-cip-core/-/blob/master/recipes-bsp/efibootguard/efibootguard_0.9-git%2Bisar.bb#L42

The way binaries installed to DEPLOY_DIR_IMAGE definitely need to be 
changed, but does anything speak against merging this patchset?
Anton Mikanovich Feb. 20, 2022, 11:23 p.m. UTC | #3
4.02.22 19:41, henning.schild@siemens.com wrote:
> changes since v2:
>   - add p3
>
> changes since v1:
>   - add p2
>   - fix typo in p1
>   - move code around in p1, later touched again in p2
>
> This is just a single patch but i think it fixes bugs and should be
> considered before a release ... i usually do not do cover letters for
> single patches. So here i still do it to point out that it might be
> important.
>
> Henning Schild (3):
>    sstate: change rootfs task to cache
>    sstate: fix task order and deps
>    buildchroot: make function buildchroot_install_files idempotent
>
>   meta/classes/rootfs.bbclass                   | 26 +++++++++----------
>   .../buildchroot/buildchroot.inc               |  2 +-
>   2 files changed, 14 insertions(+), 14 deletions(-)
>
Applied to next, thanks.