4 ms·

Make compares timestamp of object-file with source-file amiright? This is nice and cheap and stateless. Do modern build-systems maintain a separate tree of hash

by rusk 8y ago

Make compares timestamp of object-file with source-file amiright? This is nice and cheap and stateless. Do modern build-systems maintain a separate tree of hashes? I haven't seen this in my day-to-day but I don't tend to work on the bleeding edge ... I'd have thought the cost of calculating hashes for trivial source-files would be maybe only an order of magnitude cheaper than compiling one? Over a large source-tree I'd expect these costs to mount - I think I'd prefer to take my chance with timestamps!

yetihehe 8y ago
Typical optimisation in such cases is to check file hash only when file timestamp or size changes. If after timestamp change hash is the same, it means file was not modified.
- rusk 8y ago
  Okay that's half the problem. Where do I keep my database of hashes though? W̶h̶i̶c̶h̶ ̶b̶u̶i̶l̶d̶ ̶s̶y̶s̶t̶e̶m̶s̶ ̶a̶c̶t̶u̶a̶l̶l̶y̶ ̶d̶o̶ ̶t̶h̶i̶s̶?̶ [answered by sibling comment]
dfgdghdf 8y ago
Modern build systems (Buck, Bazel, Pants, etc) will compute a tree of hashes and manage it for you. They will also do this in the background (e.g. using Watchman). This is how Google, Facebook, Uber, Amazon etc. build their code, so the approach scales well.
- rusk 8y ago
  So what I guess is that this is for building at scale? In my experience this kind of incremental build is a matter of developer convenience so that small changes can be made and checked quickly, but builds for test and release are always built clean. I guess the codebases we're talking about here are at such a scale that the incremental approach is actually used for releases too? It still seems to me as though this is something that s should be solved architecturally though with an appropriate composition of modules ...
  - dfgdghdf 8y ago
    If your choices are: - A: Builds well for small projects, scales poorly for large projects - B: Builds well for small projects and scales well for large projects Why not choose B?
    - rusk 8y ago
      I'd always choose B given a green field, but it's when you already have A that you have to justify the effort to migrate ... EDIT my proposition is that the described build system actually scales poorly for large projects, because you've to maintain an entire tree of hashes! Notwithstanding of course having a helper process in the background to do this for me ...
      - dfgdghdf 8y ago
        Ah, I see your point now. However, in practice this is not the case. These hash-based systems scale very well in large organizations. Some more info from Google about Bazel: https://qconsf.com/sf2010/dl/qcon-sanfran-2010/slides/AshishKumar_DevelopingProductsattheSpeedandScaleofGoogle.pdf https://qconsf.com/sf2010/dl/qcon-sanfran-2010/slides/Ashish...
        rusk 8y ago
        Very interesting ... Two bullet points jumped out at me straightway: - Single monolithic code tree with mixed language code - Development on head; all releases from source Just one other concern I have now. Something like Make, or most other build-systems run pretty much straight out of the box. Something like Bazel lets say is apparently an order of magnitude more complicated. There's a few more moving parts. My only remaining qualm relates to the additional cognitive load of this stuff. At scale, as in Google this isn't so much a problem because these costs would diminish at the point where you can actually assign people to look after it. I really like the idea though! Would be nice to work in a place like Google some day ...
  - tom_ 8y ago
    It's for your day-to-day builds, so when you switch between build types while working you don't have to suffer a full rebuild.
jcelerier 8y ago
> Over a large source-tree I'd expect these costs to mount - I think I'd prefer to take my chance with timestamps! timestamps really sucks when e.g. you change header $foo which is used by a lot of your software (say, a basic data structure, or the definition of a 3d point), build (which rebuilds everything because of time stamps), do a git pull --rebase, so git undoes your changes, applies the remote commits, and redoes your change, and so you re-have to build everything again even though the header that was used everywhere has not actually changed
- rusk 8y ago
  I'd actually expect and desire a full rebuild in the case that a widely used header is changed. After a rebase it makes sense too, but I see what you mean about where no actual changes have been made though ... dare I suggest this sounds more like a bug in git though ...
  - jcelerier 8y ago
    > I'd actually expect and desire a full rebuild in the case that a widely used header is changed. yes, me too, but in this case the rebuild happens twice while the header changes only once (from the point of view of the programmer of course, from the point of view of the filesystem it changes twice). If hashes were used this would not be a problem at all.
    - rusk 8y ago
      I don't understand your point, how does this happen twi ... ohhh because dependent files get changed as well? Surely this is just a matter of how you specify your dependencies?
      - jcelerier 8y ago
        imagine your project structure is the following : header.h src1.cpp src2.cpp src3.cpp you do the following : 1. change header.h 2. rebuild. this rebuilds all your .cpp because they all use the header - this is fine and expected. 3. git pull --rebase because your coworker fixed a bug in src1.cpp which landed in the master branch 4. here when you rebuild with make, even though only src1.cpp has changed from your point of view, everything actually gets rebuilt because git stashes your change, applies the remote commits and unstashes your change, so header.h's timestamp gets changed again.

by rusk 8y ago

yetihehe 8y ago

Typical optimisation in such cases is to check file hash only when file timestamp or size changes. If after timestamp change hash is the same, it means file was not modified.

rusk 8y ago
Okay that's half the problem. Where do I keep my database of hashes though? W̶h̶i̶c̶h̶ ̶b̶u̶i̶l̶d̶ ̶s̶y̶s̶t̶e̶m̶s̶ ̶a̶c̶t̶u̶a̶l̶l̶y̶ ̶d̶o̶ ̶t̶h̶i̶s̶?̶ [answered by sibling comment]

dfgdghdf 8y ago

Modern build systems (Buck, Bazel, Pants, etc) will compute a tree of hashes and manage it for you. They will also do this in the background (e.g. using Watchman). This is how Google, Facebook, Uber, Amazon etc. build their code, so the approach scales well.

rusk 8y ago
So what I guess is that this is for building at scale? In my experience this kind of incremental build is a matter of developer convenience so that small changes can be made and checked quickly, but builds for test and release are always built clean. I guess the codebases we're talking about here are at such a scale that the incremental approach is actually used for releases too? It still seems to me as though this is something that s should be solved architecturally though with an appropriate composition of modules ...
- dfgdghdf 8y ago
  If your choices are: - A: Builds well for small projects, scales poorly for large projects - B: Builds well for small projects and scales well for large projects Why not choose B?
  - rusk 8y ago
    I'd always choose B given a green field, but it's when you already have A that you have to justify the effort to migrate ... EDIT my proposition is that the described build system actually scales poorly for large projects, because you've to maintain an entire tree of hashes! Notwithstanding of course having a helper process in the background to do this for me ...
    - dfgdghdf 8y ago
      Ah, I see your point now. However, in practice this is not the case. These hash-based systems scale very well in large organizations. Some more info from Google about Bazel: https://qconsf.com/sf2010/dl/qcon-sanfran-2010/slides/AshishKumar_DevelopingProductsattheSpeedandScaleofGoogle.pdf https://qconsf.com/sf2010/dl/qcon-sanfran-2010/slides/Ashish...
      - rusk 8y ago
        Very interesting ... Two bullet points jumped out at me straightway: - Single monolithic code tree with mixed language code - Development on head; all releases from source Just one other concern I have now. Something like Make, or most other build-systems run pretty much straight out of the box. Something like Bazel lets say is apparently an order of magnitude more complicated. There's a few more moving parts. My only remaining qualm relates to the additional cognitive load of this stuff. At scale, as in Google this isn't so much a problem because these costs would diminish at the point where you can actually assign people to look after it. I really like the idea though! Would be nice to work in a place like Google some day ...
- tom_ 8y ago
  It's for your day-to-day builds, so when you switch between build types while working you don't have to suffer a full rebuild.

jcelerier 8y ago

> Over a large source-tree I'd expect these costs to mount - I think I'd prefer to take my chance with timestamps! timestamps really sucks when e.g. you change header $foo which is used by a lot of your software (say, a basic data structure, or the definition of a 3d point), build (which rebuilds everything because of time stamps), do a git pull --rebase, so git undoes your changes, applies the remote commits, and redoes your change, and so you re-have to build everything again even though the header that was used everywhere has not actually changed

rusk 8y ago
I'd actually expect and desire a full rebuild in the case that a widely used header is changed. After a rebase it makes sense too, but I see what you mean about where no actual changes have been made though ... dare I suggest this sounds more like a bug in git though ...
- jcelerier 8y ago
  > I'd actually expect and desire a full rebuild in the case that a widely used header is changed. yes, me too, but in this case the rebuild happens twice while the header changes only once (from the point of view of the programmer of course, from the point of view of the filesystem it changes twice). If hashes were used this would not be a problem at all.
  - rusk 8y ago
    I don't understand your point, how does this happen twi ... ohhh because dependent files get changed as well? Surely this is just a matter of how you specify your dependencies?
    - jcelerier 8y ago
      imagine your project structure is the following : header.h src1.cpp src2.cpp src3.cpp you do the following : 1. change header.h 2. rebuild. this rebuilds all your .cpp because they all use the header - this is fine and expected. 3. git pull --rebase because your coworker fixed a bug in src1.cpp which landed in the master branch 4. here when you rebuild with make, even though only src1.cpp has changed from your point of view, everything actually gets rebuilt because git stashes your change, applies the remote commits and unstashes your change, so header.h's timestamp gets changed again.