4 ms·

The hard part of build systems aren't map/reduce operations, but dependency discovery as you go - because suddenly, you're rebuilding your transform pipeline wh

The hard part of build systems aren't map/reduce operations, but dependency discovery as you go - because suddenly, you're rebuilding your transform pipeline while you're executing it. That's where the fun dragons are. (I really think they're fun. But type 2 fun ;)

convolvatron 4y ago
amin vadhat wrote a little paper while in grad school about sniffing the workload while you were building it by hand and pooping out a makefile based on what you had done. seems like if you ran a makefile, recorded what the compiler looked at, you'd be in good shape wrt dependencies without having to parse the course or use other tools
- emmelaich 4y ago
  There were similar tools to watch commercial installations and reproducing them outside of the proprietary tools. Also Red Hat labs had something which watched an installation including the users commands including sudo for later replay. If I find it's name I'll edit or reply to this.
- pasc1878 4y ago
  That technology was available in the 1990s. Clearcase ran on a file system that recorded what files the build using clearmake looked at so your makefile just needed to know how to make a .o from a ,c file and all the dependencies were calculated. The build was on a server so you could get the results of other peoples (or automated) builds if they used the same dependencies. We stopped using that as the filesystem had to be on a server and so the builds were slower as involved network access and we did not have large repos where this might have been offset by a large number of build artifacts. (plus we did more on Windows)
  - groby_b 4y ago
    Plus let's not forget you needed roughly one fulltime Clearcase admin for every 10 engineers. It was a good idea with horrible execution :)
    - pasc1878 4y ago
      Don't think it was that high but yes needed admin.
- groby_b 4y ago
  I believe this is Amin's paper you refer to: https://www.usenix.org/legacy/publications/library/proceedings/usenix98/full_papers/vahdat/vahdat.pdf https://www.usenix.org/legacy/publications/library/proceedin... And for shorter reading, there's also djb's 'redo' thing: http://cr.yp.to/redo.html http://cr.yp.to/redo.html - specifically, the section of "Target files depend on nonexistent files" is an interesting one if you have tools that are allowed to generate dependencies for source files that aren't part of the source files. (It turns out, if you follow that road, that your build tools and all their libraries are implicit dependencies in the graph as well. Second law of build graphs: You will at some point question if it should be legal to do unspeakable things to people who rely on pseudo-randomness in their build tools) It is a conceptually simple problem which turns out to be surprisingly hairy in practice. That's why it's fun, and so many folks noodle on it :) If you're interested in the field, I know Google has talked about theirs: https://google-engtools.blogspot.com/2011/06/build-in-cloud-accessing-source-code.html https://google-engtools.blogspot.com/2011/06/build-in-cloud-... - and I'm sure many other companies as well.

convolvatron 4y ago

amin vadhat wrote a little paper while in grad school about sniffing the workload while you were building it by hand and pooping out a makefile based on what you had done. seems like if you ran a makefile, recorded what the compiler looked at, you'd be in good shape wrt dependencies without having to parse the course or use other tools

emmelaich 4y ago
There were similar tools to watch commercial installations and reproducing them outside of the proprietary tools. Also Red Hat labs had something which watched an installation including the users commands including sudo for later replay. If I find it's name I'll edit or reply to this.
pasc1878 4y ago
That technology was available in the 1990s. Clearcase ran on a file system that recorded what files the build using clearmake looked at so your makefile just needed to know how to make a .o from a ,c file and all the dependencies were calculated. The build was on a server so you could get the results of other peoples (or automated) builds if they used the same dependencies. We stopped using that as the filesystem had to be on a server and so the builds were slower as involved network access and we did not have large repos where this might have been offset by a large number of build artifacts. (plus we did more on Windows)
- groby_b 4y ago
  Plus let's not forget you needed roughly one fulltime Clearcase admin for every 10 engineers. It was a good idea with horrible execution :)
  - pasc1878 4y ago
    Don't think it was that high but yes needed admin.
groby_b 4y ago
I believe this is Amin's paper you refer to: https://www.usenix.org/legacy/publications/library/proceedings/usenix98/full_papers/vahdat/vahdat.pdf https://www.usenix.org/legacy/publications/library/proceedin... And for shorter reading, there's also djb's 'redo' thing: http://cr.yp.to/redo.html http://cr.yp.to/redo.html - specifically, the section of "Target files depend on nonexistent files" is an interesting one if you have tools that are allowed to generate dependencies for source files that aren't part of the source files. (It turns out, if you follow that road, that your build tools and all their libraries are implicit dependencies in the graph as well. Second law of build graphs: You will at some point question if it should be legal to do unspeakable things to people who rely on pseudo-randomness in their build tools) It is a conceptually simple problem which turns out to be surprisingly hairy in practice. That's why it's fun, and so many folks noodle on it :) If you're interested in the field, I know Google has talked about theirs: https://google-engtools.blogspot.com/2011/06/build-in-cloud-accessing-source-code.html https://google-engtools.blogspot.com/2011/06/build-in-cloud-... - and I'm sure many other companies as well.