5 ms·

I agree with the blog, in that it seems disappointing that it doesn't integrate fabricate.py/memoize.py automatic-dependency-detection via strace. Those projec

I agree with the blog, in that it seems disappointing that it doesn't integrate fabricate.py/memoize.py automatic-dependency-detection via strace. Those projects seem to advance the state of the art a lot more as a make-replacement. The git analogy is great though; while in general I think there should be a very high bar for throwing away working systems like make, in this case there's a potentially worthwhile set of different ideas: hashing for change detection, strace for dependency detection, and using scripts/whatever instead of a make-specific DSL.

unshift 16y ago
how does automatic dependency detection using strace work?
- pyre 16y ago
  Maybe you get strace output from a running program and it uses that to evaluate what dependencies/versions you need?
  - unshift 16y ago
    how does that fit into a build system? why would you require inspection of a running program to resolve build-time dependencies? and moreover, it's not like running a binary requires it to open all of the header and source files needed at build time, so how would you glean any useful data from it? and what about systems that don't have strace, like BSD or OSX or solaris or windows? there's dtrace on those platforms (minus windows), which requires root, and requiring root access for dependency resolution isn't exactly a great idea.
    - prodigal_erik 16y ago
      It sounds like you don't strace your project, you strace the compilers and linkers to find out which headers and libraries they actually use. Then you know exactly which builds are possibly out of date at any point, even after a platform upgrade. Nobody bothers to note stuff like libc in their makefiles, a mistake which make can't help you avoid.
    - erickt 16y ago
      I think reading strace's output was a great experiment, but I don't think it's really that necessary. Most programs have some way of spitting out the dependencies, and some others follow pretty standard rules for file generation. For instance, ocaml optional interface files generate a .cmi file from a .mli file, and implementation files generate a .cmo file from a .ml file. However, if no .mli file exists, compiling the .ml file will also generate a .cmi file. It can be a pain, but it's really not that hard to handle if your build system provides a way to abstract out build patterns.
    - Freaky 16y ago
      FreeBSD has kernel DTrace (off by default), with userspace tracing on its way (in CURRENT). ktrace is a better option there, especially if you want to support other BSD systems as well. I've used ktrace to help create minimal jail environments, quite handy: http://www.aagh.net/files/mkjail http://www.aagh.net/files/mkjail
  - burgerbrain 16y ago
    So basically if you don't completely exercise your program while creating your build system, your build system won't know about all your dependencies? Suddenly make is starting to seem pretty nice...
    - justinsb 16y ago
      We're talking about build dependencies, not runtime dependencies. A build dependency is - by definition - one that is touched during the build process. strace exactly identifies that set.
      - unshift 16y ago
        i don't see how... if my code requires foo.h and bar.c, but those get compiled and linked into one executable. how will running that program expose that? moreover, how can you run strace to figure out dependencies on a program that hasn't been built yet? how does it determine when new dependencies are added?
        justinsb 16y ago
        You strace the build steps (gcc bar.c), not the output program
      - burgerbrain 16y ago
        This reminds me a lot of CDE Pack, can you explain how this is different? If my program needs to be dynamically linked with, say, OpenSSL, but rarely actually uses it, how does introducing strace into this situation help?
      - Vitaly 16y ago
        you don't care about a dependencies of an non-existing target. you have to run it anyway. and once you run it at least once you will have the full set of its dependencies for the next time you run your build.
- justinsb 16y ago
  http://benno.id.au/blog/2008/06/06/memoize-build-framework http://benno.id.au/blog/2008/06/06/memoize-build-framework
Peaker 16y ago
Does the strace approach allow parallelizing the build?
- justinsb 16y ago
  I suspect you'd have to repeat a task if you discovered that it was dependent on another task's output. You would still discover this though, so your final output would still be correct.
erickt 16y ago
I agree, I feel like redo isn't going far enough. I'm not sure if I'm missing something, but semantically this is very similar to make, even though it looks like it is a better syntax. Of course, I'm completely biased :) I have my own build system called fbuild (http://github.com/erickt/fbuild http://github.com/erickt/fbuild) that's conceptually similar to fabricate/memoize. There's also mem (http://srp.github.com/mem/getting-started.html http://srp.github.com/mem/getting-started.html) and wonderbuild (http://retropaganda.info/~bohan/work/sf/psycle/branches/bohan/wonderbuild/wonderbuild/README http://retropaganda.info/~bohan/work/sf/psycle/branches/boha... -- although that link appears to be broken at the moment) if you're interested in some other projects that have similar roots. I believe the key insight that we all independently made is that we can use the procedural execution stack for the dependency tree, and cache the function arguments and results to avoid work duplication. I believe this helps someone reason about what's going on in the build system compared to the declarative make systems because you can easily trace the inputs and outputs of some dependency. Furthermore, you can do some pretty interesting things with this model if you push this idea. It's pretty simple to do integrated configuration and building. For instance in fbuild, I can do (fbuild uses python3 for it's host language): import fbuild.builders.c import fbuild.config.c.linux def build(ctx): # make a c compiler. builder = fbuild.builders.c.guess_static(ctx) # check if the system supports epoll if fbuild.config.c.linux.sys_epoll_h(builder).header: use_epoll = True else: use_epoll = False builder.build_exe('foo', ['foo.c'], defines=['USE_EPOLL' if use_epoll else 'USE_SELECT']) I don't think many of the new build systems have looked into the configuration issue yet, which is a shame. I'd love to get some more ideas out there.
abdulla 16y ago
You might be interested in tup (http://gittup.org/tup http://gittup.org/tup) in that case. It finds dependencies by preloading a library that tracks all the file accesses that a program makes. IMHO I think this is a nicer approach than scanning strace output.
- justinsb 16y ago
  Thanks - great link. Certainly much faster than strace!
wnoise 16y ago
That can be handled entirely as a separate process that creates the dependencies that redo then uses. Separation of concerns argues that here's no need to tie them together. In fact, they have an example of using gcc to generate header-file dependencies for C. A command-line tool that ran a command and spat out what files it accessed (via strace, dynamic library imposition, or even crazier methods like interpreting the binary) would hook into this system fairly seamlessly.
acon 16y ago
I really like the idea of strace for dependency detection. Unfortunately some people still use Windows which means I have to build on Windows. Does anyone know of an equivalent to strace one could use for dependency detection?

unshift 16y ago

how does automatic dependency detection using strace work?

pyre 16y ago
Maybe you get strace output from a running program and it uses that to evaluate what dependencies/versions you need?
- unshift 16y ago
  how does that fit into a build system? why would you require inspection of a running program to resolve build-time dependencies? and moreover, it's not like running a binary requires it to open all of the header and source files needed at build time, so how would you glean any useful data from it? and what about systems that don't have strace, like BSD or OSX or solaris or windows? there's dtrace on those platforms (minus windows), which requires root, and requiring root access for dependency resolution isn't exactly a great idea.
  - prodigal_erik 16y ago
    It sounds like you don't strace your project, you strace the compilers and linkers to find out which headers and libraries they actually use. Then you know exactly which builds are possibly out of date at any point, even after a platform upgrade. Nobody bothers to note stuff like libc in their makefiles, a mistake which make can't help you avoid.
  - erickt 16y ago
    I think reading strace's output was a great experiment, but I don't think it's really that necessary. Most programs have some way of spitting out the dependencies, and some others follow pretty standard rules for file generation. For instance, ocaml optional interface files generate a .cmi file from a .mli file, and implementation files generate a .cmo file from a .ml file. However, if no .mli file exists, compiling the .ml file will also generate a .cmi file. It can be a pain, but it's really not that hard to handle if your build system provides a way to abstract out build patterns.
  - Freaky 16y ago
    FreeBSD has kernel DTrace (off by default), with userspace tracing on its way (in CURRENT). ktrace is a better option there, especially if you want to support other BSD systems as well. I've used ktrace to help create minimal jail environments, quite handy: http://www.aagh.net/files/mkjail http://www.aagh.net/files/mkjail
- burgerbrain 16y ago
  So basically if you don't completely exercise your program while creating your build system, your build system won't know about all your dependencies? Suddenly make is starting to seem pretty nice...
  - justinsb 16y ago
    We're talking about build dependencies, not runtime dependencies. A build dependency is - by definition - one that is touched during the build process. strace exactly identifies that set.
    - unshift 16y ago
      i don't see how... if my code requires foo.h and bar.c, but those get compiled and linked into one executable. how will running that program expose that? moreover, how can you run strace to figure out dependencies on a program that hasn't been built yet? how does it determine when new dependencies are added?
      - justinsb 16y ago
        You strace the build steps (gcc bar.c), not the output program
    - burgerbrain 16y ago
      This reminds me a lot of CDE Pack, can you explain how this is different? If my program needs to be dynamically linked with, say, OpenSSL, but rarely actually uses it, how does introducing strace into this situation help?
    - Vitaly 16y ago
      you don't care about a dependencies of an non-existing target. you have to run it anyway. and once you run it at least once you will have the full set of its dependencies for the next time you run your build.
justinsb 16y ago
http://benno.id.au/blog/2008/06/06/memoize-build-framework http://benno.id.au/blog/2008/06/06/memoize-build-framework

Peaker 16y ago

Does the strace approach allow parallelizing the build?

justinsb 16y ago
I suspect you'd have to repeat a task if you discovered that it was dependent on another task's output. You would still discover this though, so your final output would still be correct.

erickt 16y ago

I agree, I feel like redo isn't going far enough. I'm not sure if I'm missing something, but semantically this is very similar to make, even though it looks like it is a better syntax. Of course, I'm completely biased :) I have my own build system called fbuild (http://github.com/erickt/fbuild http://github.com/erickt/fbuild) that's conceptually similar to fabricate/memoize. There's also mem (http://srp.github.com/mem/getting-started.html http://srp.github.com/mem/getting-started.html) and wonderbuild (http://retropaganda.info/~bohan/work/sf/psycle/branches/bohan/wonderbuild/wonderbuild/README http://retropaganda.info/~bohan/work/sf/psycle/branches/boha... -- although that link appears to be broken at the moment) if you're interested in some other projects that have similar roots. I believe the key insight that we all independently made is that we can use the procedural execution stack for the dependency tree, and cache the function arguments and results to avoid work duplication. I believe this helps someone reason about what's going on in the build system compared to the declarative make systems because you can easily trace the inputs and outputs of some dependency. Furthermore, you can do some pretty interesting things with this model if you push this idea. It's pretty simple to do integrated configuration and building. For instance in fbuild, I can do (fbuild uses python3 for it's host language): import fbuild.builders.c import fbuild.config.c.linux def build(ctx): # make a c compiler. builder = fbuild.builders.c.guess_static(ctx) # check if the system supports epoll if fbuild.config.c.linux.sys_epoll_h(builder).header: use_epoll = True else: use_epoll = False builder.build_exe('foo', ['foo.c'], defines=['USE_EPOLL' if use_epoll else 'USE_SELECT']) I don't think many of the new build systems have looked into the configuration issue yet, which is a shame. I'd love to get some more ideas out there.

abdulla 16y ago

You might be interested in tup (http://gittup.org/tup http://gittup.org/tup) in that case. It finds dependencies by preloading a library that tracks all the file accesses that a program makes. IMHO I think this is a nicer approach than scanning strace output.

justinsb 16y ago
Thanks - great link. Certainly much faster than strace!

wnoise 16y ago

That can be handled entirely as a separate process that creates the dependencies that redo then uses. Separation of concerns argues that here's no need to tie them together. In fact, they have an example of using gcc to generate header-file dependencies for C. A command-line tool that ran a command and spat out what files it accessed (via strace, dynamic library imposition, or even crazier methods like interpreting the binary) would hook into this system fairly seamlessly.

acon 16y ago

I really like the idea of strace for dependency detection. Unfortunately some people still use Windows which means I have to build on Windows. Does anyone know of an equivalent to strace one could use for dependency detection?