6 ms·

Tup's main problem is it's unusual, and it doesn't have a library of build rules. But it's fast! On a related note, I've always wondered if it was possible to

by adekok 10y ago

Tup's main problem is it's unusual, and it doesn't have a library of build rules. But it's fast! On a related note, I've always wondered if it was possible to have a build system based on dynamic library injection / strace. The idea would be that you just write your build rules in shell script. Then, you run it with a special shell that catches open(), etc. in child processes (via library injection, etc). These system calls get tracked, and stored in a special build table. One that you don't have to edit. Then, when you want to run the build again, you just re-run the magic shell. It catches the various commands, and checks their inputs / outputs, and then skips running the command if the targets are up to date. e.g. $(CC) -c foo.c -o foo.o Hmm... "foo.o" is up to date with "foo.c", so I don't need to run the compiler. I just return "success!" That would get rid of all magic build systems. All build syntax. All dependency ordering. The build system would just take care of it itself. I've played with this before, enough to note that it's likely possible. But I haven't got far enough to publish it.

exDM69 10y ago
There is such a build system, but I can't remember the name right now. It tracks system calls to see every file opened by the compiler to produce exact dependency graphs (assuming compiler is deterministic). The downside is that it's Linux only. If anyone remembers the name, please do share.
- ghkbrew 10y ago
  I don't know if it's the one you're thinking of, but fabricate.py does basically that.
- artisdom 10y ago
  https://github.com/rizsotto/Bear/blob/master/README.md https://github.com/rizsotto/Bear/blob/master/README.md Bear do something similar.
- rwmj 10y ago
  I wrote something called auto-buildrequires which lets you figure out build dependencies automatically, and works roughly as you describe. https://people.redhat.com/~rjones/auto-buildrequires/ https://people.redhat.com/~rjones/auto-buildrequires/
- hendzen 10y ago
  You may be thinking of Kenton Varda's Ekam.
- codemac 10y ago
  Well.. that's what tup does so it's probably what you're thinking of!
  - exDM69 10y ago
    Yeah, tup does something similar with FUSE (afaik) but that's not what I'm thinking about.
- rjsw 10y ago
  Maybe you are thinking of ClearCase [1]. [1] https://en.wikipedia.org/wiki/Rational_ClearCase https://en.wikipedia.org/wiki/Rational_ClearCase
  - exDM69 10y ago
    No, definitely not that.
- fred256 10y ago
  Ekam perhaps? https://github.com/sandstorm-io/ekam https://github.com/sandstorm-io/ekam
KirinDave 10y ago
> Tup's main problem is it's unusual, and it doesn't have a library of build rules. But it's fast! To be honest, the only time I haven't felt like an army of build rules isn't a weight I have to carry uphill is when I use Lein (which is more of a build project tool as opposed to something as elemental as tup). Systems like SBT and Ant end up with things that feel to me like a complete and total reliance on hand-tooled plugins as opposed to a reliance on executing pre-existing executables. Gulp is the same way; even though there are perfectly serviceable and stream-like executables on every OS ready for use, Gulp insists you write more javascript to glue things into its awkward model. Maybe that's not the end of the world (as opposed to yet another make-like DSL), but in cases with truly and almost comically incomprehensible codebases like SBT (oh SBT team, every time you "simplify" your codebase you totally miss the point we don't care about your abstractions and just need to write command extensions) or Gulp (oh Gulp team, please make Gulp more like Storm/Trident or Hakyll's buidler arrows so it does more than just compile javascript and sass) then you're totally out of luck. And when we consider GNU Automake and Autoconf that do have a ton of prebuilt rules, we see an even more dire situation: where the language describing the rules is more complicated than the underlying concepts they service.
- moron4hire 10y ago
  The one thing I like about Gulp is that it makes it easy to make a project out of your build process, to basically enforce your own convention over configuration. I have a single, canonical build process for all of my projects and it gets installed through NPM like every other dependency. The individual gulpfiles for my projects end up being very simple and readable. It has the side effect of drastically cleaning up my package.json file, too. The only other thing I like about Gulp is that it works without too much hassle on Windows and *Nix. Other than that, it's dung. Though it's certainly better than Grunt, it's not objectively good. That seems to be a problem with a lot of things these days, "relative better than the alternatives, not something you'd actually want on its own merits."
  - KirinDave 10y ago
    Recently I had a local Gulp expert who had written a plugin try and help me with a simple problem: I produce a .json of load balancers from a CF stack. We need to run a command on every entry in the map to annotate it with extra information from another AWS call. It's a tricky problem not amenable to usual shell scripting (unless you use a shell with a native notion of maps and json). For the life of us, we could not find a way to make Gulp do this without essentially writing a solid block of Javascript that just does exactly what we wanted (in which case, why use anything but the shell script approach?). I understand that stream rejoining is a complex problem, but Gulp seems ideologically opposed to a more generic approach here. They get mad when people try to inject non-file based values into the system, and actively close off alternatives and shame them on their github issues. Gulp is a complete mess, in my opinion.
CmdrKrool 10y ago
See fabricate.py: https://github.com/SimonAlfie/fabricate https://github.com/SimonAlfie/fabricate Here's an earlier comment I wrote about it, which is also in the middle of a discussion about Tup, funnily enough: https://news.ycombinator.com/item?id=4190804 https://news.ycombinator.com/item?id=4190804 I'm still using Fabricate quite happily today for personal projects in a variety of languages, though I've come to realise that the main downside of it is that programs run slower under strace - a typical gcc invocation to create a single object file might take say, 40% longer. Fabricate.py is also then quite slow to parse the textual output from strace. Inspired by https://blog.nelhage.com/2010/08/write-yourself-an-strace-in-70-lines-of-code/ https://blog.nelhage.com/2010/08/write-yourself-an-strace-in... I wrote a mini-strace that's hard coded to catch the open() and other relevant calls as fast as possible. That helped a fair bit. I've done a lot of hacking on my own copy of Fabricate too which helped a bit, but ultimately I ought to make it do the dependency parsing and saving in a seperate thread, which should help a lot - I'm still not doing parallel builds so I do have spare cores sitting there doing nothing. I tried a FUSE backend (https://github.com/SimonAlfie/fabricate/pull/60 https://github.com/SimonAlfie/fabricate/pull/60) whereby the filesystem tells you what's being accessed, and that's practically the same as full speed but the downside is that the path jiggery-pokery involved then causes some subsequent problems with eg. gdb not being able to find source files. I've gone back to the strace method for now. Despite all these niggles, I still use this system wherever possible because it's so, so good having a 'proper' programming language to run my builds, and being able to work forwards in the manner of a shell script. And yeah the automatic detection of changes ultimately saves a lot of time. I can share some of my hacked-up scripts if anyone's interested.
- tedunangst 10y ago
  Among other options, Electric Cloud, now Electric Accelerator, is a product I tried about, oh, ten years ago. Looks like they're still around.
- thechao 10y ago
  I love fabricate---even helped with the OSX port. I had to give up on it when El Cap came out; I just didn't have time to work around SIP. The other problem with fabricate/tup/not-Make is they store the structure of the dependency graph. This leads to ugly build ... worries when you have subtle dependency changes caused by, e.g., updates to system headers, or interactions in large rule-bases. I just never had confidence the builds were correct. However, using Make + ccache + a custom ar wrapper (port the deterministic flags from Linux to BSD), I got all the speed up of tup/fabricate, using off-the-shelf open source tooling. I've never looked back.
norswap 10y ago
Tup does this. See here: http://gittup.org/tup/ex_a_first_tupfile.html http://gittup.org/tup/ex_a_first_tupfile.html > The trick is that tup instruments all commands that it executes in order to determine what files were actually read from (the inputs) and written to (the outputs). When the C preprocessor opens the header file, tup will notice that and automatically add the dependency. In fact, we don't have to specify the C input file either, but you can leave that in there for now since we'll use it in the next section.
- iainmerrick 10y ago
  Not really, although it's closer than other build systems. You can omit some dependencies but not all. If I remember right (I tried out tup a few months back) you still have to specify outputs, and there are some fiddly restrictions around writing files to other directories (although they were in the process of improving that). Writing a tup build script doesn't really feel like just writing a shell script that does the thing you want. It's very cool technology and fun to play with, but I didn't find it fundamentally easier to use than other build systems.
iainmerrick 10y ago
I had the same idea. I would absolutely love this! I haven't tried building it because it seems like it needs FUSE or something similar and FUSE is flaky on OS X. Every time I upgraded OS X I'd have a hard time getting tup to work again.
skybrian 10y ago
I don't think this really gets rid of all dependencies; it just reduces the build graph a bit. Even a shell script implicitly provides a simple dependency graph: you run the commands from top to bottom, so that's a linear order. If you have two build targets, do you have two shell scripts? Do you create a library so they share code? That's a dependency graph too. What's really going on is that the build system doesn't know the true dependency graph until it's run each command once. The first time around, it uses a conservative approximation that you specify. In tup's case, you don't need to specify any dependencies that aren't themselves generated. That's generally true of header files. If you have a generated header file, you still need to explicitly add the dependency to the tup file [1]. [1] http://gittup.org/tup/ex_generated_header.html http://gittup.org/tup/ex_generated_header.html
wahern 10y ago
The switch to statically compiled binaries means that approach is increasingly impractical. AFAIU Go, for example, completely bypasses libc for those system calls. The same might be true for Java, too. Anybody know if modern JVMs bother calling into libc for basic system calls like open, or otherwise obey dynamic interposition of standard library routines?
onionjake 10y ago
I've hacked on audited objects before to accomplish something similar (https://github.com/boyski/audited-objects https://github.com/boyski/audited-objects) It uses ld preload to do it and not strace, but same idea. I was checking existing makefile rules, but was considering using audited objects to help transition to tup or bazil. IIRC strace wouldn't work if you used any setuid binaries (horrible but our makefiles used sudo for some things). There are also some tricky corner cases like sed -i which creates a new file then moves it.

by adekok 10y ago

exDM69 10y ago

There is such a build system, but I can't remember the name right now. It tracks system calls to see every file opened by the compiler to produce exact dependency graphs (assuming compiler is deterministic). The downside is that it's Linux only. If anyone remembers the name, please do share.

ghkbrew 10y ago
I don't know if it's the one you're thinking of, but fabricate.py does basically that.
artisdom 10y ago
https://github.com/rizsotto/Bear/blob/master/README.md https://github.com/rizsotto/Bear/blob/master/README.md Bear do something similar.
rwmj 10y ago
I wrote something called auto-buildrequires which lets you figure out build dependencies automatically, and works roughly as you describe. https://people.redhat.com/~rjones/auto-buildrequires/ https://people.redhat.com/~rjones/auto-buildrequires/
hendzen 10y ago
You may be thinking of Kenton Varda's Ekam.
codemac 10y ago
Well.. that's what tup does so it's probably what you're thinking of!
- exDM69 10y ago
  Yeah, tup does something similar with FUSE (afaik) but that's not what I'm thinking about.
rjsw 10y ago
Maybe you are thinking of ClearCase [1]. [1] https://en.wikipedia.org/wiki/Rational_ClearCase https://en.wikipedia.org/wiki/Rational_ClearCase
- exDM69 10y ago
  No, definitely not that.
fred256 10y ago
Ekam perhaps? https://github.com/sandstorm-io/ekam https://github.com/sandstorm-io/ekam

KirinDave 10y ago

> Tup's main problem is it's unusual, and it doesn't have a library of build rules. But it's fast! To be honest, the only time I haven't felt like an army of build rules isn't a weight I have to carry uphill is when I use Lein (which is more of a build project tool as opposed to something as elemental as tup). Systems like SBT and Ant end up with things that feel to me like a complete and total reliance on hand-tooled plugins as opposed to a reliance on executing pre-existing executables. Gulp is the same way; even though there are perfectly serviceable and stream-like executables on every OS ready for use, Gulp insists you write more javascript to glue things into its awkward model. Maybe that's not the end of the world (as opposed to yet another make-like DSL), but in cases with truly and almost comically incomprehensible codebases like SBT (oh SBT team, every time you "simplify" your codebase you totally miss the point we don't care about your abstractions and just need to write command extensions) or Gulp (oh Gulp team, please make Gulp more like Storm/Trident or Hakyll's buidler arrows so it does more than just compile javascript and sass) then you're totally out of luck. And when we consider GNU Automake and Autoconf that do have a ton of prebuilt rules, we see an even more dire situation: where the language describing the rules is more complicated than the underlying concepts they service.

moron4hire 10y ago
The one thing I like about Gulp is that it makes it easy to make a project out of your build process, to basically enforce your own convention over configuration. I have a single, canonical build process for all of my projects and it gets installed through NPM like every other dependency. The individual gulpfiles for my projects end up being very simple and readable. It has the side effect of drastically cleaning up my package.json file, too. The only other thing I like about Gulp is that it works without too much hassle on Windows and *Nix. Other than that, it's dung. Though it's certainly better than Grunt, it's not objectively good. That seems to be a problem with a lot of things these days, "relative better than the alternatives, not something you'd actually want on its own merits."
- KirinDave 10y ago
  Recently I had a local Gulp expert who had written a plugin try and help me with a simple problem: I produce a .json of load balancers from a CF stack. We need to run a command on every entry in the map to annotate it with extra information from another AWS call. It's a tricky problem not amenable to usual shell scripting (unless you use a shell with a native notion of maps and json). For the life of us, we could not find a way to make Gulp do this without essentially writing a solid block of Javascript that just does exactly what we wanted (in which case, why use anything but the shell script approach?). I understand that stream rejoining is a complex problem, but Gulp seems ideologically opposed to a more generic approach here. They get mad when people try to inject non-file based values into the system, and actively close off alternatives and shame them on their github issues. Gulp is a complete mess, in my opinion.

CmdrKrool 10y ago

See fabricate.py: https://github.com/SimonAlfie/fabricate https://github.com/SimonAlfie/fabricate Here's an earlier comment I wrote about it, which is also in the middle of a discussion about Tup, funnily enough: https://news.ycombinator.com/item?id=4190804 https://news.ycombinator.com/item?id=4190804 I'm still using Fabricate quite happily today for personal projects in a variety of languages, though I've come to realise that the main downside of it is that programs run slower under strace - a typical gcc invocation to create a single object file might take say, 40% longer. Fabricate.py is also then quite slow to parse the textual output from strace. Inspired by https://blog.nelhage.com/2010/08/write-yourself-an-strace-in-70-lines-of-code/ https://blog.nelhage.com/2010/08/write-yourself-an-strace-in... I wrote a mini-strace that's hard coded to catch the open() and other relevant calls as fast as possible. That helped a fair bit. I've done a lot of hacking on my own copy of Fabricate too which helped a bit, but ultimately I ought to make it do the dependency parsing and saving in a seperate thread, which should help a lot - I'm still not doing parallel builds so I do have spare cores sitting there doing nothing. I tried a FUSE backend (https://github.com/SimonAlfie/fabricate/pull/60 https://github.com/SimonAlfie/fabricate/pull/60) whereby the filesystem tells you what's being accessed, and that's practically the same as full speed but the downside is that the path jiggery-pokery involved then causes some subsequent problems with eg. gdb not being able to find source files. I've gone back to the strace method for now. Despite all these niggles, I still use this system wherever possible because it's so, so good having a 'proper' programming language to run my builds, and being able to work forwards in the manner of a shell script. And yeah the automatic detection of changes ultimately saves a lot of time. I can share some of my hacked-up scripts if anyone's interested.

tedunangst 10y ago
Among other options, Electric Cloud, now Electric Accelerator, is a product I tried about, oh, ten years ago. Looks like they're still around.
thechao 10y ago
I love fabricate---even helped with the OSX port. I had to give up on it when El Cap came out; I just didn't have time to work around SIP. The other problem with fabricate/tup/not-Make is they store the structure of the dependency graph. This leads to ugly build ... worries when you have subtle dependency changes caused by, e.g., updates to system headers, or interactions in large rule-bases. I just never had confidence the builds were correct. However, using Make + ccache + a custom ar wrapper (port the deterministic flags from Linux to BSD), I got all the speed up of tup/fabricate, using off-the-shelf open source tooling. I've never looked back.

norswap 10y ago

Tup does this. See here: http://gittup.org/tup/ex_a_first_tupfile.html http://gittup.org/tup/ex_a_first_tupfile.html > The trick is that tup instruments all commands that it executes in order to determine what files were actually read from (the inputs) and written to (the outputs). When the C preprocessor opens the header file, tup will notice that and automatically add the dependency. In fact, we don't have to specify the C input file either, but you can leave that in there for now since we'll use it in the next section.

iainmerrick 10y ago
Not really, although it's closer than other build systems. You can omit some dependencies but not all. If I remember right (I tried out tup a few months back) you still have to specify outputs, and there are some fiddly restrictions around writing files to other directories (although they were in the process of improving that). Writing a tup build script doesn't really feel like just writing a shell script that does the thing you want. It's very cool technology and fun to play with, but I didn't find it fundamentally easier to use than other build systems.

iainmerrick 10y ago

I had the same idea. I would absolutely love this! I haven't tried building it because it seems like it needs FUSE or something similar and FUSE is flaky on OS X. Every time I upgraded OS X I'd have a hard time getting tup to work again.

skybrian 10y ago

I don't think this really gets rid of all dependencies; it just reduces the build graph a bit. Even a shell script implicitly provides a simple dependency graph: you run the commands from top to bottom, so that's a linear order. If you have two build targets, do you have two shell scripts? Do you create a library so they share code? That's a dependency graph too. What's really going on is that the build system doesn't know the true dependency graph until it's run each command once. The first time around, it uses a conservative approximation that you specify. In tup's case, you don't need to specify any dependencies that aren't themselves generated. That's generally true of header files. If you have a generated header file, you still need to explicitly add the dependency to the tup file [1]. [1] http://gittup.org/tup/ex_generated_header.html http://gittup.org/tup/ex_generated_header.html

wahern 10y ago

The switch to statically compiled binaries means that approach is increasingly impractical. AFAIU Go, for example, completely bypasses libc for those system calls. The same might be true for Java, too. Anybody know if modern JVMs bother calling into libc for basic system calls like open, or otherwise obey dynamic interposition of standard library routines?

onionjake 10y ago

I've hacked on audited objects before to accomplish something similar (https://github.com/boyski/audited-objects https://github.com/boyski/audited-objects) It uses ld preload to do it and not strace, but same idea. I was checking existing makefile rules, but was considering using audited objects to help transition to tup or bazil. IIRC strace wouldn't work if you used any setuid binaries (horrible but our makefiles used sudo for some things). There are also some tricky corner cases like sed -i which creates a new file then moves it.