23 ms·

Tup – A file-based build system for Linux, OS X, and Windows

169 points by mynameislegion 10y ago

tf2manu994 10y ago
Damn this is quick. Also, the URL made me think it was going to be on so.e GitHub competitor called gittup, heh.
wreft 10y ago
I first saw Tup a few years ago as it seemed to be the preferred way to filewatch/automagically-transpire moonscript to lua.
deleted 10y ago
[deleted]
jpgvm 10y ago
We use Tup to build Flynn [1], it's a pretty neat build system. The only real drawback is that it's hard to get to work on some operating systems because of dependency on FUSE. That said it still beats the crap out GNU Make or CMake etc. [1] https://github.com/flynn/flynn https://github.com/flynn/flynn
- falcolas 10y ago
  > That said it still beats the crap out GNU Make How? Their description was useless in describing why it's "obviously" so much better.
  - nerdponx 10y ago
    > See the difference? The arrows go up. This makes it very fast. I don't think I've seen a more intelligence-insulting description of a product.
    - chrismonsanto 10y ago
      it's a joke...
    - todd8 10y ago
      Oh, you didn't read far enough; the description is written with a sense of humor, essentially parodying the very "intelligence-insulting descriptions" that one sees frequently on the internet.
    - exDM69 10y ago
      Well that was a bit light hearted but that's highlights the big difference: tup works bottom up, make works top down. The paper linked there provides a more thorough explanation.
  - mynameislegion 10y ago
    The very last part of this page explains it quite well: http://gittup.org/tup/ex_a_first_tupfile.html http://gittup.org/tup/ex_a_first_tupfile.html
  - adekok 10y ago
    GNU make starts of with targets, and then talks the build graph (target -> source) to find source files which will re-build the targets. And GNU Make has to scan the files. Tup starts off with the list of sources which have changed, and then walks the build graph (source -> target) to find out which targets need to be rebuilt. And Tup gets notified of file changes. So where Make scans all targets and all source files, Tup gets told which source file has changed, and then rebuilds only the necessary targets. That being said, the following comparison is a bit odd: http://gittup.org/tup/make_vs_tup.html http://gittup.org/tup/make_vs_tup.html I don't think it's a requirement of GNU Make that building a project with N source files takes exponential time. I think its just that the GNU Make code is terrible.
    - LukeShu 10y ago
      GNU Make doesn't take exponential time, it takes ~linear time. The Y value on the graphs is exponential, because the input is growing exponentially. The graph is just rubbing in that a noop Tup build is O(1) instead of Make's O(n) by showing crazy large n. Which is certainly great on Tup's part![1] The Tup author's description of Tup's algorithm in comparison has always rubbed me the wrong way, though. It isn't some entirely new generation of build system like they present in the Tup paper. It's just caching stat(2) between iterations[2]. The build naturally flows down like in the Make diagram on the main Tup page; but with Tup we cache the status of each node, and become informed when one of them becomes invalid; then the upward flowing arrows of the Tup diagram is the cache-invalidation of the lower nodes flowing up to the higher nodes. This is the totally natural way of implementing caching if we were to add that to GNU Make. So why don't GNU Make and other systems cache the DAG of stat(2) calls? As the saying says, cache invalidation is hard. Tup gets away with caching by receiving an a priori list of filesystem changes, so it knows which nodes to invalidate; other build systems have to scan the filesystem for changes. Tup gets this list of changes by having a FUSE filesystem sit between the user and the actual filesystem, logging all changes. While this is great for many users, it isn't quite general. It will break if we edit anything "offline", perhaps on a thumb drive on another computer. It will break in a number of different NFS setups. I don't mean to say that the caching doesn't have value, but it isn't the silver bullet that the Tup author makes it out to be. [1]: In the Tup paper, the author gives some fancy O(...) expressions. These are... odd; they're kind of hand-wavey, as they are based on expected fan-out of directories, and don't really give a good apples-to-apples comparison. They certainly are valuable to analyze, but are pretty misleading if you aren't critical. [2]: OK, that's not strictly true. The "Beta build system" (the next generation of build systems that Tup is supposedly part of) algorithm also has the feature that it will automatically remove a generated file that was part of the DAG, but isn't anymore. This is a nifty feature that is made possible by the caching; but IMO it doesn't constitute a core change.
piyush_soni 10y ago
To the authors : It would be really great if you could compare the SCons build system to Tup (with some numbers) - so that I can convince my managers to switch :).
agumonkey 10y ago
I can't recall what exactly but I hit expressiveness problems with tup (after proper RTFM). Probably some self referential issue. For the use case listed it's indeed very nice and very fast.
qwertyuiop924 10y ago
The FUSE dependancy is pretty unfortunate. Is there any way to get rid of it? Other than that, it's pretty cool. And the creator clearly has a sense of humor, something which is far rarer than it should be.
- adekok 10y ago
  The same functionality should be available in various inotify / dnotify implementations.
  - majewsky 10y ago
    Not really. inotify requires you to set up a watch on every single file or directory that you want to watch. To see how this escalates, install the inotify-tools and do inotifywatch -d /path/to/directory/with/a/lot/of/files
    - qwertyuiop924 10y ago
      What about kqueue or eventports?
      - krakensden 10y ago
        Kqueue and OS X's file watcher thing are worse than inotify. They give inexact reports of changes, so you have to do a bunch of manual scanning afterwards. I really don't understand the grandparent's gripe. Inotify scales well, supports race free "watch a whole directory tree", and has a nice API.
        codyps 10y ago
        There is a per-user limit to the number of inotify handles available (max_user_watches) and the default value is 8192. The limit exists because there is a ~1KiB kernel memory overhead per watch (though there should really be a way for them to take part in normal memory accounting per-process). If one wants to watch a directory tree, one needs an inotify watch handle per subdirectory in that tree. On large trees (or if more than 1 process is using inotify), that number of watches can be exceeded. As lots of folks are looking for recursive watches, they aren't happy with needing to allocate & manage a bunch of handles when they see what they want as a single item. That said, I'm not sure the way the kernel thinks about fs notifications internally would allow a single handle recursive watch at the moment. In any case, the amount of info one can obtain by using fuse (or any fs like nfs or 9p) to intercept filesystem accesses is a bit larger. At the very least, one can (in most cases) directly observe the ranges of the file that were modified (though that's not quite so important for tup, afaik). There also aren't any queue overruns (which can happen in inotify) because one will just slow the filesystem operations down instead (whether this is desirable or not depends on the application).
ggambetta 10y ago
While I can believe the claim it's better than make, it would be way more interesting to see how it compares to Bazel. After using it (or rather Blaze) at Google and now Bazel at Improbable, I consider it the gold standard in build tools. If anything, I wish Google would open source the rest of the build "ecosystem" that together with Blaze let you build the whole codebase in seconds. It was pretty amazing.
- simonhorlick 10y ago
  I've also been using Bazel for a couple of months now and it's fantastic. Simple, declarative syntax, blazingly fast. I can't recommend it highly enough.
- Tojot 10y ago
  I'm a Bazel person myself as well, but tup with the lua syntax might be getting somewhere.
adekok 10y ago
Tup's main problem is it's unusual, and it doesn't have a library of build rules. But it's fast! On a related note, I've always wondered if it was possible to have a build system based on dynamic library injection / strace. The idea would be that you just write your build rules in shell script. Then, you run it with a special shell that catches open(), etc. in child processes (via library injection, etc). These system calls get tracked, and stored in a special build table. One that you don't have to edit. Then, when you want to run the build again, you just re-run the magic shell. It catches the various commands, and checks their inputs / outputs, and then skips running the command if the targets are up to date. e.g. $(CC) -c foo.c -o foo.o Hmm... "foo.o" is up to date with "foo.c", so I don't need to run the compiler. I just return "success!" That would get rid of all magic build systems. All build syntax. All dependency ordering. The build system would just take care of it itself. I've played with this before, enough to note that it's likely possible. But I haven't got far enough to publish it.
- exDM69 10y ago
  There is such a build system, but I can't remember the name right now. It tracks system calls to see every file opened by the compiler to produce exact dependency graphs (assuming compiler is deterministic). The downside is that it's Linux only. If anyone remembers the name, please do share.
  - ghkbrew 10y ago
    I don't know if it's the one you're thinking of, but fabricate.py does basically that.
  - artisdom 10y ago
    https://github.com/rizsotto/Bear/blob/master/README.md https://github.com/rizsotto/Bear/blob/master/README.md Bear do something similar.
  - rwmj 10y ago
    I wrote something called auto-buildrequires which lets you figure out build dependencies automatically, and works roughly as you describe. https://people.redhat.com/~rjones/auto-buildrequires/ https://people.redhat.com/~rjones/auto-buildrequires/
brudgers 10y ago
The related linux distribution, Gittup: http://gittup.org/gittup/ http://gittup.org/gittup/
brudgers 10y ago
The related linux distribution, Gittup: http://gittup.org/gittup/ http://gittup.org/gittup/
todd8 10y ago
This reminds me of DJB's ideas for a build system, redo [1]. However, it never seemed to gain any traction. (or did it? [2]) [1] http://cr.yp.to/redo.html http://cr.yp.to/redo.html [2] http://apenwarr.ca/log/?m=201012#14 http://apenwarr.ca/log/?m=201012#14
- Tharre 10y ago
  Well, if you want to see redo in action, there are plenty of implementations available: [0] [0] http://news.dieweltistgarnichtso.net/bin/redo-sh.html http://news.dieweltistgarnichtso.net/bin/redo-sh.html
- sigil 10y ago
  I can recommend apenwarr's redo implementation [0]. There's occasional activity on the mailing list [1], which leads me to believe people are using it, but not promoting it much. The apenwarr implementation includes a full Python implementation as well as a minimal version in < 200 lines of sh. The minimal version doesn't support incremental rebuilds -- the out-of-date-ness tracking in the full Python version uses sqlite -- but it's good for understanding the redo concept. Also good for embedded contexts. I used the full redo implementation for some data processing tasks once, with mixed results. It was a situation where I couldn't declare the dependency graph up front. With redo, each target declares its dependencies locally when it builds, and redo assembles and tracks the dynamic dependency graph. It's pretty neat, but become difficult to reason about and debug. Could be that I never got comfortable with the new paradigm, or could be that essential tooling was missing, not sure. I still think redo is promising. Anyway after a decade of messing with shiny new build tools, I finally learned to stop worrying and love the bomb (make). It's weird and warty but surprisingly capable. Worth the learning investment. Oh and jgrahamc's "GNU Make Book" is great. [2] [0] https://github.com/apenwarr/redo https://github.com/apenwarr/redo [1] https://groups.google.com/forum/#!forum/redo-list https://groups.google.com/forum/#!forum/redo-list [2] https://www.nostarch.com/gnumake https://www.nostarch.com/gnumake
ricardobeat 10y ago
The index page makes it sound like a parody of something, took me a while to figure it is functional software.
- ubertaco 10y ago
  I do love the quirky sense of humor though: "In a typical build system, the dependency arrows go down. Although this is the way they would naturally go due to gravity, it is unfortunately also where the enemy's gate is. This makes it very inefficient and unfriendly. In tup, the arrows go up. This is obviously true because it rhymes."
RossBencina 10y ago
I'd not heard of tup, thought I'd try it out on Windows. Unfortunately I hit a bug straight away: Tup is not directly compatible with MSVC 2015 (without disabling VCToolsTelemetry.dat generation in the registry) [0]. I don't fancy adopting a tool that forces me to opt-out of being able to send compiler debug telemetry to Microsoft the next time I hit a compiler bug. There is a nice (but a bit dated, 2010) review here [1] which discusses some other features and shortcomings. [0] https://github.com/gittup/tup/issues/182 https://github.com/gittup/tup/issues/182 [1] 2010: https://chadaustin.me/2010/06/scalable-build-systems-an-analysis-of-tup/ https://chadaustin.me/2010/06/scalable-build-systems-an-anal...
- deleted 10y ago
  [deleted]
RossBencina 10y ago
Previous HN discussion from Nov 1, 2014: https://news.ycombinator.com/item?id=8539564 https://news.ycombinator.com/item?id=8539564
ccostes 10y ago
Been using tup in production for about a year now and absolutely love it. The speed is nice, but compared to make projects where you need to 'make clean' to be sure everything gets properly rebuilt the confidence that tup will do the right thing every time is fantastic.
gravypod 10y ago
Is tup capable of building out of tree? By this I mean having: project_1/src/main.c project_1/src/something/a.c project_1/src/something/a.h Is there some way for tup to manage discovering the files to build and everything else needed or will I need to add every file path in manually like make?
- codyps 10y ago
  regarding "out of tree": I'm not quite sure about your explanation here (just looks like a list of source files), but presuming you mean "creates output files in a seperate directory from source", it doesn't really have complete support for that. You can use "variants" to place output files in a subdirectory of the source tree, though. > "some way for tup to manage discovering the files to build" Well, no. It's not a "convention" build tool like rust's `cargo` where you just place things in the default locations and it figures it out. You can use the `run ./script args` mechanism in tup to run your own script that emits tup rules, though. The manual has details: http://gittup.org/tup/manual.html http://gittup.org/tup/manual.html
  - gravypod 10y ago
    By out of tree I mean discover all the source files from the file tree.
bluejekyll 10y ago
I didn't see any examples of .phony type rules. Can tup do this? I've recently found myself returning to make for multi-build system orchestration, e.g. Rust and C and PHP libraries. Does anyone have examples of using tup for this type of thing? Also, I've found myself really enjoying declarative build systems more, e.g. Cargo or Maven. It seems like for C there could be a simple set of standard tup files that are run by a tool like Cargo over a standard tree layout. I didn't notice this in there, but could see a simple wrapper to tup to give this experience to almost any language. In fact, maybe using Cargo as a base and adding tup as a supported src type or something through a Cargo extension. It would probably need to be a default for the entire project for sanities sake.
- DSMan195276 10y ago
  I heard about tup a while back and finally attempted to look into it today and try replacing some of my Makefile's with Tupfile's. Unfortunately my googling and researching all seem to indicate that tup simply doesn't support any type of .PHONY targets. To that end, `tup` also doesn't seem to support even "basic" stuff (from my POV) like a `clean` and `install` target - which is fine, except that without .PHONY targets you can't add support for such things in your Tupfile's. Even the tup project's Tupfile doesn't support any way of directly installing it, I had to just copy the files by hand. IMO that's just a killer for me, which stinks because I don't really feel like such a feature should be that complicated, but the developers seem to take the stance that phony targets, and `clean` and `install` are unnecessary.
  - Nullabillity 10y ago
    Not providing a .PHONY equivalent makes sense to me: why not just turn those into separate shell scripts? Clean and install seem like special cases though, the build system already knows what to clean up (or, at least, Tup seems to), and install should be handled by a proper package instead.
    - DSMan195276 10y ago
      Generally speaking a lot of .PHONY targets require information that the build system, and build system alone has. You would have to replicate that information in the scripts if you wanted them to work, which is at the very least a messy situation, and somewhat defeats part of the purpose of using tup in the first place. This is obvious for targets like `clean` - hence why you suggest it should be supported in the build system - but most phony's require a certain amount of information from the build system. I also disagree on `install`. `install` is useful for creating packages in the first place. The build system already knows (or is told) which files are suppose to go in which general locations (configured via environment variables for the paths) and then install the correct files for your configuration. It is easy enough to use this to create a package by installing into a separate directory then /. The key is that the build system knows all the information to make `install` work as wanted (Along with being provided some paths). Separate scripts would not. Providing nothing at all to do for an `install` like the tup project does is simply not user-friendly. I install packages directly to `/usr/local/` all the time, and even ignoring that use-case there's no reason to leave the user wondering "Where do I put what files?". The developer know this information, the person attempting to use it or make a package generally doesn't (Or doesn't know all of them). Allowing you to define such a 'command' in the build system makes it a lot easier to write, and providing such a command makes the entire project tons easier to use and make into a package.
vq 10y ago
How does it handle building from LaTeX sources where you need to "rebuild" the document multiple times to get page numbers and references right?
- perlgeek 10y ago
  IIRC at least miktex came with a script that runs latex as often as needed until cross-references are resolved and the output is stable. Instead of plugging latex directly into a Makefile or tup, use such a wrapper. It seems that latexmk could be such a script: http://mirror.unl.edu/ctan/support/latexmk/README http://mirror.unl.edu/ctan/support/latexmk/README
- shoover 10y ago
  I don't see how that could work in tup or any deterministic build system. If there is any possible way to parameterize explicit filenames unique to the stages, that will map better to tup. It requires you to be explicit about every file generated, even temp files. I've searched the mailing list for a few scenarios and the response is often to wrap tup in a makefile.
sriram_sun 10y ago
Checkout ninja-build. Generate ninja build files automatically using CMake. Ninja is really fast.
- rspeer 10y ago
  Ninja is definitely the best and simplest build system I've used. A great example of the "do one thing and do it well" philosophy. The drawback comes from the simplicity: all dependencies have to be explicitly listed, which you probably don't want to do by hand, so you need some other tool to make the dependency list. I haven't seen how CMake works for this purpose, and I'd like to know more. What in particular do you do with CMake to use it with Ninja?
  - anderspitman 10y ago
    cmake -GNinja ..
- KKKKkkkk1 10y ago
  I can understand why tup would be faster than make for certain projects - when you have many targets and intermediates, it's more efficient to probe only the source files. But with ninja, it's not clear why would it be faster than make. I can see how a simpler file format makes processing the build faster, but ultimately ninja seems to be generating the same amount of IO as make.
  - iainmerrick 10y ago
    It does seem like it should be the same as make, but Ninja really is faster -- it just has close to zero cruft, whereas make is absolutely full of cruft. I use Ninja to build a medium-sized C/C++/ObjC project, around 1000 source files, and the dependency scan only takes a second or two. After that, it runs as fast as the compiler and linker are able. Edit to clarify: there might not be much speed difference for a full build, but incremental builds are much faster in Ninja.
    - 10y ago
sama1 10y ago
Broncos vs Buccaneers Live Stream Free http://animarathon.com/node/14821 http://animarathon.com/node/14821
- sama1 10y ago
  https://voat.co/v/pennstatevmichigan/comments/1321539 https://voat.co/v/pennstatevmichigan/comments/1321539
bleair 10y ago
I may have missed how but.. Cmake solves the problem of "locate the library FOO of version X.Y, add the compilation flags, link flags, include folder, link folder, static link options, dynamic link options" and all the other details needed to make use of another software component. Sometimes that component is found in my operating systems "default" spot and other times it's in an install directory that I explicitly input. How do I tell Tup to find these components/libraries and then have Tup also add in everything needed for all the commands related to building things that use that component. Also, I often have very different components going into different build targets that my project makes. How do the rules chain and build. In other words, just because I link one of my libraries with libssl doesn't mean I want every single source file and library I create in my project to then be linked with ssl
- aseipp 10y ago
  > How do I tell Tup to find these components/libraries and then have Tup also add in everything needed for all the commands related to building things that use that component. You don't do that, because it's not really the job of Tup to do that. It's better to think of it as an alternative to Make as opposed to CMake, really. Because CMake is more like a generic build system (which is 'compiled' to a variety of other systems), with a billion built in rules and utilities and libraries for making the common cases easy amongst all them. Tup is really just not in the same design space, although it is still a build tool. It'd probably be more appropriate to think of Tup as a thing that CMake would target, like Makefiles, MSVC Projects, or Ninja build files.
- bluejekyll 10y ago
  > just because I link one of my libraries with libssl doesn't mean I want every single source file and library I create in my project to then be linked with ssl Is this because you want to guarantee that all libraries have the dependencies explicitly defined? If nothing in that library is used, I don't think it has an effect on the output. Honestly just curious, bc it sounds like a pain to declare different sets of libraries inside the same project.
toolslive 10y ago
We've been using it for a few years. It's great. The only issue we ever had was when we tried it inside a docker container. It's related to fuse. https://github.com/docker/docker/issues/1916 https://github.com/docker/docker/issues/1916
asgardiator 10y ago
> tup, transitive verb: To have sex with. https://en.wiktionary.org/wiki/tup https://en.wiktionary.org/wiki/tup
- aidenn0 10y ago
  It's an archaic britishism. The only time I've ever seen it actually used is in the works of Morgan Howell.
  - nevhan 10y ago
    Shakespeare uses it in Othello, learnt something in 9th grade English!
  - jonathonf 10y ago
    Not all that archaic. Male sheep kept, or just added to a field, for breeding are known as tups (which are distinct to rams).
aidenn0 10y ago
My $0.02 on Tup: First of all, I cannot express how much more I like it than make. If Tup is an option, I will use it. What it does well: 1) It prevents you from making dependency mistakes: it hooks into the FS layer using fuse and tracks all input and output files that are inside your build directory. If you make any mistakes that could cause a future incremental build to be improper, it errors out rather than continuing. 2) It is opinionated about how your project should be structured. This has some negatives if you are trying to duplicate a particular structure from Make, but all in all does guide you in the right direction. 3) There isn't a lot of syntax to learn. This is good because the syntax is very different from anything else I've used. #1 is really the killer feature for me; the amount of time I want to spend debugging makefiles is just slightly less than zero. What it doesn't do, but I'm not bothered by: Tup literally only manages commands that have 1 or more inputs and one or more outputs, and which must be run IFF the inputs have changed or the outputs do not exist; the outputs must be within the hierarchy of the project. 1) Configuration must be done before Tup is launched 2) Anything you might use a .PHONY for in make needs to be done outside of tup 3) Install commands must be done outside of tup. This means that configuring, installing &c. must be done outside of tup. I find that having a make file that handles the above 3 steps works fine; others using tup tend to use a shell script. What it doesn't do that I wish it did: 1) No clean command; I currently work around this by having it generate .gitignore file and git clean -X; still it's annoying that this isn't possible. 2) It does not handle paths with spaces. This is actually safe as it enforces relative paths, so if it works on your system it should work everywhere even if the project is unpacked to a path with spaces.
xwvvvvwx 10y ago
How does this compare in speed to cmake + ninja?

169 points by mynameislegion 10y ago

Damn this is quick. Also, the URL made me think it was going to be on so.e GitHub competitor called gittup, heh.

wreft 10y ago

I first saw Tup a few years ago as it seemed to be the preferred way to filewatch/automagically-transpire moonscript to lua.

deleted 10y ago

[deleted]

jpgvm 10y ago

We use Tup to build Flynn [1], it's a pretty neat build system. The only real drawback is that it's hard to get to work on some operating systems because of dependency on FUSE. That said it still beats the crap out GNU Make or CMake etc. [1] https://github.com/flynn/flynn https://github.com/flynn/flynn

falcolas 10y ago
> That said it still beats the crap out GNU Make How? Their description was useless in describing why it's "obviously" so much better.
- nerdponx 10y ago
  > See the difference? The arrows go up. This makes it very fast. I don't think I've seen a more intelligence-insulting description of a product.
  - chrismonsanto 10y ago
    it's a joke...
  - todd8 10y ago
    Oh, you didn't read far enough; the description is written with a sense of humor, essentially parodying the very "intelligence-insulting descriptions" that one sees frequently on the internet.
  - exDM69 10y ago
    Well that was a bit light hearted but that's highlights the big difference: tup works bottom up, make works top down. The paper linked there provides a more thorough explanation.
- mynameislegion 10y ago
  The very last part of this page explains it quite well: http://gittup.org/tup/ex_a_first_tupfile.html http://gittup.org/tup/ex_a_first_tupfile.html
- adekok 10y ago
  GNU make starts of with targets, and then talks the build graph (target -> source) to find source files which will re-build the targets. And GNU Make has to scan the files. Tup starts off with the list of sources which have changed, and then walks the build graph (source -> target) to find out which targets need to be rebuilt. And Tup gets notified of file changes. So where Make scans all targets and all source files, Tup gets told which source file has changed, and then rebuilds only the necessary targets. That being said, the following comparison is a bit odd: http://gittup.org/tup/make_vs_tup.html http://gittup.org/tup/make_vs_tup.html I don't think it's a requirement of GNU Make that building a project with N source files takes exponential time. I think its just that the GNU Make code is terrible.
  - LukeShu 10y ago
    GNU Make doesn't take exponential time, it takes ~linear time. The Y value on the graphs is exponential, because the input is growing exponentially. The graph is just rubbing in that a noop Tup build is O(1) instead of Make's O(n) by showing crazy large n. Which is certainly great on Tup's part![1] The Tup author's description of Tup's algorithm in comparison has always rubbed me the wrong way, though. It isn't some entirely new generation of build system like they present in the Tup paper. It's just caching stat(2) between iterations[2]. The build naturally flows down like in the Make diagram on the main Tup page; but with Tup we cache the status of each node, and become informed when one of them becomes invalid; then the upward flowing arrows of the Tup diagram is the cache-invalidation of the lower nodes flowing up to the higher nodes. This is the totally natural way of implementing caching if we were to add that to GNU Make. So why don't GNU Make and other systems cache the DAG of stat(2) calls? As the saying says, cache invalidation is hard. Tup gets away with caching by receiving an a priori list of filesystem changes, so it knows which nodes to invalidate; other build systems have to scan the filesystem for changes. Tup gets this list of changes by having a FUSE filesystem sit between the user and the actual filesystem, logging all changes. While this is great for many users, it isn't quite general. It will break if we edit anything "offline", perhaps on a thumb drive on another computer. It will break in a number of different NFS setups. I don't mean to say that the caching doesn't have value, but it isn't the silver bullet that the Tup author makes it out to be. [1]: In the Tup paper, the author gives some fancy O(...) expressions. These are... odd; they're kind of hand-wavey, as they are based on expected fan-out of directories, and don't really give a good apples-to-apples comparison. They certainly are valuable to analyze, but are pretty misleading if you aren't critical. [2]: OK, that's not strictly true. The "Beta build system" (the next generation of build systems that Tup is supposedly part of) algorithm also has the feature that it will automatically remove a generated file that was part of the DAG, but isn't anymore. This is a nifty feature that is made possible by the caching; but IMO it doesn't constitute a core change.

piyush_soni 10y ago

To the authors : It would be really great if you could compare the SCons build system to Tup (with some numbers) - so that I can convince my managers to switch :).

agumonkey 10y ago

I can't recall what exactly but I hit expressiveness problems with tup (after proper RTFM). Probably some self referential issue. For the use case listed it's indeed very nice and very fast.

qwertyuiop924 10y ago

The FUSE dependancy is pretty unfortunate. Is there any way to get rid of it? Other than that, it's pretty cool. And the creator clearly has a sense of humor, something which is far rarer than it should be.

adekok 10y ago
The same functionality should be available in various inotify / dnotify implementations.
- majewsky 10y ago
  Not really. inotify requires you to set up a watch on every single file or directory that you want to watch. To see how this escalates, install the inotify-tools and do inotifywatch -d /path/to/directory/with/a/lot/of/files
  - qwertyuiop924 10y ago
    What about kqueue or eventports?
    - krakensden 10y ago
      Kqueue and OS X's file watcher thing are worse than inotify. They give inexact reports of changes, so you have to do a bunch of manual scanning afterwards. I really don't understand the grandparent's gripe. Inotify scales well, supports race free "watch a whole directory tree", and has a nice API.
      - codyps 10y ago
        There is a per-user limit to the number of inotify handles available (max_user_watches) and the default value is 8192. The limit exists because there is a ~1KiB kernel memory overhead per watch (though there should really be a way for them to take part in normal memory accounting per-process). If one wants to watch a directory tree, one needs an inotify watch handle per subdirectory in that tree. On large trees (or if more than 1 process is using inotify), that number of watches can be exceeded. As lots of folks are looking for recursive watches, they aren't happy with needing to allocate & manage a bunch of handles when they see what they want as a single item. That said, I'm not sure the way the kernel thinks about fs notifications internally would allow a single handle recursive watch at the moment. In any case, the amount of info one can obtain by using fuse (or any fs like nfs or 9p) to intercept filesystem accesses is a bit larger. At the very least, one can (in most cases) directly observe the ranges of the file that were modified (though that's not quite so important for tup, afaik). There also aren't any queue overruns (which can happen in inotify) because one will just slow the filesystem operations down instead (whether this is desirable or not depends on the application).

ggambetta 10y ago

While I can believe the claim it's better than make, it would be way more interesting to see how it compares to Bazel. After using it (or rather Blaze) at Google and now Bazel at Improbable, I consider it the gold standard in build tools. If anything, I wish Google would open source the rest of the build "ecosystem" that together with Blaze let you build the whole codebase in seconds. It was pretty amazing.

simonhorlick 10y ago
I've also been using Bazel for a couple of months now and it's fantastic. Simple, declarative syntax, blazingly fast. I can't recommend it highly enough.
Tojot 10y ago
I'm a Bazel person myself as well, but tup with the lua syntax might be getting somewhere.

adekok 10y ago

Tup's main problem is it's unusual, and it doesn't have a library of build rules. But it's fast! On a related note, I've always wondered if it was possible to have a build system based on dynamic library injection / strace. The idea would be that you just write your build rules in shell script. Then, you run it with a special shell that catches open(), etc. in child processes (via library injection, etc). These system calls get tracked, and stored in a special build table. One that you don't have to edit. Then, when you want to run the build again, you just re-run the magic shell. It catches the various commands, and checks their inputs / outputs, and then skips running the command if the targets are up to date. e.g. $(CC) -c foo.c -o foo.o Hmm... "foo.o" is up to date with "foo.c", so I don't need to run the compiler. I just return "success!" That would get rid of all magic build systems. All build syntax. All dependency ordering. The build system would just take care of it itself. I've played with this before, enough to note that it's likely possible. But I haven't got far enough to publish it.

exDM69 10y ago
There is such a build system, but I can't remember the name right now. It tracks system calls to see every file opened by the compiler to produce exact dependency graphs (assuming compiler is deterministic). The downside is that it's Linux only. If anyone remembers the name, please do share.
- ghkbrew 10y ago
  I don't know if it's the one you're thinking of, but fabricate.py does basically that.
- artisdom 10y ago
  https://github.com/rizsotto/Bear/blob/master/README.md https://github.com/rizsotto/Bear/blob/master/README.md Bear do something similar.
- rwmj 10y ago
  I wrote something called auto-buildrequires which lets you figure out build dependencies automatically, and works roughly as you describe. https://people.redhat.com/~rjones/auto-buildrequires/ https://people.redhat.com/~rjones/auto-buildrequires/

brudgers 10y ago

The related linux distribution, Gittup: http://gittup.org/gittup/ http://gittup.org/gittup/

brudgers 10y ago

The related linux distribution, Gittup: http://gittup.org/gittup/ http://gittup.org/gittup/

todd8 10y ago

This reminds me of DJB's ideas for a build system, redo [1]. However, it never seemed to gain any traction. (or did it? [2]) [1] http://cr.yp.to/redo.html http://cr.yp.to/redo.html [2] http://apenwarr.ca/log/?m=201012#14 http://apenwarr.ca/log/?m=201012#14

Tharre 10y ago
Well, if you want to see redo in action, there are plenty of implementations available: [0] [0] http://news.dieweltistgarnichtso.net/bin/redo-sh.html http://news.dieweltistgarnichtso.net/bin/redo-sh.html
sigil 10y ago
I can recommend apenwarr's redo implementation [0]. There's occasional activity on the mailing list [1], which leads me to believe people are using it, but not promoting it much. The apenwarr implementation includes a full Python implementation as well as a minimal version in < 200 lines of sh. The minimal version doesn't support incremental rebuilds -- the out-of-date-ness tracking in the full Python version uses sqlite -- but it's good for understanding the redo concept. Also good for embedded contexts. I used the full redo implementation for some data processing tasks once, with mixed results. It was a situation where I couldn't declare the dependency graph up front. With redo, each target declares its dependencies locally when it builds, and redo assembles and tracks the dynamic dependency graph. It's pretty neat, but become difficult to reason about and debug. Could be that I never got comfortable with the new paradigm, or could be that essential tooling was missing, not sure. I still think redo is promising. Anyway after a decade of messing with shiny new build tools, I finally learned to stop worrying and love the bomb (make). It's weird and warty but surprisingly capable. Worth the learning investment. Oh and jgrahamc's "GNU Make Book" is great. [2] [0] https://github.com/apenwarr/redo https://github.com/apenwarr/redo [1] https://groups.google.com/forum/#!forum/redo-list https://groups.google.com/forum/#!forum/redo-list [2] https://www.nostarch.com/gnumake https://www.nostarch.com/gnumake

ricardobeat 10y ago

The index page makes it sound like a parody of something, took me a while to figure it is functional software.

ubertaco 10y ago
I do love the quirky sense of humor though: "In a typical build system, the dependency arrows go down. Although this is the way they would naturally go due to gravity, it is unfortunately also where the enemy's gate is. This makes it very inefficient and unfriendly. In tup, the arrows go up. This is obviously true because it rhymes."

RossBencina 10y ago

I'd not heard of tup, thought I'd try it out on Windows. Unfortunately I hit a bug straight away: Tup is not directly compatible with MSVC 2015 (without disabling VCToolsTelemetry.dat generation in the registry) [0]. I don't fancy adopting a tool that forces me to opt-out of being able to send compiler debug telemetry to Microsoft the next time I hit a compiler bug. There is a nice (but a bit dated, 2010) review here [1] which discusses some other features and shortcomings. [0] https://github.com/gittup/tup/issues/182 https://github.com/gittup/tup/issues/182 [1] 2010: https://chadaustin.me/2010/06/scalable-build-systems-an-analysis-of-tup/ https://chadaustin.me/2010/06/scalable-build-systems-an-anal...

deleted 10y ago
[deleted]

RossBencina 10y ago

Previous HN discussion from Nov 1, 2014: https://news.ycombinator.com/item?id=8539564 https://news.ycombinator.com/item?id=8539564

ccostes 10y ago

Been using tup in production for about a year now and absolutely love it. The speed is nice, but compared to make projects where you need to 'make clean' to be sure everything gets properly rebuilt the confidence that tup will do the right thing every time is fantastic.

gravypod 10y ago

Is tup capable of building out of tree? By this I mean having: project_1/src/main.c project_1/src/something/a.c project_1/src/something/a.h Is there some way for tup to manage discovering the files to build and everything else needed or will I need to add every file path in manually like make?

codyps 10y ago
regarding "out of tree": I'm not quite sure about your explanation here (just looks like a list of source files), but presuming you mean "creates output files in a seperate directory from source", it doesn't really have complete support for that. You can use "variants" to place output files in a subdirectory of the source tree, though. > "some way for tup to manage discovering the files to build" Well, no. It's not a "convention" build tool like rust's `cargo` where you just place things in the default locations and it figures it out. You can use the `run ./script args` mechanism in tup to run your own script that emits tup rules, though. The manual has details: http://gittup.org/tup/manual.html http://gittup.org/tup/manual.html
- gravypod 10y ago
  By out of tree I mean discover all the source files from the file tree.

bluejekyll 10y ago

I didn't see any examples of .phony type rules. Can tup do this? I've recently found myself returning to make for multi-build system orchestration, e.g. Rust and C and PHP libraries. Does anyone have examples of using tup for this type of thing? Also, I've found myself really enjoying declarative build systems more, e.g. Cargo or Maven. It seems like for C there could be a simple set of standard tup files that are run by a tool like Cargo over a standard tree layout. I didn't notice this in there, but could see a simple wrapper to tup to give this experience to almost any language. In fact, maybe using Cargo as a base and adding tup as a supported src type or something through a Cargo extension. It would probably need to be a default for the entire project for sanities sake.

DSMan195276 10y ago
I heard about tup a while back and finally attempted to look into it today and try replacing some of my Makefile's with Tupfile's. Unfortunately my googling and researching all seem to indicate that tup simply doesn't support any type of .PHONY targets. To that end, `tup` also doesn't seem to support even "basic" stuff (from my POV) like a `clean` and `install` target - which is fine, except that without .PHONY targets you can't add support for such things in your Tupfile's. Even the tup project's Tupfile doesn't support any way of directly installing it, I had to just copy the files by hand. IMO that's just a killer for me, which stinks because I don't really feel like such a feature should be that complicated, but the developers seem to take the stance that phony targets, and `clean` and `install` are unnecessary.
- Nullabillity 10y ago
  Not providing a .PHONY equivalent makes sense to me: why not just turn those into separate shell scripts? Clean and install seem like special cases though, the build system already knows what to clean up (or, at least, Tup seems to), and install should be handled by a proper package instead.
  - DSMan195276 10y ago
    Generally speaking a lot of .PHONY targets require information that the build system, and build system alone has. You would have to replicate that information in the scripts if you wanted them to work, which is at the very least a messy situation, and somewhat defeats part of the purpose of using tup in the first place. This is obvious for targets like `clean` - hence why you suggest it should be supported in the build system - but most phony's require a certain amount of information from the build system. I also disagree on `install`. `install` is useful for creating packages in the first place. The build system already knows (or is told) which files are suppose to go in which general locations (configured via environment variables for the paths) and then install the correct files for your configuration. It is easy enough to use this to create a package by installing into a separate directory then /. The key is that the build system knows all the information to make `install` work as wanted (Along with being provided some paths). Separate scripts would not. Providing nothing at all to do for an `install` like the tup project does is simply not user-friendly. I install packages directly to `/usr/local/` all the time, and even ignoring that use-case there's no reason to leave the user wondering "Where do I put what files?". The developer know this information, the person attempting to use it or make a package generally doesn't (Or doesn't know all of them). Allowing you to define such a 'command' in the build system makes it a lot easier to write, and providing such a command makes the entire project tons easier to use and make into a package.

vq 10y ago

How does it handle building from LaTeX sources where you need to "rebuild" the document multiple times to get page numbers and references right?

perlgeek 10y ago
IIRC at least miktex came with a script that runs latex as often as needed until cross-references are resolved and the output is stable. Instead of plugging latex directly into a Makefile or tup, use such a wrapper. It seems that latexmk could be such a script: http://mirror.unl.edu/ctan/support/latexmk/README http://mirror.unl.edu/ctan/support/latexmk/README
shoover 10y ago
I don't see how that could work in tup or any deterministic build system. If there is any possible way to parameterize explicit filenames unique to the stages, that will map better to tup. It requires you to be explicit about every file generated, even temp files. I've searched the mailing list for a few scenarios and the response is often to wrap tup in a makefile.

sriram_sun 10y ago

Checkout ninja-build. Generate ninja build files automatically using CMake. Ninja is really fast.

rspeer 10y ago
Ninja is definitely the best and simplest build system I've used. A great example of the "do one thing and do it well" philosophy. The drawback comes from the simplicity: all dependencies have to be explicitly listed, which you probably don't want to do by hand, so you need some other tool to make the dependency list. I haven't seen how CMake works for this purpose, and I'd like to know more. What in particular do you do with CMake to use it with Ninja?
- anderspitman 10y ago
  cmake -GNinja ..
KKKKkkkk1 10y ago
I can understand why tup would be faster than make for certain projects - when you have many targets and intermediates, it's more efficient to probe only the source files. But with ninja, it's not clear why would it be faster than make. I can see how a simpler file format makes processing the build faster, but ultimately ninja seems to be generating the same amount of IO as make.
- iainmerrick 10y ago
  It does seem like it should be the same as make, but Ninja really is faster -- it just has close to zero cruft, whereas make is absolutely full of cruft. I use Ninja to build a medium-sized C/C++/ObjC project, around 1000 source files, and the dependency scan only takes a second or two. After that, it runs as fast as the compiler and linker are able. Edit to clarify: there might not be much speed difference for a full build, but incremental builds are much faster in Ninja.
  - 10y ago

sama1 10y ago

Broncos vs Buccaneers Live Stream Free http://animarathon.com/node/14821 http://animarathon.com/node/14821

sama1 10y ago
https://voat.co/v/pennstatevmichigan/comments/1321539 https://voat.co/v/pennstatevmichigan/comments/1321539

bleair 10y ago

I may have missed how but.. Cmake solves the problem of "locate the library FOO of version X.Y, add the compilation flags, link flags, include folder, link folder, static link options, dynamic link options" and all the other details needed to make use of another software component. Sometimes that component is found in my operating systems "default" spot and other times it's in an install directory that I explicitly input. How do I tell Tup to find these components/libraries and then have Tup also add in everything needed for all the commands related to building things that use that component. Also, I often have very different components going into different build targets that my project makes. How do the rules chain and build. In other words, just because I link one of my libraries with libssl doesn't mean I want every single source file and library I create in my project to then be linked with ssl

aseipp 10y ago
> How do I tell Tup to find these components/libraries and then have Tup also add in everything needed for all the commands related to building things that use that component. You don't do that, because it's not really the job of Tup to do that. It's better to think of it as an alternative to Make as opposed to CMake, really. Because CMake is more like a generic build system (which is 'compiled' to a variety of other systems), with a billion built in rules and utilities and libraries for making the common cases easy amongst all them. Tup is really just not in the same design space, although it is still a build tool. It'd probably be more appropriate to think of Tup as a thing that CMake would target, like Makefiles, MSVC Projects, or Ninja build files.
bluejekyll 10y ago
> just because I link one of my libraries with libssl doesn't mean I want every single source file and library I create in my project to then be linked with ssl Is this because you want to guarantee that all libraries have the dependencies explicitly defined? If nothing in that library is used, I don't think it has an effect on the output. Honestly just curious, bc it sounds like a pain to declare different sets of libraries inside the same project.

toolslive 10y ago

We've been using it for a few years. It's great. The only issue we ever had was when we tried it inside a docker container. It's related to fuse. https://github.com/docker/docker/issues/1916 https://github.com/docker/docker/issues/1916

asgardiator 10y ago

> tup, transitive verb: To have sex with. https://en.wiktionary.org/wiki/tup https://en.wiktionary.org/wiki/tup

aidenn0 10y ago
It's an archaic britishism. The only time I've ever seen it actually used is in the works of Morgan Howell.
- nevhan 10y ago
  Shakespeare uses it in Othello, learnt something in 9th grade English!
- jonathonf 10y ago
  Not all that archaic. Male sheep kept, or just added to a field, for breeding are known as tups (which are distinct to rams).

aidenn0 10y ago

My $0.02 on Tup: First of all, I cannot express how much more I like it than make. If Tup is an option, I will use it. What it does well: 1) It prevents you from making dependency mistakes: it hooks into the FS layer using fuse and tracks all input and output files that are inside your build directory. If you make any mistakes that could cause a future incremental build to be improper, it errors out rather than continuing. 2) It is opinionated about how your project should be structured. This has some negatives if you are trying to duplicate a particular structure from Make, but all in all does guide you in the right direction. 3) There isn't a lot of syntax to learn. This is good because the syntax is very different from anything else I've used. #1 is really the killer feature for me; the amount of time I want to spend debugging makefiles is just slightly less than zero. What it doesn't do, but I'm not bothered by: Tup literally only manages commands that have 1 or more inputs and one or more outputs, and which must be run IFF the inputs have changed or the outputs do not exist; the outputs must be within the hierarchy of the project. 1) Configuration must be done before Tup is launched 2) Anything you might use a .PHONY for in make needs to be done outside of tup 3) Install commands must be done outside of tup. This means that configuring, installing &c. must be done outside of tup. I find that having a make file that handles the above 3 steps works fine; others using tup tend to use a shell script. What it doesn't do that I wish it did: 1) No clean command; I currently work around this by having it generate .gitignore file and git clean -X; still it's annoying that this isn't possible. 2) It does not handle paths with spaces. This is actually safe as it enforces relative paths, so if it works on your system it should work everywhere even if the project is unpacked to a path with spaces.

xwvvvvwx 10y ago

How does this compare in speed to cmake + ninja?