5 ms·
fabricate: The better "make". Finds dependencies automatically for any language.
- dinther 17y agoSpot on. A must have for every developer.
- benhoyt 17y agoWe've been using memoize to build stuff for a while now, but we needed something that worked under Windows too. So here's our "fabricate" build tool. It's BSD licensed -- enjoy!
- loup-vaillant 17y agoI don't get it. Does it intercept file accesses to determine the dependencies? Looks like magic.
- silentbicycle 17y agoYes, via access times and strace: "When this build stage ran, it checked these files, so remember them as dependencies." Not magic at all, and should automatically work for languages that haven't even been invented yet.
- joe_bleau 17y agoCan it tell (using access times under windows) that the files were modified by that stage of the build command, and not by another unrelated process? How does it know which drives and directories to scan for recently modified files? Scanning all of them would take a while.
- silentbicycle 17y agoIt uses strace, a Linux-specific file change notification hook. It's not available on BSD or (IIRC) OS X, though ktrace is, so it could conceivably be ported. (The page says something vague about cygwin having a comparable hook to strace. I haven't developed for cygwin, though.) It looks like it checks for files modified specifically by its subprocesses.
- modeless 17y agoIt is possible to do strace-like stuff on Windows too as evidenced by Process Monitor. http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx http://technet.microsoft.com/en-us/sysinternals/bb896645.asp... If scanning atimes is too slow or unreliable, perhaps fabricate could be updated to use procmon?
- benhoyt 17y agoHere's a version of strace for Windows: http://www.intellectualheaven.com/default.asp?BH=projects&H=strace.htm http://www.intellectualheaven.com/default.asp?BH=projects... I've hacked it to print the actual file names on CreateFile() calls, but it's failing because it doesn't recurse and trace sub-processes. It should be theoretically possible by hooking CreateProcess(), but it's probably hard.
- kragen 17y agoDoes it scan your entire filesystem for atimes so as not to miss the .h files over in /usr/include, and if not, how does it know which directories need scanning?
- benhoyt 17y agoNo, it defaults to recursively scanning the current directory, but you can easily override this if you want, eg: setup(dirs=['.', '../lib'])
- growt 17y agoI haven't tried fabricate yet, but I think the number of build-tools out there is a sure sign that something fundamental is broken.
- tvon 17y agoI think it's just the nature of the beast, and an example of a complex problem with a lot of different solutions.
- sp332 17y agoI think it's just down to the popularity of make. People who are starting new projects use make because "everybody's using it, it must be the best!" Then, when make becomes a pain in the butt, they write their own because they've already decided that there's nothing better out there.
- kscaldef 17y agoI wouldn't claim it's the full story, but IMO part of what's fundamentally broken is the phenomenon of single-language programmers. This leads to the belief that your entire tool chain needs to be written in a single language. So now every language has it's own build tool (or 2, or 3), and god help you if you have a multi-language codebase and need to have a unified build system.
- modeless 17y agoWhat's broken is the C preprocess/compile/link process. C needs to get a modern module system and ditch the antiquated preprocessor and linker.
- herge 17y agoNot to be confused with Fabric (http://www.nongnu.org/fab/ http://www.nongnu.org/fab/), also a very cool python tool.
- st3fan 17y ago"""For a start, fabricate won't say "* missing separator" if you use spaces instead of tabs.""" - No, but it will say "IndentationError: unindent does not match any outer indentation level" if you mess up Python's indentation :-)
- st3fan 17y agoI wish someone would pick up http://a-a-p.sourceforge.net/ http://a-a-p.sourceforge.net/ again. It is a great build tool written by the Vim author. I've used it to build complex projects in the past with a minimal amount of aap configuration. It is much less verbose than fabricate.
- silentbicycle 17y agoIt looks like they hardcoded gcc as the compiler in the example. What about trying to port a program to a platform without it (or that calls it differently), that needs extra compilation flags, etc.? A lot of the seemingly esoteric stuff in Makefiles is there to make porting feasible (and the rest is autogenerated by autoconf, automake, etc.). Not just which files an operation depends on, but e.g. the options used for building shared libraries correctly on this platform, paths for libraries, etc. While the build system will remember what it found as dependencies, it still needs to know what to find the first time. (It will note header files as dependencies, though.)
- pmorici 17y agolooks a lot like SCube only way more verbose for the simple case. http://code.google.com/p/scube/wiki/Example1 http://code.google.com/p/scube/wiki/Example1 The problem with this and other Python build tools is They are a huge pain to use on older systems that either don't have a recent enough version of Python installed or don't have Python installed at all. It's just not as portable as Make is in that respect.
- tsally 17y agoYou don't need to enter in dependencies with 'make' either (suffix rules or makedepend). Don't get me wrong, fabricate looks great, but let's not misrepresent the power of make in the right hands. From the docs: files.o : files.c defs.h buffer.h command.h cc -c files.c I don't know a programmer beyond the introductory level that still writes this in their make file. :-p
- silentbicycle 17y agoRight. Make already knows how to convert .c to .o, all you need to say is files.c: defs.h buffer.h command.h and there are tools such as makedepend (for C and OCaml, off the top of my head) that generate those listings automatically, so you just add this to the Makefile: include (autogenerated dependency listing file) While the mandatory tab in Make syntax is a genuine wart, make has been around long enough that there are several other tools that know how to work with it. (Now, having the scan for access time changes added as a warning to Make might not be a bad idea. "WARNING: You forgot this dependency")
- garnet7 17y ago> make has been around long enough that there are several other tools that know how to work with it. I'd prefer an improved from-scratch 2nd-generation tool instead of using one that simply wraps around a complicated 1st-gen one that I'd rather not touch. Make has more warts than just the tab characters. Lots of obscurely-named variables, for starters.
- Bluem00 17y agoYou might want to check out tup. It uses the dependency tree in a fundamentally different way from make, in order to improve build times. http://gittup.org/tup/ http://gittup.org/tup/
- silentbicycle 17y ago> [complicated tool ...] Lots of obscurely-named variables, for starters. Are you talking about make or just GNU make? The $< $* $$@ stuff comes from gmake. Make itself is pretty simple. Here's a good tutorial: http://www.freebsd.org/doc/en/books/pmake/ http://www.freebsd.org/doc/en/books/pmake/
- thingie 17y agoDo I miss something? That example is like three times longer than an equivalent Makefile would needed to be, and I need to know Python, obviously. For a trivial project, I guess that Make would be easier to use, and for something more complex, I think I would have to do quite a lot of programming, with Fabricate…
- st3fan 17y agoWhat misses in my opinion is 'build by convention'. Most things that you want to build have recipes that are mostly the same. You have some sources that you want to compile and then link against some libraries. Using some platform specific compiler and linker flags. So, instead of writing a program to do the build I would rather see a simple DSL that allows me to specify just the minimal amount of configuration to get things going. For example building the example on the fabricate wiki page, should not be more complicated than: :program programname : program.c utils.c (This is actually an a-a-p recipe that will do the same. Throw this in a main.aap file and you get everything that fabricate does, build, dependency checks, clean, etc.) I don't mind to script the build when things get more complex. But only then and only when there really is no simple built-in alternative. Fabricate seems to be more like ant in the Java world. Where you specify in detail what your build should do. Most people have agreed by now that it is a waste of time and instead use Maven, which completely works on simple conventions and standard rules.
- jimbokun 17y agoWhat's a-a-p?
- st3fan 17y agoSee http://a-a-p.sf.net http://a-a-p.sf.net
- benhoyt 17y agoWell, our experience says that as soon as you step outside that "minimal amount" you have to write all the build steps explicitly anyway, so we're specifically avoiding implicit rules. And we end up being explicit pretty quickly in all our projects -- maybe more than some folks because we're embedded developers and tend to need full control, but even so.
- gchpaco 17y agoThe central problem with these sorts of "make replacements" is they miss out on the whole thing that makes make useful. I care not a whit about implicit rules, and I would welcome a real language over GNU make's function apparatus (filenames with spaces in them tend to get split in the functions, due to the fundamentally macro style), but without pattern matching rules and a real dependency engine I won't even think once about it. I use both far, far too often, in almost every makefile I've ever written.
- bryhoyt 17y agogchpaco, could you give an example of a situation where you'd use a real dependency engine, which could not be solved (at least not easily) by using fabricate.py? I know such cases exist in theory, but have never run across a need, so I'm interested.
- gchpaco 17y agoSince fabricate makes Python, a full language available, it's not that there's things that cannot be expressed, it's really about ease. So I'll tell you about something I did when I was a grad student. I was using an up to date TeX install. My advisor wasn't. To placate him, I had to copy all of my packages, and all the packages they referred to, and so on down the line so that he could compile my papers. I did this entirely within make + some small shell scripts. The fact that I could write scanners for %.tex and %.sty and %.cls and have GNU make (obviously) automagically use them to build the dependency scripts I instructed it to include was immensely convenient. This is not impossible to do procedurally, but it was inconvenient, and the all-make solution had the attractive side that whenever there were bugfixes to the packages they got automatically included in what I was doing. A better example, come to think of it, is the typical LaTeX woe; you can't build the final .dvi until you have accurate page numbers, references, tables of contents, indexes. You get all of those by--building the .dvi. So there's a strange circular dependency here. Make doesn't deal with this especially well, although it can be done. Ant and SCons don't even try.
- sorbits 17y agoI have a machine with two quad-core CPUs, I want to run 8 parallel jobs at a time! Most of these make replacements start with design decisions (like using Python) that pretty much rule out ever reaching what make does best, and which is essential to get my build times improved by at least a factor of 6. Using a tool (like CMake) to generate your Makefiles is an excellent solution, heck, even a Python script that generated a Makefile and then ran make would be to prefer over this tool.
- vr 17y agoUsing Python does not prevent parallel execution of tasks. It's just the build tasks graph that gets created by a single process, but then the build system can use all the available CPUs to actually execute it. See e.g. -j flag to SCons http://www.scons.org/doc/production/HTML/scons-user.html#AEN2099 http://www.scons.org/doc/production/HTML/scons-user.html#AEN....
- sorbits 17y agoAs I said, generating a Makefile (i.e. the dependency graph) from Python would be fine, which conceptually sounds like what SCons do (from your comment). But look at the build file example on the fabricate page, it is a user supplied build file with a build function which calls a compile function which loops over sources. Having the fabricate system take input like that and do parallel execution is as hard as solving the halting problem.
- benhoyt 17y agoFair call. So far that hasn't been a problem for us -- but if you want to make it one, please donate us some cores! :-) Seriously, as vr said, Python's not stopping us. And I imagine it wouldn't be too hard to add code to spawn off a few background tasks based on the dependency list.