8 ms·
What's in a Build Tool?
- xyzzy_plugh 11y agoRegarding make, the author states: Parallelize different commands: Okay: There's no automatic way to parallelize different targets, though you can still parallelize a single target using Bash. This is wrong, make has supported the -j flag since 1988: -j [jobs], --jobs[=jobs] Specifies the number of jobs (commands) to run simultaneously. If there is more than one -j option, the last one is effective. If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously.
- microcolonel 11y agoI can't believe anyone could use make in the last ~12 years without the -j flag. We've had dual cores at least for ages.
- Ace17 11y agoActually, I think make should default to parallel builds.
- spc476 11y agoIt only works if all the dependencies are specified correctly, otherwise, the make will fail.
- MereInterest 11y agoThe problem there is that it would break makefiles that do not handle their dependencies correctly. With non-parallel builds, the order of execution is deterministic, and so you can get away with sloppiness in your dependencies. If it works the first time that you test it, it will continue to work. I can understand why they would be hesitant to change the default, as they would rather that old, tested scripts continue to work without modification.
- hamstergene 11y agoIt will also not work with recursive make invocations, e.g. when top level makefile simply invokes makefiles of several independent projects.
- opk 11y agoThis actually works fine in GNU make too. The parent make acts as a job server. The implementation is ingenious, see http://make.mad-scientist.net/papers/jobserver-implementation/ http://make.mad-scientist.net/papers/jobserver-implementatio...
- _pmf_ 11y ago> The problem there is that it would break makefiles that do not handle their dependencies correctly. Serious question: In what way are Makefiles that are not specifying their dependencies correctly ever to be considered non-broken?
- pgeorgi 11y agoAs said: their behavior is still deterministic in non-parallel execution. And it looks like gnu make developers aren't like compiler developers that exploit any ambiguity as an excuse to really mess up somebody's day.
- dllthomas 11y agoOf course, they are broken. If dependencies are specified incorrectly, there will be files that, if modified, do not lead to correct rebuilds. But that doesn't mean there aren't stable workflows using those broken files that are not sensitive to the brokenness.
- MereInterest 11y agoIt's the exact same argument that comes up whenever gcc improves their optimization algorithms by exploiting undefined behavior, making some code no longer work. In both cases, the original code was fundamentally broken from the start, and the change in tooling only revealed the brokenness, not causing it. I would completely see such makefiles as being broken.
- oriolid 11y agoI have a co-worker who was told about -j a couple of weeks ago. Before that, he would bill half an hour for every time he did a clean build.
- herge 11y agoIt's a pity that rake is not included in the table at the bottom. One huge advantage of rake is the ability to easily debug what is happening. A quick `p [filename, task]` lets you know what is happening. Even, if needed, you can quickly do $ irb >>> require 'rake' >>> p FileList["foo/*"] and actually play around with the libraries. If you know anything about ruby, and especially if you are looking to build anything other than a c/c++ project, it is definitely worth looking at.
- djKianoosh 11y agoGradle is a glaring omission if maven, ant and sbt are included.
- jschwartzi 11y agoI'm curious what the author thinks of CMake. It reminded me a lot of Maven, but for C/C++ projects.
- codyps 11y agoThe author also doesn't cover autoconf/automake, which operates in the same vein as cmake (generating files for use by another build tool). I agree, it would be useful to evaluate it here as it provides a bunch of the features the other "build tools" provide.
- imron 11y agoFor c and c++, I've recently starting using meson. It's a breath of fresh air compared to some other build tools, and the config files are both expressive and concise.
- sa2ajj 11y agoFor those curious (like me): meson's home page -- http://mesonbuild.com/ http://mesonbuild.com/
- luckydude 11y agoI thought this was going to be a rant against autoconf. When that rant comes I'll post our 186 line shell script that does what autoconf does for pretty much any platform. Handles AIX, Solaris, FreeBSD, HP-UX, IRIX, Linux, NetBSD, OpenBSD, SCO, SunOS, Windows, MacOS. I hate autoconf but that's not part of this thread. I'll wait.
- yeowMeng 11y agoCan you please GNU me yr 186 lines of bytes. I am curious.
- qznc 11y agoWhy not write a preemptive blog post?
- Ace17 11y agoI'm curious about this shell script ; this looks to good to be true! Having automated the build of dozens of FOSS packages, I can definitely say that autoconf-based packages are _by far_ the easiest ones to build ; especially if you want to do funny stuff like out-of-tree builds or cross-compilation (can your 186-line script do this?). Some projects (x264, ffmpeg) provide 'configure'-like custom scripts, trying to imitate what autoconf does ; but in practise you can't automate their build the same way you did with autoconf-based projects.
- bhaak 11y agoI see where the hate for autoconf comes from. Properly done autoconf is very developer-friendly for building but if you have to maintain it, it can be a nightmare. OTOH, because you can always escape to shell scripts, I usually can fix any issue that crops up. With cmake, I regularly fail, e.g. configuring succeeds but building fails because of a missing library and because I actually have no clue what the standard approach should be (online documentation is abysmal with cmake).
- atilaneves 11y agoHave you tried CMake?
- codyps 11y agoninja? tup? Article doesn't note the property of being able to depend upon the entire command that generates an output (ie: re-generate when compiler flags change). This is something that makes doing reliable builds much easier (when present). It's notably very hard to do in make (and even then is very inefficient). Also, on "download" the author seems to presume that one takes the naive approach in each tool. In most cases, if one spends a bunch of time on it the downloads can be done fairly efficiently (especially in make, without even much work there). Most of these build systems are fully programmable, so the rating should probably focus more on the level of difficulty to do well (with some requirements specifying what "well" is)
- thelema314 11y agoTo depend on compiler flags, I do this: ## compiler_flags file keeps a copy of the compile flags $(builddir)/compiler_flags: force mkdir -p $(builddir) echo '$(CPPFLAGS) $(CFLAGS)' | cmp -s - $@ || echo '$(CPPFLAGS) $(CFLAGS)' > $@ ##rebuild all objects if compile flags change $(LIBOBJECTS) $(RTLLIBOBJECTS) $(OPTLIBOBJECTS) $(TESTOBJECTS) $(builddir)/init_qt_workdir: $(builddir)/compiler_flags I"m pretty happy with the results.
- Debilski 11y agoDepending on compiler flags was implicitly mentioned for sbt, though (point 8). Depending on the `.value` of the `scalaVersion` configuration property is the same thing and can be done for all other settings input as well.
- dlandis 11y agoAnt and Maven are fairly obsolete at this point in the Java world having been superseded by Gradle, and so aren't really worthy of much discussion. But Gradle is quite good and I think it would stack up very well against the others including sbt. Like sbt, I think it fuses a task based and configuration based approach and also does dependency management.
- aikah 11y agoIsn't Gradle based on and using Maven (under the hood) ?
- smadge 11y agoGradle allows you to 1) use maven plugins 2) download dependencies from maven repositories. So my understanding is that Gradle is backwards compatible but separate.
- smadge 11y agoYikes, I still use Maven. I don't like being left behind in the dust. Is there any compelling reason to prefer Gradle? What mistakes did Maven make that gradle avoided? One thing I am hesitant about is the I don't need my build tool to be a turing complete language.
- a_imho 11y agoTechnology radar [1] said 'Adopt' in 2013: "Language-based build tools like Gradle and Rake continue to offer finer-grained abstractions and more flexibility long term than XML and plug-in based tools like Ant and Maven. This allows them to grow gracefully as projects become more complex." My personal opinion after managing the build system of a fairly complex java project for years, I'm glad we skipped Maven in the evolution. For me the most important part was error messages, and the most helpful were Ant > Gradle > Maven. We had very different conventions than what Maven offered, and shaping it to our process was cumbersome. Coupled with Maven 2.x and 3.x never played nicely together (at least for me), now when I see a Maven build I just sigh (looking at you Atlassian) Finally, strictly personal observation, but Ant/Maven tend to correlate with SVN (or older VCS) usage, which you can argue still does its job, but is pretty much considered obsolete. [1] https://www.thoughtworks.com/radar/a-z#gradle https://www.thoughtworks.com/radar/a-z#gradle
- lasermike026 11y agoI was just writing a Rakefile and have a good time of it. I like using make but rake gives me more functionality.
- shoover 11y agoRake is a pleasure for small projects, like generating HTML from a handful of org-mode files or zipping up a Chrome extension. I've not tried to take it much farther than that, though there is a series of blog posts I can't find attached the moment in which the author used rake for real C projects with GCC.
- falsedan 11y agoI think the author does a great disservice lumping together deploy, test, and dev sandbox tools under 'build tools'. They are different scenarios which all happen to require a built copy of the code. There's no fundamental reason for them to use the same tool, except the inertial tug of whatever build tool your project happens to use.
- Too 11y agoAm i the only one thinking that doing a simple compile is something that should be handled fully by the compiler and not the build tool. The compiler should automatically detect which files are being #included and allow for incremental compile based on this. The Typescript and SASS compilers are good examples of this, you just invoke it like 'tsc main.ts --watch' and it will automatically know that main.ts included other files and these will also be watched. If your compiler does not support this, the only way to build a proper dependency tree is 1. if the compiler can generate one in a standard format (gcc -MMd), 2. your buildtool knows as much about the language as the compiler does, 3. you write the dependency tree manually. 1. Is bad because it requires the compiler to support generating a wide range of dependency-graph formats, or every build tool must support the make syntax. 2. Is bad because the build-tool becomes one massive monolith and there is no way they can reliably support every language in the world. 3. Is bad because humans are lazy and make mistakes. For the build tool, this leaves it's responsibility to doing things before and after compilation (generating/preprocessing code, copy assests, run tests, deploy to staging), which usually means invoking arbiatry and custom shell-commands, the order of this is always so application-specific that #3 is inevitable. If doing these things are complicated, your build tool sucks, and sadly i found most build tools to fall into this category because they try #2 and want to be the center of the universe. Even doing a simple thing such as copying a file requires you to learn a new awkard xml-syntax, install the "copy-file-plugin" or write your own copy-file-plugin in the awkward syntax. What i would want is somthing with the simplicity of Gulp or Make, but with a bit more widespread syntax and more batteries included for file-manipulation, like Python would be perfect. Rake looks promising, maybe it makes it worth learning Ruby?
- Ace17 11y agoIf the syntax is the only issue, maybe we should fork GNU make.
- lmm 11y agoDisagree. Compilers, like anything else, should be small composable libraries. Compilation and dependency parsing should perhaps be separate, and the same dependency-tracking library should be usable by the compiler, wider build tools, and IDEs or similar.
- davidpronk 11y agoAfter reading this article on NPM as a build tool I immediately gave up on Gulp and Grunt. Highly recommended! http://blog.keithcirkel.co.uk/how-to-use-npm-as-a-build-tool/ http://blog.keithcirkel.co.uk/how-to-use-npm-as-a-build-tool...
- manaskarekar 11y agoA nice summary of some build tools: https://bitbucket.org/scons/scons/wiki/SconsVsOtherBuildTools https://bitbucket.org/scons/scons/wiki/SconsVsOtherBuildTool...
- shoover 11y agoThis is helpful and uses a good matrix to compare the tools in question. I've been wanting to see a version of this for cross-language tools: Bazel et al, msbuild, scons, and do not forget the quiet but powerful waf.
- gkya 11y agoThe OP mistakes what a build tool is. A build tool generates the executable from sources. Use other tools for other tasks. I once wrote a tool for running commands on file system events, but I lost it. But for example I write files rather often, a build on each write would be annoying for me. It's a personal thing, so give me the build script and let me use it the way i like.