11 ms·

Falsehoods programmers believe about build systems (2012)

69 points by jnxx 6y ago

deleted 6y ago
[deleted]
sabujp 6y ago
this is from 2012 and some of the points haven't aged well (see blaze/bazel)
- RealityVoid 6y ago
  Not familiar with bazel. Can you expand a bit?
  - klodolph 6y ago
    There are a few things which are either enforced directly by Bazel (i.e., Bazel takes an opinion on how build systems should work and gives an error if you try to make it work otherwise). There are also a few assumptions which are now harmless, because Bazel will either check that your assumptions are correct, or make the build work as if those assumptions hold. For example, "Build graphs are acyclic." Bazel enforces this. "Every build step updates at least one file." If your build step doesn't produce output, Bazel will complain. "It's possible to tell the compiler which file to write its output to." "It's possible to tell the compiler which directory to write its output to." Bazel does this with sandboxes.
- carlmr 6y ago
  Which ones and why? Is blaze/bazel free of all these falsehoods. Did it manage to make new mistakes?
  - sabujp 6y ago
    I'm no blaze/bazel expert (hopefully someone will add their own thoughts), but: 1. build graphs are trees/acyclic : bazel enforces this, no cyclic dependencies allowed 2. Compilers will always modify the timestamps on every file they are expected to output : if a write happens why not update mtime? nanosecond precision, what year is it? 3. It's possible to tell the compiler which file to write its output to : makes no sense to me, we expect our compilers to be writing to the correct files in the correct directories. 4. It's possible to predict in advance which files the compiler will update : yes it is, this is how caching works again see (1) 5. It's possible to determine the dependencies of a target without building it : yes yes it's completely possible given (1) 6. Detecting changes via file hashes is never the right thing : Bazel i think uses a combination of hashes and stamps, but why would using hashes not work 99.99999999999999998% of the time? 7. Non-experts can reliably write portable shell script : lol, experts can't reliably write portable shell scripts either 8. Your build tool is a great opportunity to invent a whole new language: or just coopt pieces of one (python: starlark, skylark) 9. Said language does not need to be a full-featured programming language : yes, it doesn't (again .bzl files) 10. Said language should be based on textual expansion : not sure what the alternative to textual expansion (parsing?) is 11. Adding an Nth layer of textual expansion will fix the problems of the preceding N-1 layers : again how is this different from parsing
    - deleted 6y ago
      [deleted]
    - tzs 6y ago
      OT: when did HN start wrapping long lines in indented text?
      - SilasX 6y ago
        It's not because the lines are long but because they're prefixed by four spaces (and for anyone reading, please don't do that unless it's actual code).
        tzs 6y ago
        It's wrapping lines prefixed with two spaces now. The comment I just posted a couple minutes ago with indented shell input and output definitely used two spaces and it is wrapping on my phone and on my desktop if I narrow the window enough.
    - tzs 6y ago
      > 2. Compilers will always modify the timestamps on every file they are expected to output : if a write happens why not update mtime? nanosecond precision, what year is it? It has nanosecond precision, but on most systems it does not have nanosecond resolution. $ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify: Modify: 2020-07-18 21:37:56.421487075 +0000 Modify: 2020-07-18 21:37:56.421487075 +0000 $ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify: Modify: 2020-07-18 21:37:57.397502229 +0000 Modify: 2020-07-18 21:37:57.397502229 +0000 $ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify: Modify: 2020-07-18 21:37:58.273515840 +0000 Modify: 2020-07-18 21:37:58.277515902 +0000 $ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify: Modify: 2020-07-18 21:41:00.736348218 +0000 Modify: 2020-07-18 21:41:00.736348218 +0000 $ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify: Modify: 2020-07-18 21:41:01.708363288 +0000 Modify: 2020-07-18 21:41:01.708363288 +0000 $ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify: Modify: 2020-07-18 21:41:02.504375653 +0000 Modify: 2020-07-18 21:41:02.508375715 +0000 Note that the difference is either 0 or 0.004000062. This is on a Debian 9 system running on an Amazon Lightsail instance with an ext4 filesystem. Here's a quick and dirty script to play with this: C=0 R=0 while : do R=$((R+1)) touch a1; sleep $1; touch a2 T1=$(stat a1 | grep Modify | sed -e 's/.*://' | sed -e 's/ .*//') T2=$(stat a2 | grep Modify | sed -e 's/.*://' | sed -e 's/ .*//') if [ $T1 != $T2 ]; then dc <<HERE 20k $T2 $T1 - p HERE C=$((C+1)) if [ $C -eq 100 ]; then echo "$R runs" exit 0 fi fi done Output on the above system: $ bash t.sh 0.001 | sort | uniq -c 4 .004000061 83 .004000062 13 .004000063 1 222 runs and on a VMWare Fusion VM running Debian 8 on my Mac: $ bash t.sh 0.001 | sort | uniq -c 29 .003999599 44 .003999600 27 .003999601 1 323 runs
  - gravypod 6y ago
    These are some of my personal opinions from using, and violently advocating for, bazel: > Build graphs are trees. Build graphs are acyclic. Bazel uses some tricks/magic to hide outputs of build rules from other build rules. Bazel also prevents cyclic dependencies (A depends on B which depends on A). > Compilers will always modify the timestamps on every file they are expected to output. Source files a target depends on will be hashed. The operations applied to the files will also be hashed. > It's possible to determine the dependencies of a target without building it. Bazel forces people to explicitly define all deps that the source code uses. It uses sandboxing to hide unspecified deps from the source code. All of the definition of these rules is defined statically-ish (if you ignore WORKSPACE which technically applies to Blaze). > Targets do not depend on the rules used to build them. Bazel hashes the operations the rules will perform. > Targets depend on every rule in the whole build system. Bazel understands the full DAG and knows what each target depends on and what depends on each target. > Detecting changes via file hashes is always the right thing. Detecting changes via file hashes is never the right thing. It's a tradeoff that someone has to decide. Neither is right or wrong. Using a file content+metadata hash allows you gauntness that the build is always correct. > It's possible to narrow down the set of possibly-updated files to a small hand-enumerated set. As long as you use file hashing and are explicit about your inputs and outputs of every build rule, you can. > Nobody will ever want to rebuild a subset of the available dirty targets. You can build a specific target and it's deps at any time. Unrelated dependency trees are ignored. > stat is slow on modern filesystems. Anything is slow given a large enough scale. Any syscall is slow once you have millions of source files. > Non-experts can reliably write portable shell script. In bazel people interact with higher level rules. You don't need to write shell scripts unless you're implementing something that requires shell scripts. If you'd like you can also use an output of a target as a tool to do further compiling. For example: build protoc then use protoc to compile protos into libraries. Every build step updates at most one file. Every build step updates at least one file. It's possible to tell the compiler which file to write its output to. It's possible to tell the compiler which directory to write its output to. It's possible to predict in advance which files the compiler will update. > Your build tool is a great opportunity to invent a whole new language. Said language does not need to be a full-featured programming language. In particular, said language does not need a module system more sophisticated than #include. Said language should be based on textual expansion. Adding an Nth layer of textual expansion will fix the problems of the preceding N-1 layers. It is a great opportunity to implement a domain specific language as "building software" is a domain that we would like to define programs for. Bazel uses skylark, it's a "fully featured" programming language in that there's loops, conditionals, functions (called "macros"), and a module-ish system ("load()"). It does not operate on textual expansion. > It's totally OK to spend over four hours calculating how much of a 25-minute build you should do. Bazel's target analysis takes a while but for my scale of code base it doesn't take hours, it takes seconds. If you have a huge codebase then these "wasted" seconds would be earned back by it's effective caching and remote build execution. > All the code you will ever need to compile is written in precisely one language. Bazel is multilingual. > Everything lives in a single repository. What is a "repository"? It's really a file system. What is a file system? A fancy kv store. Could you build a KV store that could return the values for every key that is a file in your source tree? Probably. Bazel is built around a monorepo design but you could just as easily clone libraries into shared folders and executre bazel from above your project. You could also use remote repository rules. IMO you're better off just vendoring any source code you're writing in a folder in your repo so you can guarantee old versions of things are around when you need them and you never have a leftpad incident. > Files only ever get updated with timestamps by a single machine. Version control systems will always update the timestamp on a file. Version control systems will never update the timestamp on a file. Version control systems will never change the time to one earlier than the previous timestamp. File contents are check so this is not as much of an issue. > Programmers don't want a system for writing build scripts; they want a system for writing systems that write build scripts. I think people don't want systems like this because they've been burned. I've used a lot of makefile generators and they're all a really poor caching system. You essentially make cache invalidation part of your dev flow. AS a developer I don't want to think about that. I want to ask something to build me something and have it know when it needs to reload build definitions or code. I also don't want to care about what specific "gcc" or "javac" commands are being run. I need to be able to go down to that level but 99% of the time I just need to say "Turn a.c, b.c, and c.c into foobar.bin". As for what bazel is really bad at, these all still apply: > People will only want to build software on Linux. People will only want to build software on a Unix derivative. Nobody will want to build software on Windows. People will only want to build software on Windows. Bazel support in windows is shaky. Bazel itself works but many tools (rules_docker) do not work. One place bazel would be really amazing for, in my opinion, is the game development industry. You have massive source trees, very complex packaging steps, you often have tools that are used as a compiler further in the compilation process (asset packets), you can benefit from shared build clusters, etc. > Single-character magic variables are a good idea in a language that most programmers will rarely use. Not the same but make-like variable substitution exists and it's a bit of a pain: https://docs.bazel.build/versions/master/be/make-variables.html https://docs.bazel.build/versions/master/be/make-variables.h... > System libraries and globally-installed tools never change. Version numbers of system libraries and globally-installed tools only ever increase. Bazel is mostly hermetic except for extremely strange things. Build toolchains that are "expected" to be in the host bleed in and effect a lot of things. It's a really gross side effect of certain things and can cause some nasty bugs. And for some new rules that bazel sins with: 1. People are ok with running builds on hosts of separate architectures to build their software for different architectures Cross compiling in bazel is extremely painful. 2. IDE integration doesn't matter. A lot of people who use bazel don't really care about how well, or how poorly, it integrates with an IDE. As such a lot of the open source world around bazel's IDE integrations are lack luster. Getting things like gRPC working in bazel with auto-completions for multiple languages is nearly impossible. 3. Everyone wants 100% reproducible builds. Many people do not want reproducible builds. That may sound shocking to some but a lot of other tools depend on non-reproducible elements of a build chain. For instance: when using bazel to build a docker container it sets the container's creation timestamp to the unix epoch. Because of this I cannot use Gitlab's registry cleanup script as it'll wipe all images since they're all over N days old. 4. All of the compilers I want to use during my build are installed on my host system. Docker was probably the most amazing improvement in build/tooling workflow in recent years. If I want to build something with docker all I need is docker. If bazel to make this workflow the same it could take over the world. If there was a single place to configure something like this: toolchains( arm: { c: "gcc:10.1.0" } x86: { c: "icc:....." } ) And if the default was that for any language not specified, it would either not be available or would be easily buildable from source as long as you have a C compiler. I don't know if this is the best way to represent this concept but I have no idea why I have to even be concerned what version my host OS ships of gcc. I also don't know why I have to care if my host system is x86 but I want to build a binary for arm. Either use cross compiling features of the compiler or spin up a VM and run the build using native tooling in there. I shouldn't have to manually do that.
- dmoy 6y ago
  Yea y'all can pry blaze from my cold, dead hands. It may have its own problems, but working on a team that compiles things in like 10-15 different languages (c++, c#, go, java, javascript, typescript, lisp, haskell, blaze itself (whatever they call it now... skylark?), python, swift, kotlin, soon rust, a couple flavors of sql, etc), it saves us a lot of hassle.
  - yegle 6y ago
    Starlark, because reasons.
- amelius 6y ago
  Bazel is cute but doesn't solve the real problem that if one changes only a single line in a C file in general it is wasteful to recompile that file from scratch.
trishankkarthik 6y ago
YAML IS NOT AN EFFING PROGRAMMING LANGUAGE, EVEN IN 2020
- com2kid 6y ago
  There are multiple XML based programming languages, so no doubt properly complex YAML configs are Turing complete by now. I mean what do you expect? People cram Turing completeness everywhere and anywhere.
  - ben0x539 6y ago
    If you don't want people to reach for Turing complete config languages, don't make them solve weird problems in config files, imo.
- thetanil 6y ago
  this, so much this
- dang 6y ago
  Please don't use uppercase for emphasis. If you want to emphasize a word or phrase, put asterisks around it and it will get italicized. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html.
  - tdons 6y ago
    yaml deserves all caps
    - dang 6y ago
      If we take the union of all allcaps-deservers across the set of all users with such feelings, half the site will run through an upcase function. As in any mutually assured destruction scenario, we must all restrain ourselves.
      - 6y ago
klodolph 6y ago
> Build graphs are acyclic. While yes, this is a falsehood... in practice, you generally want to figure out a way to break cycles and remove them from the build system. In general, make the user responsible for breaking cycles in the build system. The only major use case for cycles that I can think of is self-hosting compilers. I would add some additional falsehoods: - It is always the right thing to rebuild files when the inputs have changed. - The host and target systems are the same (you are never cross-compiling). - The host and target systems are different (you are always cross-compiling). - You can reliably figure out the version number of whatever you are building. - The build steps always produce the correct outputs if they do not fail. - You always want to build third-party dependencies from source. - You never want to build third-party dependencies from source. - The input files do not change while the build is running. - All of the inputs are files (and not, say, the current timestamp). - The entire build should use the same build settings (optimization / debug), you don't want to do a partial-debug, partial-optimized build. - You don't want to rebuild when e.g. files in /usr/include change. - You always want to rebuild when e.g. files in /usr/include change. - Installing additional software will not break the build. - Running the same build twice will result in the same output. - Build products do not contain absolute paths. - You can move an executable from one path to another and it will still work. - You can delete object files and everything will still work.
- jnxx 6y ago
  > > Build graphs are acyclic. Normal C code, not that I can think of. But it is not difficult to give a real-world example: A LaTeX document with a table of contents and an index has a cyclic dependency: The final document, including its pagination, depends on the table of contents, and the table of contents depends on the final document.
  - amelius 6y ago
    > A LaTeX document with a table of contents and an index has a cyclic dependency One of the uglier problems of LaTeX, which people of the likes of Leslie Lamport and Donald Knuth should have been able to solve, one might think.
jpollock 6y ago
Build systems are tools just the same as an editor. In the same way that editors are opinionated about keymappings, build systems are opinionated about things like timestamps vs hashing. If you're using a tool, you really should know how it works and where it will break. You should also choose the tool based on the problem you're trying to solve. If you've got a build system that rebuilds when /usr/include changes, and you're trying to do cross-compiles, choose another build system.
- raverbashing 6y ago
  Timestamps vs hashes shouldn't be even a thing. Why does this even matter? Sure ok maybe you touched a file and the timestamp wasn't updated (it should, and your tool is making things harder on you). Or if you're going by hash, same hash, same file. Period. (Unless you're doing something funny with file metadata, and then again, you're shooting yourself in the foot) Same with some comments on the original article saying "tool doesn't return correct status codes for success/failure" well, maybe you need tools that work then Most of the advice in the article is sound though
com2kid 6y ago
Since 2012 we have, as a field, learned better. You see, we all realized that just building code wasn't enough! What your 100 line JavaScript microservice really needs to be accompanied by is a build file that determines how to create an entire Linux VM. Also, because scale, your project also needs to know how to scale that VM out across multiple clusters! The only answer of course is more build systems! Oh but JavaScript sucks, so make that Typescript, so shove a transpiler in front of your scripting language. Because somehow we've gone down the path of needing build systems for interpreted languages. I use all the above tech and get why it is there, but... seriously as a field this is the best we can do? :/ (I legit like Babel and TS, but you have to admit it is /weird/ that we're going through the complexity of having a build system but are reaping none of the performance benefits!)
- ReactiveJelly 6y ago
  Technically containers are not VMs And re: build systems for interpreted languages, what would the alternative be? Asking Firefox and Chrome to integrate a language they didn't design? Making it a runtime module like MoonScript, and letting everyone who loads the page pay for compilation, on battery-constrained computers, millions of times over?
  - EE84M3i 6y ago
    >Technically containers are not VMs Some build systems do build proper VM images, like AMIs.
    - rumanator 6y ago
      You're talking about a cloud service provider, whose bread and butter is selling you time on their VMs. Of course they manage machine images.
- qubidt 6y ago
  It's the classic question of "Reform or Revolution?" These systems were built as incremental mitigations to old problems. I don't think it'll change until they collapse under their own weight and people switch over to something else completely.
dang 6y ago
If curious see also 2018 https://news.ycombinator.com/item?id=16196899 https://news.ycombinator.com/item?id=16196899
ancarda 6y ago
>I want to see an end to Make in my lifetime Why? What's wrong with Make?
- asveikau 6y ago
  Make is actually pretty amazing for its simplicity, longevity, how it gets so much of the job done without doing very much. Yes there are pitfalls and caveats, good uses and bad, but the balance reached between simplicity and ability to be productive and get you most of the way is amazing. I feel that when I hear somebody say they don't like make, usually they misunderstand this, or they think somebody's bad experiences with makefiles is somehow representative of the capability of the tool. One person created make in a weekend in 1976. We are still talking about it. Most humans and most software will never achieve this.
  - eddieh 6y ago
    Exactly this! Make is the worst build system, except for all the others.
- rhn_mk1 6y ago
  Tabs having a different meaning than spaces. Literally the only reason I'm not using it. Yes, I realize GNU Make has a config option to fix this. That doesn't seem to be widely adopted. On a second thought, the shellisms it borrows are also annoying. POSIX shell syntax needs to die on its own right.
  - dima55 6y ago
    Of all the things to complain about, this one is just silly. You can use ; instead of TAB. There. Problem solved? Do you like Make now?
    - rhn_mk1 6y ago
      Thanks for the tip! I had no idea about this. I reserve the right to be fully satisfied only after the majority of random Makefiles I find on the net or emitted from build systems will use that convention. To add another thing to the pot, let me repeat some things already mentioned or implied: Make is too focused on files. Not every input is a file, and not every artifact is supposed to be a file either. Make doesn't allow choosing the update criterion per file. Sometimes it's the has, sometimes the timestamp, and sometimes the parsed AST of the source code that should trigger a rebuild when changed.
eddieh 6y ago
If the build system isn't first a DSL for specifying a DAG, then it fails at being at least as good as Make. Maybe "build graphs are acyclic" is a falsehood, but I would never want to use a build system that isn't fundamentally constructing a DAG. There are plenty of imperative build systems (or build systems that people us imperatively) and they're all horrible.
sdenton4 6y ago
> "Programmers don't want a system for writing build scripts; they want a system for writing systems that write build scripts." It is a truth universally acknowledged that a shitty scripting language in possession of a large user base will be in search of a new scripting language that automatically generates the shitty scripting language. But then you have two problems.
- marcosdumay 6y ago
  That's a good observation. Any deficiency in a complex, widely used and difficult to change development tool will be fixed by adding an extra layer over it. If the deficiency is something like complexity or lack of reliability that can not be fixed by adding extra layers, read this comment again.
  - hoskdoug 6y ago
    Great example is Fastlane abstracting over the impenetrable iOS command line tools.
- jnxx 6y ago
  Is that how cmake came to exist?
  - rumanator 6y ago
    Cake came into existence because all other alternatives were pure crap. And since cmake's inception, all cmake alternatives are still pure crap. The definition of "well" is relative.
    - btschaegg 6y ago
      > And since cmake's inception, all cmake alternatives are still pure crap. …including CMake. :-) Although, to be fair, I think CMake did at least achieve to prove that creating a turing complete "jack of all trades" meta build system is not a good idea in pretty much every case (although I'd argue that boost had done that well in advance). It's too sad that with the myriad of legacy patterns around in the C/C++ build space (in production), there is pretty much 0 chance of simplifying the problem for a majority of cases instead.
yongjik 6y ago
IMHO the greatest falsehood I believed about build systems was: - If build system X is used, people will read its documentation and write build rules according to the documentation. No, people will just copy what's already there. And if it pulls in 35 external dependencies, compile 500 unnecessary files, and fails 25% of the time because of race conditions between external dependencies, people will add "Note: if build fails, run again." Sigh.
- dcow 6y ago
  And then there's `make`. I've seen a philosophically correct Makefile maybe 3 times in my life. Most of the time I see people making every target .PHONY and using patterns to save keystrokes rather than match multiple files. You'd tend to think people learn over time but no. It's gotten worse. Right when you thought the industry was on the cusp of a build revolution the go community decides to push an old tool that most people can't utilize even 3% of when paired with `go build`. I kinda blame the go ecosystem for pushing Makefile adoption when they already have a caching build tool just so the maintainers didn't have to give the compiler a polished UI. Simplicity at the cost of functionality even if it's simply broken amirite? Sigh.
  - ancarda 6y ago
    I've never seen Go projects use Make? When did this change happen?
    - dcow 6y ago
      The golang project itself is a perfect example: https://github.com/golang/go/blob/master/src/Make.dist https://github.com/golang/go/blob/master/src/Make.dist
  - rumanator 6y ago
    > and using patterns to save keystrokes rather than match multiple files. To be fair, I'm still yet to see a document clarifying how to use patterns with make.
- 6y ago
saurik 6y ago
> stat is slow on modern filesystems. Is this not true?!? While working on my build times I benchmarked that something like a quarter of my time in clang was due to my having large numbers of sparse -I folders as the stat calls were so expensive (this was a 12" MacBook running macOS 10.14 on, I assume, APFS).
- deleted 6y ago
  [deleted]
somewhereoutth 6y ago
Source ---[Big Red Button]---> Artifacts How did this go wrong??
- marcosdumay 6y ago
  I think it started with developers not wanting to write all of the source and artifact files into the makefile. Then there were some portability issues that should have been handled by the compiler, but by that time there were already metabuilders, so nobody bothered.
bigdict 6y ago
> It is accepted by all decent people that Make sucks and needs to die falsehood #1
- fmajid 6y ago
  Amen. The worst are crackpot systems like ant. Whoever thought a half-baked DSL with a XML concrete syntax was a good idea?
  - deleted 6y ago
    [deleted]
  - raverbashing 6y ago
    > ant. > Whoever thought a half-baked DSL with a XML concrete syntax was a good idea? You kinda answered your own question there no?
- arauhala 6y ago
  Preach to the choir! :) Make does so many things correctly: - it gives user total freedom to modify the built - yet it has excellent defaults for most situations - it is full blown programming language, - yet it's syntax is extremely specialized for the purpose and familiar at the same time (it's bash) - it's basic assumptions/structure are extremely simple (timestamps, dependencies) - yet it's extremely powerful and can take into account most situations. It's be extremely Unix-style tool with very simple building blocks that combine together in extremely powerful way. Its fast to learn and easy to master and as such, the best kind of design. Sometimes making a build with it is a small programming project, yet after using all kinds of built tools, I end up just wishing that I could use make instead.
  - pydry 6y ago
    bash is a pretty terrible programming language and it's terribleness increases geometrically with size. This is a large part of why people don't like make I think.
rainbow-road 6y ago
Most of the listed assumptions are bad, but some of them, to me, seem essential. In particular, dealing with cyclic build graphs sounds like an absolute nightmare, and I doubt that any build system that allows for such a thing can be easy-to-use.
morelisp 6y ago
This "falsehoods programmers believe" is fundamentally different from the more classic ones (e.g. "about names") because its topic is also wholly controlled by programmers. That means in the end it doesn't share much with those documents, but can instead also be characterized as "just" a list of bugs in other programs. If a program doesn't handle my name, that's a problem with the program. My name is my name; the program doesn't decide that. Depending on the context some government might; the program also generally isn't supposed to decide for the government. Often in practice the program controls both of us; this is the core complaint of "falsehoods" articles, that a program artifact is overstepping its ethical/legal/otherwise extra-computational authority. But there's no moral demand that e.g. a program exit with 0 status even when it's failed, or that build trees should be cyclic. These are tools by programmers and for programmers concerning programs and when we run into issues it's usually because we chose a poor modeling tool (e.g. a build system should not be ideal to calculate the fixed point of a data processing pipeline, independent of any cycles) or because of a flat-out mistake (there's no reason a compiler can't be given some output directive other than inventor oversight). Other "falsehoods" are criticism of the software encoding the falsehood, but I read probably 90% of the complaints here as equal or greater failings on the tools invoked. Not only for the sake of my build system do I want DAG pipelines, updated timestamps, or meaningful exit codes - if anything these are the demands I wish to place on other software for the sake of my brain! More like "falsehoods compilers believe about their primacy in program generation." Or as one of the comments suggests, > build systems are a symptom of software languages that are not designed to build software systems
ex_amazon_sde 6y ago
In 2020 we can add additional falsehoods based on my experience in Amazon: - It's OK to embed/vendor dependencies - It's OK to hardcode VCS URLs for your dependencies - you can pull random stuff from the Internet at compile time and large companies will accept it - It's OK to release libraries without version numbering or using git commitish - Everybody uses containers - Nobody cares about reproducible builds - Nobody cares about security, license compliance and being able to rebuild things reliably - All FAANGS will use your project even if it cannot packaged into OS packages
- Apofis 6y ago
  This happens everywhere... some platforms became so complicated to do initial setup with that Docker became a necessity instead of an install script. It now takes me weeks to setup a fresh machine to do local dev on. I guess I should make a docker image of my dev box. cd devbox docker-compose up ...would be nice, now that I think about it.
  - ex_amazon_sde 6y ago
    I suspect you missed the point about falsehoods.
- blandflakes 6y ago
  God the number of classpath conflicts I untangled there that were caused by some bullshit library bundling some ancient apache-commons jar...
- ben0x539 6y ago
  While I agree with a bunch of what you're saying, I expect that FAANG compatibility is not people's top priority when they're not getting paid FAANG salaries.
haolez 6y ago
I tend to use whichever build system is mainstream in the environment I'm in. However, if I'm dealing with something special, or if the build system is unstable, I always fall back to Make. Such a beautiful tool! Tup is also worth to mention: http://gittup.org/tup/ http://gittup.org/tup/
pwm 6y ago
Anyone curious about the core abstractions and capabilities of build systems should read the amazing Build Systems à la Carte paper: https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/0... Here is a teaser, the 1st table: Build system | Persistent build information | Scheduler | Dependencies | Minimal | Cutoff | Cloud Make | File modification times | Topological | Static | Yes | No | No Excel | Dirty cells, calc chain | Restarting | Dynamic | No | No | No Shake | Previous dependency graph | Suspending | Dynamic | Yes | Yes | No Bazel | Cloud cache, command history | Restarting | Dynamic(∗) | No | Yes | Yes which shows that Excel can be viewed as a dynamic build system!
scoot_718 6y ago
Some of those aren't believed assumptions, but asserted ones. For instance, I would think a build system that asserts that builds are acyclic is good. People who want a cyclic build are unimportant and can go elsewhere, while I enjoy my simpler build system because of it. Simplifying assumptions are good.

69 points by jnxx 6y ago

deleted 6y ago

[deleted]

sabujp 6y ago

this is from 2012 and some of the points haven't aged well (see blaze/bazel)

RealityVoid 6y ago
Not familiar with bazel. Can you expand a bit?
- klodolph 6y ago
  There are a few things which are either enforced directly by Bazel (i.e., Bazel takes an opinion on how build systems should work and gives an error if you try to make it work otherwise). There are also a few assumptions which are now harmless, because Bazel will either check that your assumptions are correct, or make the build work as if those assumptions hold. For example, "Build graphs are acyclic." Bazel enforces this. "Every build step updates at least one file." If your build step doesn't produce output, Bazel will complain. "It's possible to tell the compiler which file to write its output to." "It's possible to tell the compiler which directory to write its output to." Bazel does this with sandboxes.
carlmr 6y ago
Which ones and why? Is blaze/bazel free of all these falsehoods. Did it manage to make new mistakes?
- sabujp 6y ago
  I'm no blaze/bazel expert (hopefully someone will add their own thoughts), but: 1. build graphs are trees/acyclic : bazel enforces this, no cyclic dependencies allowed 2. Compilers will always modify the timestamps on every file they are expected to output : if a write happens why not update mtime? nanosecond precision, what year is it? 3. It's possible to tell the compiler which file to write its output to : makes no sense to me, we expect our compilers to be writing to the correct files in the correct directories. 4. It's possible to predict in advance which files the compiler will update : yes it is, this is how caching works again see (1) 5. It's possible to determine the dependencies of a target without building it : yes yes it's completely possible given (1) 6. Detecting changes via file hashes is never the right thing : Bazel i think uses a combination of hashes and stamps, but why would using hashes not work 99.99999999999999998% of the time? 7. Non-experts can reliably write portable shell script : lol, experts can't reliably write portable shell scripts either 8. Your build tool is a great opportunity to invent a whole new language: or just coopt pieces of one (python: starlark, skylark) 9. Said language does not need to be a full-featured programming language : yes, it doesn't (again .bzl files) 10. Said language should be based on textual expansion : not sure what the alternative to textual expansion (parsing?) is 11. Adding an Nth layer of textual expansion will fix the problems of the preceding N-1 layers : again how is this different from parsing
  - deleted 6y ago
    [deleted]
  - tzs 6y ago
    OT: when did HN start wrapping long lines in indented text?
    - SilasX 6y ago
      It's not because the lines are long but because they're prefixed by four spaces (and for anyone reading, please don't do that unless it's actual code).
      - tzs 6y ago
        It's wrapping lines prefixed with two spaces now. The comment I just posted a couple minutes ago with indented shell input and output definitely used two spaces and it is wrapping on my phone and on my desktop if I narrow the window enough.
  - tzs 6y ago
    > 2. Compilers will always modify the timestamps on every file they are expected to output : if a write happens why not update mtime? nanosecond precision, what year is it? It has nanosecond precision, but on most systems it does not have nanosecond resolution. $ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify: Modify: 2020-07-18 21:37:56.421487075 +0000 Modify: 2020-07-18 21:37:56.421487075 +0000 $ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify: Modify: 2020-07-18 21:37:57.397502229 +0000 Modify: 2020-07-18 21:37:57.397502229 +0000 $ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify: Modify: 2020-07-18 21:37:58.273515840 +0000 Modify: 2020-07-18 21:37:58.277515902 +0000 $ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify: Modify: 2020-07-18 21:41:00.736348218 +0000 Modify: 2020-07-18 21:41:00.736348218 +0000 $ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify: Modify: 2020-07-18 21:41:01.708363288 +0000 Modify: 2020-07-18 21:41:01.708363288 +0000 $ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify: Modify: 2020-07-18 21:41:02.504375653 +0000 Modify: 2020-07-18 21:41:02.508375715 +0000 Note that the difference is either 0 or 0.004000062. This is on a Debian 9 system running on an Amazon Lightsail instance with an ext4 filesystem. Here's a quick and dirty script to play with this: C=0 R=0 while : do R=$((R+1)) touch a1; sleep $1; touch a2 T1=$(stat a1 | grep Modify | sed -e 's/.*://' | sed -e 's/ .*//') T2=$(stat a2 | grep Modify | sed -e 's/.*://' | sed -e 's/ .*//') if [ $T1 != $T2 ]; then dc <<HERE 20k $T2 $T1 - p HERE C=$((C+1)) if [ $C -eq 100 ]; then echo "$R runs" exit 0 fi fi done Output on the above system: $ bash t.sh 0.001 | sort | uniq -c 4 .004000061 83 .004000062 13 .004000063 1 222 runs and on a VMWare Fusion VM running Debian 8 on my Mac: $ bash t.sh 0.001 | sort | uniq -c 29 .003999599 44 .003999600 27 .003999601 1 323 runs
- gravypod 6y ago
  These are some of my personal opinions from using, and violently advocating for, bazel: > Build graphs are trees. Build graphs are acyclic. Bazel uses some tricks/magic to hide outputs of build rules from other build rules. Bazel also prevents cyclic dependencies (A depends on B which depends on A). > Compilers will always modify the timestamps on every file they are expected to output. Source files a target depends on will be hashed. The operations applied to the files will also be hashed. > It's possible to determine the dependencies of a target without building it. Bazel forces people to explicitly define all deps that the source code uses. It uses sandboxing to hide unspecified deps from the source code. All of the definition of these rules is defined statically-ish (if you ignore WORKSPACE which technically applies to Blaze). > Targets do not depend on the rules used to build them. Bazel hashes the operations the rules will perform. > Targets depend on every rule in the whole build system. Bazel understands the full DAG and knows what each target depends on and what depends on each target. > Detecting changes via file hashes is always the right thing. Detecting changes via file hashes is never the right thing. It's a tradeoff that someone has to decide. Neither is right or wrong. Using a file content+metadata hash allows you gauntness that the build is always correct. > It's possible to narrow down the set of possibly-updated files to a small hand-enumerated set. As long as you use file hashing and are explicit about your inputs and outputs of every build rule, you can. > Nobody will ever want to rebuild a subset of the available dirty targets. You can build a specific target and it's deps at any time. Unrelated dependency trees are ignored. > stat is slow on modern filesystems. Anything is slow given a large enough scale. Any syscall is slow once you have millions of source files. > Non-experts can reliably write portable shell script. In bazel people interact with higher level rules. You don't need to write shell scripts unless you're implementing something that requires shell scripts. If you'd like you can also use an output of a target as a tool to do further compiling. For example: build protoc then use protoc to compile protos into libraries. Every build step updates at most one file. Every build step updates at least one file. It's possible to tell the compiler which file to write its output to. It's possible to tell the compiler which directory to write its output to. It's possible to predict in advance which files the compiler will update. > Your build tool is a great opportunity to invent a whole new language. Said language does not need to be a full-featured programming language. In particular, said language does not need a module system more sophisticated than #include. Said language should be based on textual expansion. Adding an Nth layer of textual expansion will fix the problems of the preceding N-1 layers. It is a great opportunity to implement a domain specific language as "building software" is a domain that we would like to define programs for. Bazel uses skylark, it's a "fully featured" programming language in that there's loops, conditionals, functions (called "macros"), and a module-ish system ("load()"). It does not operate on textual expansion. > It's totally OK to spend over four hours calculating how much of a 25-minute build you should do. Bazel's target analysis takes a while but for my scale of code base it doesn't take hours, it takes seconds. If you have a huge codebase then these "wasted" seconds would be earned back by it's effective caching and remote build execution. > All the code you will ever need to compile is written in precisely one language. Bazel is multilingual. > Everything lives in a single repository. What is a "repository"? It's really a file system. What is a file system? A fancy kv store. Could you build a KV store that could return the values for every key that is a file in your source tree? Probably. Bazel is built around a monorepo design but you could just as easily clone libraries into shared folders and executre bazel from above your project. You could also use remote repository rules. IMO you're better off just vendoring any source code you're writing in a folder in your repo so you can guarantee old versions of things are around when you need them and you never have a leftpad incident. > Files only ever get updated with timestamps by a single machine. Version control systems will always update the timestamp on a file. Version control systems will never update the timestamp on a file. Version control systems will never change the time to one earlier than the previous timestamp. File contents are check so this is not as much of an issue. > Programmers don't want a system for writing build scripts; they want a system for writing systems that write build scripts. I think people don't want systems like this because they've been burned. I've used a lot of makefile generators and they're all a really poor caching system. You essentially make cache invalidation part of your dev flow. AS a developer I don't want to think about that. I want to ask something to build me something and have it know when it needs to reload build definitions or code. I also don't want to care about what specific "gcc" or "javac" commands are being run. I need to be able to go down to that level but 99% of the time I just need to say "Turn a.c, b.c, and c.c into foobar.bin". As for what bazel is really bad at, these all still apply: > People will only want to build software on Linux. People will only want to build software on a Unix derivative. Nobody will want to build software on Windows. People will only want to build software on Windows. Bazel support in windows is shaky. Bazel itself works but many tools (rules_docker) do not work. One place bazel would be really amazing for, in my opinion, is the game development industry. You have massive source trees, very complex packaging steps, you often have tools that are used as a compiler further in the compilation process (asset packets), you can benefit from shared build clusters, etc. > Single-character magic variables are a good idea in a language that most programmers will rarely use. Not the same but make-like variable substitution exists and it's a bit of a pain: https://docs.bazel.build/versions/master/be/make-variables.html https://docs.bazel.build/versions/master/be/make-variables.h... > System libraries and globally-installed tools never change. Version numbers of system libraries and globally-installed tools only ever increase. Bazel is mostly hermetic except for extremely strange things. Build toolchains that are "expected" to be in the host bleed in and effect a lot of things. It's a really gross side effect of certain things and can cause some nasty bugs. And for some new rules that bazel sins with: 1. People are ok with running builds on hosts of separate architectures to build their software for different architectures Cross compiling in bazel is extremely painful. 2. IDE integration doesn't matter. A lot of people who use bazel don't really care about how well, or how poorly, it integrates with an IDE. As such a lot of the open source world around bazel's IDE integrations are lack luster. Getting things like gRPC working in bazel with auto-completions for multiple languages is nearly impossible. 3. Everyone wants 100% reproducible builds. Many people do not want reproducible builds. That may sound shocking to some but a lot of other tools depend on non-reproducible elements of a build chain. For instance: when using bazel to build a docker container it sets the container's creation timestamp to the unix epoch. Because of this I cannot use Gitlab's registry cleanup script as it'll wipe all images since they're all over N days old. 4. All of the compilers I want to use during my build are installed on my host system. Docker was probably the most amazing improvement in build/tooling workflow in recent years. If I want to build something with docker all I need is docker. If bazel to make this workflow the same it could take over the world. If there was a single place to configure something like this: toolchains( arm: { c: "gcc:10.1.0" } x86: { c: "icc:....." } ) And if the default was that for any language not specified, it would either not be available or would be easily buildable from source as long as you have a C compiler. I don't know if this is the best way to represent this concept but I have no idea why I have to even be concerned what version my host OS ships of gcc. I also don't know why I have to care if my host system is x86 but I want to build a binary for arm. Either use cross compiling features of the compiler or spin up a VM and run the build using native tooling in there. I shouldn't have to manually do that.
dmoy 6y ago
Yea y'all can pry blaze from my cold, dead hands. It may have its own problems, but working on a team that compiles things in like 10-15 different languages (c++, c#, go, java, javascript, typescript, lisp, haskell, blaze itself (whatever they call it now... skylark?), python, swift, kotlin, soon rust, a couple flavors of sql, etc), it saves us a lot of hassle.
- yegle 6y ago
  Starlark, because reasons.
amelius 6y ago
Bazel is cute but doesn't solve the real problem that if one changes only a single line in a C file in general it is wasteful to recompile that file from scratch.

trishankkarthik 6y ago

YAML IS NOT AN EFFING PROGRAMMING LANGUAGE, EVEN IN 2020

com2kid 6y ago
There are multiple XML based programming languages, so no doubt properly complex YAML configs are Turing complete by now. I mean what do you expect? People cram Turing completeness everywhere and anywhere.
- ben0x539 6y ago
  If you don't want people to reach for Turing complete config languages, don't make them solve weird problems in config files, imo.
thetanil 6y ago
this, so much this
dang 6y ago
Please don't use uppercase for emphasis. If you want to emphasize a word or phrase, put asterisks around it and it will get italicized. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html.
- tdons 6y ago
  yaml deserves all caps
  - dang 6y ago
    If we take the union of all allcaps-deservers across the set of all users with such feelings, half the site will run through an upcase function. As in any mutually assured destruction scenario, we must all restrain ourselves.
    - 6y ago

klodolph 6y ago

> Build graphs are acyclic. While yes, this is a falsehood... in practice, you generally want to figure out a way to break cycles and remove them from the build system. In general, make the user responsible for breaking cycles in the build system. The only major use case for cycles that I can think of is self-hosting compilers. I would add some additional falsehoods: - It is always the right thing to rebuild files when the inputs have changed. - The host and target systems are the same (you are never cross-compiling). - The host and target systems are different (you are always cross-compiling). - You can reliably figure out the version number of whatever you are building. - The build steps always produce the correct outputs if they do not fail. - You always want to build third-party dependencies from source. - You never want to build third-party dependencies from source. - The input files do not change while the build is running. - All of the inputs are files (and not, say, the current timestamp). - The entire build should use the same build settings (optimization / debug), you don't want to do a partial-debug, partial-optimized build. - You don't want to rebuild when e.g. files in /usr/include change. - You always want to rebuild when e.g. files in /usr/include change. - Installing additional software will not break the build. - Running the same build twice will result in the same output. - Build products do not contain absolute paths. - You can move an executable from one path to another and it will still work. - You can delete object files and everything will still work.

jnxx 6y ago
> > Build graphs are acyclic. Normal C code, not that I can think of. But it is not difficult to give a real-world example: A LaTeX document with a table of contents and an index has a cyclic dependency: The final document, including its pagination, depends on the table of contents, and the table of contents depends on the final document.
- amelius 6y ago
  > A LaTeX document with a table of contents and an index has a cyclic dependency One of the uglier problems of LaTeX, which people of the likes of Leslie Lamport and Donald Knuth should have been able to solve, one might think.

jpollock 6y ago

Build systems are tools just the same as an editor. In the same way that editors are opinionated about keymappings, build systems are opinionated about things like timestamps vs hashing. If you're using a tool, you really should know how it works and where it will break. You should also choose the tool based on the problem you're trying to solve. If you've got a build system that rebuilds when /usr/include changes, and you're trying to do cross-compiles, choose another build system.

raverbashing 6y ago
Timestamps vs hashes shouldn't be even a thing. Why does this even matter? Sure ok maybe you touched a file and the timestamp wasn't updated (it should, and your tool is making things harder on you). Or if you're going by hash, same hash, same file. Period. (Unless you're doing something funny with file metadata, and then again, you're shooting yourself in the foot) Same with some comments on the original article saying "tool doesn't return correct status codes for success/failure" well, maybe you need tools that work then Most of the advice in the article is sound though

com2kid 6y ago

Since 2012 we have, as a field, learned better. You see, we all realized that just building code wasn't enough! What your 100 line JavaScript microservice really needs to be accompanied by is a build file that determines how to create an entire Linux VM. Also, because scale, your project also needs to know how to scale that VM out across multiple clusters! The only answer of course is more build systems! Oh but JavaScript sucks, so make that Typescript, so shove a transpiler in front of your scripting language. Because somehow we've gone down the path of needing build systems for interpreted languages. I use all the above tech and get why it is there, but... seriously as a field this is the best we can do? :/ (I legit like Babel and TS, but you have to admit it is /weird/ that we're going through the complexity of having a build system but are reaping none of the performance benefits!)

ReactiveJelly 6y ago
Technically containers are not VMs And re: build systems for interpreted languages, what would the alternative be? Asking Firefox and Chrome to integrate a language they didn't design? Making it a runtime module like MoonScript, and letting everyone who loads the page pay for compilation, on battery-constrained computers, millions of times over?
- EE84M3i 6y ago
  >Technically containers are not VMs Some build systems do build proper VM images, like AMIs.
  - rumanator 6y ago
    You're talking about a cloud service provider, whose bread and butter is selling you time on their VMs. Of course they manage machine images.
qubidt 6y ago
It's the classic question of "Reform or Revolution?" These systems were built as incremental mitigations to old problems. I don't think it'll change until they collapse under their own weight and people switch over to something else completely.

dang 6y ago

If curious see also 2018 https://news.ycombinator.com/item?id=16196899 https://news.ycombinator.com/item?id=16196899

ancarda 6y ago

>I want to see an end to Make in my lifetime Why? What's wrong with Make?

asveikau 6y ago
Make is actually pretty amazing for its simplicity, longevity, how it gets so much of the job done without doing very much. Yes there are pitfalls and caveats, good uses and bad, but the balance reached between simplicity and ability to be productive and get you most of the way is amazing. I feel that when I hear somebody say they don't like make, usually they misunderstand this, or they think somebody's bad experiences with makefiles is somehow representative of the capability of the tool. One person created make in a weekend in 1976. We are still talking about it. Most humans and most software will never achieve this.
- eddieh 6y ago
  Exactly this! Make is the worst build system, except for all the others.
rhn_mk1 6y ago
Tabs having a different meaning than spaces. Literally the only reason I'm not using it. Yes, I realize GNU Make has a config option to fix this. That doesn't seem to be widely adopted. On a second thought, the shellisms it borrows are also annoying. POSIX shell syntax needs to die on its own right.
- dima55 6y ago
  Of all the things to complain about, this one is just silly. You can use ; instead of TAB. There. Problem solved? Do you like Make now?
  - rhn_mk1 6y ago
    Thanks for the tip! I had no idea about this. I reserve the right to be fully satisfied only after the majority of random Makefiles I find on the net or emitted from build systems will use that convention. To add another thing to the pot, let me repeat some things already mentioned or implied: Make is too focused on files. Not every input is a file, and not every artifact is supposed to be a file either. Make doesn't allow choosing the update criterion per file. Sometimes it's the has, sometimes the timestamp, and sometimes the parsed AST of the source code that should trigger a rebuild when changed.

eddieh 6y ago

If the build system isn't first a DSL for specifying a DAG, then it fails at being at least as good as Make. Maybe "build graphs are acyclic" is a falsehood, but I would never want to use a build system that isn't fundamentally constructing a DAG. There are plenty of imperative build systems (or build systems that people us imperatively) and they're all horrible.

sdenton4 6y ago

> "Programmers don't want a system for writing build scripts; they want a system for writing systems that write build scripts." It is a truth universally acknowledged that a shitty scripting language in possession of a large user base will be in search of a new scripting language that automatically generates the shitty scripting language. But then you have two problems.

marcosdumay 6y ago
That's a good observation. Any deficiency in a complex, widely used and difficult to change development tool will be fixed by adding an extra layer over it. If the deficiency is something like complexity or lack of reliability that can not be fixed by adding extra layers, read this comment again.
- hoskdoug 6y ago
  Great example is Fastlane abstracting over the impenetrable iOS command line tools.
jnxx 6y ago
Is that how cmake came to exist?
- rumanator 6y ago
  Cake came into existence because all other alternatives were pure crap. And since cmake's inception, all cmake alternatives are still pure crap. The definition of "well" is relative.
  - btschaegg 6y ago
    > And since cmake's inception, all cmake alternatives are still pure crap. …including CMake. :-) Although, to be fair, I think CMake did at least achieve to prove that creating a turing complete "jack of all trades" meta build system is not a good idea in pretty much every case (although I'd argue that boost had done that well in advance). It's too sad that with the myriad of legacy patterns around in the C/C++ build space (in production), there is pretty much 0 chance of simplifying the problem for a majority of cases instead.

yongjik 6y ago

IMHO the greatest falsehood I believed about build systems was: - If build system X is used, people will read its documentation and write build rules according to the documentation. No, people will just copy what's already there. And if it pulls in 35 external dependencies, compile 500 unnecessary files, and fails 25% of the time because of race conditions between external dependencies, people will add "Note: if build fails, run again." Sigh.

dcow 6y ago
And then there's `make`. I've seen a philosophically correct Makefile maybe 3 times in my life. Most of the time I see people making every target .PHONY and using patterns to save keystrokes rather than match multiple files. You'd tend to think people learn over time but no. It's gotten worse. Right when you thought the industry was on the cusp of a build revolution the go community decides to push an old tool that most people can't utilize even 3% of when paired with `go build`. I kinda blame the go ecosystem for pushing Makefile adoption when they already have a caching build tool just so the maintainers didn't have to give the compiler a polished UI. Simplicity at the cost of functionality even if it's simply broken amirite? Sigh.
- ancarda 6y ago
  I've never seen Go projects use Make? When did this change happen?
  - dcow 6y ago
    The golang project itself is a perfect example: https://github.com/golang/go/blob/master/src/Make.dist https://github.com/golang/go/blob/master/src/Make.dist
- rumanator 6y ago
  > and using patterns to save keystrokes rather than match multiple files. To be fair, I'm still yet to see a document clarifying how to use patterns with make.
6y ago

saurik 6y ago

> stat is slow on modern filesystems. Is this not true?!? While working on my build times I benchmarked that something like a quarter of my time in clang was due to my having large numbers of sparse -I folders as the stat calls were so expensive (this was a 12" MacBook running macOS 10.14 on, I assume, APFS).

deleted 6y ago
[deleted]

somewhereoutth 6y ago

Source ---[Big Red Button]---> Artifacts How did this go wrong??

marcosdumay 6y ago
I think it started with developers not wanting to write all of the source and artifact files into the makefile. Then there were some portability issues that should have been handled by the compiler, but by that time there were already metabuilders, so nobody bothered.

bigdict 6y ago

> It is accepted by all decent people that Make sucks and needs to die falsehood #1

fmajid 6y ago
Amen. The worst are crackpot systems like ant. Whoever thought a half-baked DSL with a XML concrete syntax was a good idea?
- deleted 6y ago
  [deleted]
- raverbashing 6y ago
  > ant. > Whoever thought a half-baked DSL with a XML concrete syntax was a good idea? You kinda answered your own question there no?
arauhala 6y ago
Preach to the choir! :) Make does so many things correctly: - it gives user total freedom to modify the built - yet it has excellent defaults for most situations - it is full blown programming language, - yet it's syntax is extremely specialized for the purpose and familiar at the same time (it's bash) - it's basic assumptions/structure are extremely simple (timestamps, dependencies) - yet it's extremely powerful and can take into account most situations. It's be extremely Unix-style tool with very simple building blocks that combine together in extremely powerful way. Its fast to learn and easy to master and as such, the best kind of design. Sometimes making a build with it is a small programming project, yet after using all kinds of built tools, I end up just wishing that I could use make instead.
- pydry 6y ago
  bash is a pretty terrible programming language and it's terribleness increases geometrically with size. This is a large part of why people don't like make I think.

rainbow-road 6y ago

Most of the listed assumptions are bad, but some of them, to me, seem essential. In particular, dealing with cyclic build graphs sounds like an absolute nightmare, and I doubt that any build system that allows for such a thing can be easy-to-use.

morelisp 6y ago

This "falsehoods programmers believe" is fundamentally different from the more classic ones (e.g. "about names") because its topic is also wholly controlled by programmers. That means in the end it doesn't share much with those documents, but can instead also be characterized as "just" a list of bugs in other programs. If a program doesn't handle my name, that's a problem with the program. My name is my name; the program doesn't decide that. Depending on the context some government might; the program also generally isn't supposed to decide for the government. Often in practice the program controls both of us; this is the core complaint of "falsehoods" articles, that a program artifact is overstepping its ethical/legal/otherwise extra-computational authority. But there's no moral demand that e.g. a program exit with 0 status even when it's failed, or that build trees should be cyclic. These are tools by programmers and for programmers concerning programs and when we run into issues it's usually because we chose a poor modeling tool (e.g. a build system should not be ideal to calculate the fixed point of a data processing pipeline, independent of any cycles) or because of a flat-out mistake (there's no reason a compiler can't be given some output directive other than inventor oversight). Other "falsehoods" are criticism of the software encoding the falsehood, but I read probably 90% of the complaints here as equal or greater failings on the tools invoked. Not only for the sake of my build system do I want DAG pipelines, updated timestamps, or meaningful exit codes - if anything these are the demands I wish to place on other software for the sake of my brain! More like "falsehoods compilers believe about their primacy in program generation." Or as one of the comments suggests, > build systems are a symptom of software languages that are not designed to build software systems

ex_amazon_sde 6y ago

In 2020 we can add additional falsehoods based on my experience in Amazon: - It's OK to embed/vendor dependencies - It's OK to hardcode VCS URLs for your dependencies - you can pull random stuff from the Internet at compile time and large companies will accept it - It's OK to release libraries without version numbering or using git commitish - Everybody uses containers - Nobody cares about reproducible builds - Nobody cares about security, license compliance and being able to rebuild things reliably - All FAANGS will use your project even if it cannot packaged into OS packages

Apofis 6y ago
This happens everywhere... some platforms became so complicated to do initial setup with that Docker became a necessity instead of an install script. It now takes me weeks to setup a fresh machine to do local dev on. I guess I should make a docker image of my dev box. cd devbox docker-compose up ...would be nice, now that I think about it.
- ex_amazon_sde 6y ago
  I suspect you missed the point about falsehoods.
blandflakes 6y ago
God the number of classpath conflicts I untangled there that were caused by some bullshit library bundling some ancient apache-commons jar...
ben0x539 6y ago
While I agree with a bunch of what you're saying, I expect that FAANG compatibility is not people's top priority when they're not getting paid FAANG salaries.

haolez 6y ago

I tend to use whichever build system is mainstream in the environment I'm in. However, if I'm dealing with something special, or if the build system is unstable, I always fall back to Make. Such a beautiful tool! Tup is also worth to mention: http://gittup.org/tup/ http://gittup.org/tup/

pwm 6y ago

scoot_718 6y ago

Some of those aren't believed assumptions, but asserted ones. For instance, I would think a build system that asserts that builds are acyclic is good. People who want a cyclic build are unimportant and can go elsewhere, while I enjoy my simpler build system because of it. Simplifying assumptions are good.