11 ms·

Parallel ./configure

219 points by brooke2k 1y ago

fmajid 1y ago
And on macOS, the notarization checks for all the conftest binaries generated by configure add even more latency. Apple reneged on their former promise to give an opt-out for this.
epistasis 1y ago
I've spent a fair amount of time over the past decades to make autotools work on my projects, and I've never felt like it was a good use of time. It's likely that C will continue to be used by everyone for decades to come, but I know that I'll personally never start a new project in C again. I'm still glad that there's some sort of push to make autotools suck less for legacy projects.
- tidwall 1y ago
  I've stopped using autotools for new projects. Just a Makefile, and the -j flag for concurrency.
- monkeyelite 1y ago
  You can use make without configure. If needed, you can also write your own configure instead of using auto tools. Creating a make file is about 10 lines and is the lowest friction for me to get programming of any environment. Familiarity is part of that.
  - edoceo 1y ago
    Write your own configure? For an internal project, where much is under domain control, sure. But for the 1000s of projects trying to multi-plarform and/or support flavours/versions - oh gosh.
    - monkeyelite 1y ago
      It depends on how much platform specific stuff you are trying to use. Also in 2025 most packages are tailored for the operating system by packagers - not the original authors. Autotools is going to check every config from the past 50 years.
      - charcircuit 1y ago
        >Also in 2025 most packages are tailored for the operating system by packagers - not the original authors. No? Most operating systems don't have a separate packager. They have the developer package the application.
        monkeyelite 1y ago
        Yes? Each operating system is very different and almost every package has patches or separate install scripts.
  - viraptor 1y ago
    It's a bit of a balance once you get bigger dependencies. A generic autoconf is annoying to write, but rarely an issue when packaging for a distro. Most issues I've had to fix in nixpkgs were for custom builds unfortunately. But if you don't plan to distribute things widely (or have no deps).. Whatever, just do what works for you.
- psyclobe 1y ago
  cmake ftw
  - aldanor 1y ago
    You mean cargo build
    - yjftsjthsd-h 1y ago
      ... can cargo build things that aren't rust? If yes, that's really cool. If no, then it's not really in the same problem domain.
      - kouteiheika 1y ago
        No it can't. It can build a Rust program (build.rs) which builds things that aren't Rust, but that's an entirely different use case (building non-Rust library to use inside of Rust programs).
      - crabbone 1y ago
        There's GprBuild (Ada tool) that can build C (not sure about C++). It also has more elaborate configuration structure, but I didn't use it extensively to tell what exactly and how exactly does it do it. In combination with Alire it can also manage dependencies Cargo-style.
        touisteur 1y ago
        Got it to build C++, CUDA and IIRC SYCL too.
  - malkia 1y ago
    cmake uses configure, or configure-like too!
    - ahartmetz 1y ago
      Same concept, but completely different implementation.
- eqvinox 1y ago
  To extend on sibling comments: autoconf is in no way, shape or form an "official" build system associated with C. It is a GNU creation and certainly popular, but not to a "monopoly" degree, and it's share is declining. (plain make & meson & cmake being popular alternatives)
blibble 1y ago
is this really a big deal given you run ./configure once? it's like systemd trading off non-determinism for boot speed, when it takes 5 minutes to get through the POST
- LegionMammal978 1y ago
  If you do a lot of bisecting, or bootstrapping, or building compatibility matrices, or really anything that needs you to compile lots of old versions, the repeated ./configure steps really start feeling like a drag.
  - kazinator 1y ago
    In a "reasonably well-behaved program", if you have the artifacts from a current configure, like a "config.h" header, they are compatible with older commits, even if configurations changed, as long as the configuration changes were additive: introducing some new test, along with a new symbol in "config.h". It's possible to skip some of the ./configure steps. Especially for someone who knows the program very well.
    - LegionMammal978 1y ago
      Perhaps you can get away with that for small, young, or self-contained projects. But for medium-to-large projects running more than a few years, the (different versions of) external or vendored dependencies tend to come and go, and they all have their own configurations. Long-running projects are also prone to internal reorganizations and overhauls to the build system. (Go back far enough, and you're having to wrangle patchsets for every few months' worth of versions since -fpermissive is no longer permissive enough to get it to build.)
      - deleted 1y ago
        [deleted]
      - kazinator 1y ago
        > Perhaps you can get away with that for small, young, or self-contained projects Only a small subset of which could just switch to the parallel configure being proposed.
fishgoesblub 1y ago
Very nice! I always get annoyed when my fancy 16 thread CPU is left barely used as one thread is burning away with the rest sitting and waiting. Bookmarking this for later to play around with whatever projects I use that still use configure. Also, I was surprised when the animated text at the top of the article wasn't a gif, but actual text. So cool!
deleted 1y ago
[deleted]
SuperV1234 1y ago
CMake also needs this, badly...
- torarnv 1y ago
  Agreed! The CMake Xcode generator is extremely slow because not only is it running the configure tests sequentially, but it generates a new Xcode project for each of them.
redleader55 1y ago
Why do we need to even run most of the things in ./configure? Why not just have a file in /etc which is updated when you install various packages which ./configure can read to learn various stats about the environment? Obviously it will still allow setting various things with parameters and create a Makefile, but much faster.
- o11c 1y ago
  Keep in mind that the build intentionally depends on environment variables, people often install non-packaged dependencies in bad ways, and cross-compiling is a thing, so it's not that simple.
- wolfgang42 1y ago
  Some relevant discussion/pointers to other notes on this sort of proposal can be found here: https://utcc.utoronto.ca/~cks/space/blog/sysadmin/AutoconfValuableFeatures?showcomments#comments https://utcc.utoronto.ca/~cks/space/blog/sysadmin/AutoconfVa... (The conclusion I distilled out of reading that at the time, I think, was that this is actually sort of happening, but slowly, and autoconf is likely to stick around for a while, if only as a compatibility layer during the transition.)
- pabs3 1y ago
  Not every OS is going to have such a file, and you also don't know if it matches the actual system ./configure runs on.
- 1718627440 1y ago
  That does already exist in GNU Autoconf: https://www.gnu.org/software/autoconf/manual/autoconf-2.65/html_node/Site-Defaults.html#Site-Defaults https://www.gnu.org/software/autoconf/manual/autoconf-2.65/h...
creatonez 1y ago
Noticed an easter egg in this article. The text below "I'm sorry, but in the year 2025, this is ridiculous:" is animated entirely without Javascript or .gif files. It's pure CSS. This is how it was done: https://github.com/tavianator/tavianator.com/blob/cf0e4ef26df95140a398257aec705e990cd4df0b/src/2025/configure.md?plain=1#L14-L31 https://github.com/tavianator/tavianator.com/blob/cf0e4ef26d...
- o11c 1y ago
  Unfortunately it forgets to HTML-escape the <wchar.h> etc.
  - tavianator 1y ago
    Whoops! Forgot to do that when I switched from a ``` block to raw html
    - tavianator 1y ago
      Fixed! https://github.com/tavianator/tavianator.com/commit/aa9d99d5de55a6f7d7138df6a64b01c435f87f0e https://github.com/tavianator/tavianator.com/commit/aa9d99d5...
moralestapia 1y ago
>The purpose of a ./configure script is basically to run the compiler a bunch of times and check which runs succeeded. Wait is this true? (!)
- Am4TIfIsER0ppos 1y ago
  Yes.
- klysm 1y ago
  The closer and deeper you look into the C toolchains the more grossed out you’ll be
  - acuozzo 1y ago
    Hands have to get dirty somewhere. "As deep as The Worker's City lay underground, so high above towered the City of Metropolis." The choices are: 1. Restrict the freedom of CPU designers to some approximation of the PDP11. No funky DSP chips. No crazy vector processors. 2. Restrict the freedom of OS designers to some approximation of Unix. No bespoke realtime OSes. No research OSes. 3. Insist programmers use a new programming language for these chips and OSes. (This was the case prior to C and Unix.) 4. Insist programmers write in assembly and/or machine code. Perhaps a macro-assembler is acceptable here, but this is inching toward C. The cost of this flexibility is gross tooling to make it manageable. Can it be done without years and years of accrued M4 and sh? Perhaps, but that's just CMake and CMake is nowhere near as capable as Autotools & friends are when working with legacy platforms.
    - klysm 1y ago
      There is no real technical justification for the absolute shit show that is the modern C toolchain
      - moralestapia 1y ago
        I like C/C++ a lot, A LOT, and I agree with your comment. Man, if this got fixed it would be one of the best languages to develop for. My wishlist: * Quick compilation times (obv.) or some sort of tool that makes it feel like an interpreted language, at least when you're developing, then do the usual compile step to get an optimized binary. * A F...... CLEAR AND CONSISTENT WAY TO TELL THE TOOLCHAIN THIS LIBRARY IS HERE AND THIS ONE IS OVER THERE (sorry but, come on ...). * A single command line argument to output a static binary. * Anything that gets us closer to the "build-once run-anywhere" philosophy of "Cosmopolitan Libc". Even if an intermediate execution layer is needed. One could say, "oh, but this is C, not Java", but it is already de facto a broken Java, because you still need an execution layer, call it stdlib, GLIB, whatever, if those shared libraries are not on your system with their exact version matching, your program breaks ... Just stop pretending and ship the "C virtual machine", lmao.
BobbyTables2 1y ago
I was really hoping he worked some autoreconf/macro magic to transform existing configure.ac files into a parallelized result. Nice writeup though.
psyclobe 1y ago
(Luckily?) With c++ your build will nearly always take longer then the configuration step.
- bitbasher 1y ago
  rust: "hold my leggings"
  - LoganDark 1y ago
    Since when? I far more often run into CMake taking ages than Cargo.
klysm 1y ago
autotools is a complete disaster. It’s mind boggling to think that everything we build is usually on top of this arcane system
gorgoiler 1y ago
On the topic* of having 24 cores and wanting to put them to work: when I were a lad the promise was that pure functional programming would trivially allow for parallel execution of functions. Has this future ever materialized in a modern language / runtime? x = 2 + 2 y = 2 * 2 z = f(x, y) print(z) …where x and y evaluate in parallel without me having to do anything. Clojure, perhaps? *And superficially off the topic of this thread, but possibly not.
- speed_spread 1y ago
  I believe it's not the language preventing it but the nature of parallel computing. The overhead of splitting up things and then reuniting them again is high enough to make trivial cases not worth it. OTOH we now have pretty good compiler autovectorization which does a lot of parallel magic if you set things right. But it's not handled at the language level either.
- deepsun 1y ago
  Sure, Tensorflow and Pytorch, here ya go :)
- chubot 1y ago
  That looks more like a SIMD problem than a multi-core problem You want bigger units of work for multiple cores, otherwise the coordination overhead will outweigh the work the application is doing I think the Erlang runtime is probably the best use of functional programming and multiple cores. Since Erlang processes are shared nothing, I think they will scale to 64 or 128 cores just fine Whereas the GC will be a bottleneck in most languages with shared memory ... you will stop scaling before using all your cores But I don't think Erlang is as fine-grained as your example ... Some related threads: https://news.ycombinator.com/item?id=40130079 https://news.ycombinator.com/item?id=40130079 https://news.ycombinator.com/item?id=31176264 https://news.ycombinator.com/item?id=31176264 AFAIU Erlang is not that fast an interpreter; I thought the Pony Language was doing something similar (shared nothing?) with compiled code, but I haven't heard about it in awhile
  - juped 1y ago
malkia 1y ago
"./configure" has always been the wrong thing for a very long long time. Also slow...
codys 1y ago
I did something like the system described in this article a few years back. [1] Instead of splitting the "configure" and "make" steps though, I chose to instead fold much of the "configure" step into the "make". To clarify, this article describes a system where `./configure` runs a bunch of compilations in parallel, then `make` does stuff depending on those compilations. If one is willing to restrict what the configure can detect/do to writing to header files (rather than affecting variables examined/used in a Makefile), then instead one can have `./configure` generate a `Makefile` (or in my case, a ninja file), and then have the "run the compiler to see what defines to set" and "run compiler to build the executable" can be run in a single `make` or `ninja` invocation. The simple way here results in _almost_ the same behavior: all the "configure"-like stuff running and then all the "build" stuff running. But if one is a bit more careful/clever and doesn't depend on the entire "config.h" for every "<real source>.c" compilation, then one can start to interleave the work perceived as "configuration" with that seen as "build". (I did not get that fancy) [1]: https://github.com/codyps/cninja/tree/master/config_h https://github.com/codyps/cninja/tree/master/config_h
- tavianator 1y ago
  Nice! I used to do something similar, don't remember exactly why I had to switch but the two step process did become necessary at some point. Just from a quick peek at that repo, nowadays you can write #if __has_attribute(cold) and avoid the configure test entirely. Probably wasn't a thing 10 years ago though :)
  - codys 1y ago
    yep. C's really come a long way with the special operators for checking if attributes exist, if builtins exist, if headers exist, etc. Covers a very large part of what is needed, making fewer and fewer things need to end up in configure scripts. I think most of what's left is checking for items (types, functions) existence and their shape, as you were doing :). I can dream about getting a nice special operator to check for fields/functions, would let us remove even more from configure time, but I suspect we won't because that requires type resolution and none of the existing special operators do that.
andreyv 1y ago
Autoconf can use cache files [1], which can greatly speed up repeated configures. With cache, a test is run at most once. [1] https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.72/html_node/Cache-Files.html https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/a...
- fanf2 1y ago
  Sadly the cache files don’t record enough about the environment to be usable if you change configure options. They are generally unreliable.
iforgotpassword 1y ago
The other issue is that people seem to just copy configure/autotools scripts over from older or other projects because either they are lazy or don't understand them enough to do it themselves. The result is that even with relatively modern code bases that only target something like x86, arm and maybe mips and only gcc/clang, you still get checks for the size of an int, or which header is needed for printf, or whether long long exists.... And then the entire code base never checks the generated macros in a single place, uses int64_t and never checks for stint.h in the configure script...
- rbanffy 1y ago
  It’s always wise to be specific about the sizes you want for your variables. You don’t want your ancient 64-bit code to act differently on your grandkids 128-bit laptops. Unless, of course, you want to let the compiler decide whether to leverage higher precision types that become available after you retire.
- IshKebab 1y ago
  I don't think it's fair to say "because they are lazy or don't understand". Who would want to understand that mess? It isn't a virtue. A fairer criticism would be that they have no sense to use a more sane build system. CMake is a mess but even that is faaaaar saner than autotools, and probably more popular at this point.
  - xiaoyu2006 1y ago
    Autotools use M4 to meta-program a bash script that meta-programs a bunch of C(++) sources and generates C(++) sources that utilizes meta-programming for different configurations; after which the meta-programmed script, again, meta-programs monolithic makefiles. This is peak engineering.
    - krior 1y ago
      Sounds like a headache. Is there a nice Python lib to generate all this M4-mumbo-jumbo?
amelius 1y ago
What I'd like to see is a configure with guarantees that if the configure succeeds, then the build will succeed too.
Chocimier 1y ago
It is possible in theory to speed up existing configure scripts by switching interpreter from /bin/sh to something that scans file, splits it to independent blocks and runs them in parallel. Is there any such previous work?
- tavianator 1y ago
  I was looking for a relevant paper and found this one, which isn't what I was looking for but is related: https://sigops.org/s/conferences/hotos/2023/papers/liargkovas.pdf https://sigops.org/s/conferences/hotos/2023/papers/liargkova...
rbanffy 1y ago
I get the impression configure not only runs sequentially, but incrementally, where previous results can change the results of tests run later. Were it just sequential, running multiple tests as separate processes would be relatively simple. Also, you shouldn’t need to run ./configure every time you run make.
- fmajid 1y ago
  No, but if you are doing something like rebuilding a distro's worth of packages from source from scratch, the configure step starts to dominate. I build around 550, and it takes around 6 hours on a single node. Most checks are common, so what can help is having a shared cache for all configure scripts so if you have 400 packages to rebuild, it doesn't check 400 times if you should use flock or fcntl. This approach is described here: https://jmmv.dev/2022/06/autoconf-caching.html https://jmmv.dev/2022/06/autoconf-caching.html It doesn't help that autoconf is basically abandonware, with one forlorn maintainer trying to resuscitate it, but creating major regressions with new releases: https://lwn.net/Articles/834682/ https://lwn.net/Articles/834682/
  - rbanffy 1y ago
    > It doesn't help that autoconf is basically abandonware A far too common tragedy of our age.
    - pdimitar 1y ago
      I don't disagree with that general premise but IMO autotools being (gradually?) abandoned is logical. It served its purpose. Not saying it's still not very useful in the darker shadows of technology but for a lot of stuff people choose Zig, Rust, Golang etc. today, with fairly good reasons too, and those PLs usually have fairly good packaging and dependency management and building subsystems built-in. Furthermore, there really has to be a better way to do what autotools is doing, no? Sure, there are some situations where you only have some bare sh shell and nothing much else but I'd venture to say that in no less than 90% of all cases you can very easily have much more stuff installed -- like the `just` task runner tool, for example, that solves most of the problems that `make` usually did. If we are talking in terms of our age, we also have to take into account that there's too much software everywhere! I believe some convergence has to start happening. There is such a thing as too much freedom. We are dispersing so much creative energy for almost no benefit of humankind...
tmtvl 1y ago
As a user I highly appreciate ./configure for the --help flag, which usually tells me how to build a program with or without particular functionalities which may or may not be applicable to my use-case.
saagarjha 1y ago
I actually think this is possible to improve if you have the autoconf files. You could parse it to find all the checks you know can run in parallel and run those.
mrrogot69 1y ago
Good idea!
tekknolagi 1y ago
This doesn't mention another use of configure which is manually enabling or disabling features via --with-X -- I might send in a PR for that
- tavianator 1y ago
  My actual "production" implementation of this concept does support that: https://github.com/tavianator/bfs/blob/main/configure https://github.com/tavianator/bfs/blob/main/configure But I wanted the blog post sized version to be simpler for exposition.
  - tekknolagi 1y ago
    Very nice
gitroom 1y ago
Man, I've spent way too many hours wrestling with build systems like autotools and cmake and they both make me want to just toss my laptop sometimes - feels like it's way harder than it needs to be each time. You ever think we'll actually get to a point where building stuff just works, no endless config scripts or chasing weird cross-platform bugs?
- pdimitar 1y ago
  I asked myself that probably no less than 200 times. Thinking in terms of technological problems, that should be a 100% solved problem at this point! Build a DAG of all tasks and just solve it and invoke stuff, right? Well, not exactly. A lot of the build systems don't allow you to specify if something is safe to execute in parallel. And that's important because sometimes even though it seems three separate tasks are completely independent and can be executed in parallel they'd still share f.ex. a filesystem-level cache and would compete for it and likely corrupt it, so... not so fast. :( But I feel the entire tech sector is collectively dragging their feet on this. We should be able to devise a good build system that allows us to encode more constraints and requirements than the current ones, and then simply build a single solver for its DAG and be done with it. How frakkin difficult can that be? Of course the problem is actually social, not technical. Meaning that most programmers wouldn't ever migrate if the decision was left to them. I still would not care about them though; if I had the free time and energy then I would absolutely work on it. It's something that might swallow a lot of energy but if solved properly (meaning it has to be reasonably extensive without falling into the trap that no two projects would even use the same implementation -- let us NOT do that!) then it will be solved only once and never again. We can dream. But again, it very much does seem like a very solvable problem to me.
db48x 1y ago
Wrong solution. Just run ‘configure -C’ so that it caches the results, or reuses the cache if it already exists. And of course most of the time you don't need to rerun configure at all, just make.
kazinator 1y ago
I've implemented a configuration caching mechanism for myself (in one important project) which stores configuration artifacts in a cache directory, associated by the commit hash. It works as a git hook: $ git bisect good Bisecting: 7 revisions left to test after this (roughly 3 steps) restored cached configuration for 2f8679c346a88c07b81ea8e9854c71dae2ade167 [2f8679c346a88c07b81ea8e9854c71dae2ade167] expander: noexpand mechanism. The "restored cached configuration" message is from the git hook. What it's not saying is that it also saved the config for the commit it is navigating away from. I primed the cache by executing a "git checkout" for each of a range of commits. Going forward, it will populate itself. This is the only issue I would conceivably care about with regard to configure performance. When not navigating in git history, I do not often run configure. Downstream distros do not care; they keep their machines and cores busy by building multiple packages in parallel. It's not ideal because the cache from one host is not applicable to another; you can't port it. I could write an intelligent script to populate it, which basically identifies commits (within some specified range) that have touched the config system, and then assumes that for all in-between commits, it's the same. The hook could do this. When it notices that the current sha doesn't have a cached configuration, it could search backwards through history for the most recent commit which does have it. If the configure script (or something influencing it) has not been touched since that commit, then its cached material can be populated for all in-between commits right through the current one. That would take care of large swaths of commits in a typical bisect session.
- kazinator 1y ago
  The right way to do this is not to rely on the git hashes, but to hash the inputs into the configuration system (those that are in version control, not the implicit environmental inputs from the platform). For instance, if the only input to the configuration system is the body of the configure script, then we hash that. That is then our key to the generated materials.

219 points by brooke2k 1y ago

And on macOS, the notarization checks for all the conftest binaries generated by configure add even more latency. Apple reneged on their former promise to give an opt-out for this.

epistasis 1y ago

I've spent a fair amount of time over the past decades to make autotools work on my projects, and I've never felt like it was a good use of time. It's likely that C will continue to be used by everyone for decades to come, but I know that I'll personally never start a new project in C again. I'm still glad that there's some sort of push to make autotools suck less for legacy projects.

tidwall 1y ago
I've stopped using autotools for new projects. Just a Makefile, and the -j flag for concurrency.
monkeyelite 1y ago
You can use make without configure. If needed, you can also write your own configure instead of using auto tools. Creating a make file is about 10 lines and is the lowest friction for me to get programming of any environment. Familiarity is part of that.
- edoceo 1y ago
  Write your own configure? For an internal project, where much is under domain control, sure. But for the 1000s of projects trying to multi-plarform and/or support flavours/versions - oh gosh.
  - monkeyelite 1y ago
    It depends on how much platform specific stuff you are trying to use. Also in 2025 most packages are tailored for the operating system by packagers - not the original authors. Autotools is going to check every config from the past 50 years.
    - charcircuit 1y ago
      >Also in 2025 most packages are tailored for the operating system by packagers - not the original authors. No? Most operating systems don't have a separate packager. They have the developer package the application.
      - monkeyelite 1y ago
        Yes? Each operating system is very different and almost every package has patches or separate install scripts.
- viraptor 1y ago
  It's a bit of a balance once you get bigger dependencies. A generic autoconf is annoying to write, but rarely an issue when packaging for a distro. Most issues I've had to fix in nixpkgs were for custom builds unfortunately. But if you don't plan to distribute things widely (or have no deps).. Whatever, just do what works for you.
psyclobe 1y ago
cmake ftw
- aldanor 1y ago
  You mean cargo build
  - yjftsjthsd-h 1y ago
    ... can cargo build things that aren't rust? If yes, that's really cool. If no, then it's not really in the same problem domain.
    - kouteiheika 1y ago
      No it can't. It can build a Rust program (build.rs) which builds things that aren't Rust, but that's an entirely different use case (building non-Rust library to use inside of Rust programs).
    - crabbone 1y ago
      There's GprBuild (Ada tool) that can build C (not sure about C++). It also has more elaborate configuration structure, but I didn't use it extensively to tell what exactly and how exactly does it do it. In combination with Alire it can also manage dependencies Cargo-style.
      - touisteur 1y ago
        Got it to build C++, CUDA and IIRC SYCL too.
- malkia 1y ago
  cmake uses configure, or configure-like too!
  - ahartmetz 1y ago
    Same concept, but completely different implementation.
eqvinox 1y ago
To extend on sibling comments: autoconf is in no way, shape or form an "official" build system associated with C. It is a GNU creation and certainly popular, but not to a "monopoly" degree, and it's share is declining. (plain make & meson & cmake being popular alternatives)

blibble 1y ago

is this really a big deal given you run ./configure once? it's like systemd trading off non-determinism for boot speed, when it takes 5 minutes to get through the POST

LegionMammal978 1y ago
If you do a lot of bisecting, or bootstrapping, or building compatibility matrices, or really anything that needs you to compile lots of old versions, the repeated ./configure steps really start feeling like a drag.
- kazinator 1y ago
  In a "reasonably well-behaved program", if you have the artifacts from a current configure, like a "config.h" header, they are compatible with older commits, even if configurations changed, as long as the configuration changes were additive: introducing some new test, along with a new symbol in "config.h". It's possible to skip some of the ./configure steps. Especially for someone who knows the program very well.
  - LegionMammal978 1y ago
    Perhaps you can get away with that for small, young, or self-contained projects. But for medium-to-large projects running more than a few years, the (different versions of) external or vendored dependencies tend to come and go, and they all have their own configurations. Long-running projects are also prone to internal reorganizations and overhauls to the build system. (Go back far enough, and you're having to wrangle patchsets for every few months' worth of versions since -fpermissive is no longer permissive enough to get it to build.)
    - deleted 1y ago
      [deleted]
    - kazinator 1y ago
      > Perhaps you can get away with that for small, young, or self-contained projects Only a small subset of which could just switch to the parallel configure being proposed.

fishgoesblub 1y ago

Very nice! I always get annoyed when my fancy 16 thread CPU is left barely used as one thread is burning away with the rest sitting and waiting. Bookmarking this for later to play around with whatever projects I use that still use configure. Also, I was surprised when the animated text at the top of the article wasn't a gif, but actual text. So cool!

deleted 1y ago

[deleted]

SuperV1234 1y ago

CMake also needs this, badly...

torarnv 1y ago
Agreed! The CMake Xcode generator is extremely slow because not only is it running the configure tests sequentially, but it generates a new Xcode project for each of them.

redleader55 1y ago

Why do we need to even run most of the things in ./configure? Why not just have a file in /etc which is updated when you install various packages which ./configure can read to learn various stats about the environment? Obviously it will still allow setting various things with parameters and create a Makefile, but much faster.

o11c 1y ago
Keep in mind that the build intentionally depends on environment variables, people often install non-packaged dependencies in bad ways, and cross-compiling is a thing, so it's not that simple.
wolfgang42 1y ago
Some relevant discussion/pointers to other notes on this sort of proposal can be found here: https://utcc.utoronto.ca/~cks/space/blog/sysadmin/AutoconfValuableFeatures?showcomments#comments https://utcc.utoronto.ca/~cks/space/blog/sysadmin/AutoconfVa... (The conclusion I distilled out of reading that at the time, I think, was that this is actually sort of happening, but slowly, and autoconf is likely to stick around for a while, if only as a compatibility layer during the transition.)
pabs3 1y ago
Not every OS is going to have such a file, and you also don't know if it matches the actual system ./configure runs on.
1718627440 1y ago
That does already exist in GNU Autoconf: https://www.gnu.org/software/autoconf/manual/autoconf-2.65/html_node/Site-Defaults.html#Site-Defaults https://www.gnu.org/software/autoconf/manual/autoconf-2.65/h...

creatonez 1y ago

Noticed an easter egg in this article. The text below "I'm sorry, but in the year 2025, this is ridiculous:" is animated entirely without Javascript or .gif files. It's pure CSS. This is how it was done: https://github.com/tavianator/tavianator.com/blob/cf0e4ef26df95140a398257aec705e990cd4df0b/src/2025/configure.md?plain=1#L14-L31 https://github.com/tavianator/tavianator.com/blob/cf0e4ef26d...

o11c 1y ago
Unfortunately it forgets to HTML-escape the <wchar.h> etc.
- tavianator 1y ago
  Whoops! Forgot to do that when I switched from a ``` block to raw html
  - tavianator 1y ago
    Fixed! https://github.com/tavianator/tavianator.com/commit/aa9d99d5de55a6f7d7138df6a64b01c435f87f0e https://github.com/tavianator/tavianator.com/commit/aa9d99d5...

moralestapia 1y ago

>The purpose of a ./configure script is basically to run the compiler a bunch of times and check which runs succeeded. Wait is this true? (!)

Am4TIfIsER0ppos 1y ago
Yes.
klysm 1y ago
The closer and deeper you look into the C toolchains the more grossed out you’ll be
- acuozzo 1y ago
  Hands have to get dirty somewhere. "As deep as The Worker's City lay underground, so high above towered the City of Metropolis." The choices are: 1. Restrict the freedom of CPU designers to some approximation of the PDP11. No funky DSP chips. No crazy vector processors. 2. Restrict the freedom of OS designers to some approximation of Unix. No bespoke realtime OSes. No research OSes. 3. Insist programmers use a new programming language for these chips and OSes. (This was the case prior to C and Unix.) 4. Insist programmers write in assembly and/or machine code. Perhaps a macro-assembler is acceptable here, but this is inching toward C. The cost of this flexibility is gross tooling to make it manageable. Can it be done without years and years of accrued M4 and sh? Perhaps, but that's just CMake and CMake is nowhere near as capable as Autotools & friends are when working with legacy platforms.
  - klysm 1y ago
    There is no real technical justification for the absolute shit show that is the modern C toolchain
    - moralestapia 1y ago
      I like C/C++ a lot, A LOT, and I agree with your comment. Man, if this got fixed it would be one of the best languages to develop for. My wishlist: * Quick compilation times (obv.) or some sort of tool that makes it feel like an interpreted language, at least when you're developing, then do the usual compile step to get an optimized binary. * A F...... CLEAR AND CONSISTENT WAY TO TELL THE TOOLCHAIN THIS LIBRARY IS HERE AND THIS ONE IS OVER THERE (sorry but, come on ...). * A single command line argument to output a static binary. * Anything that gets us closer to the "build-once run-anywhere" philosophy of "Cosmopolitan Libc". Even if an intermediate execution layer is needed. One could say, "oh, but this is C, not Java", but it is already de facto a broken Java, because you still need an execution layer, call it stdlib, GLIB, whatever, if those shared libraries are not on your system with their exact version matching, your program breaks ... Just stop pretending and ship the "C virtual machine", lmao.

BobbyTables2 1y ago

I was really hoping he worked some autoreconf/macro magic to transform existing configure.ac files into a parallelized result. Nice writeup though.

psyclobe 1y ago

(Luckily?) With c++ your build will nearly always take longer then the configuration step.

bitbasher 1y ago
rust: "hold my leggings"
- LoganDark 1y ago
  Since when? I far more often run into CMake taking ages than Cargo.

klysm 1y ago

autotools is a complete disaster. It’s mind boggling to think that everything we build is usually on top of this arcane system

gorgoiler 1y ago

On the topic* of having 24 cores and wanting to put them to work: when I were a lad the promise was that pure functional programming would trivially allow for parallel execution of functions. Has this future ever materialized in a modern language / runtime? x = 2 + 2 y = 2 * 2 z = f(x, y) print(z) …where x and y evaluate in parallel without me having to do anything. Clojure, perhaps? *And superficially off the topic of this thread, but possibly not.

speed_spread 1y ago
I believe it's not the language preventing it but the nature of parallel computing. The overhead of splitting up things and then reuniting them again is high enough to make trivial cases not worth it. OTOH we now have pretty good compiler autovectorization which does a lot of parallel magic if you set things right. But it's not handled at the language level either.
deepsun 1y ago
Sure, Tensorflow and Pytorch, here ya go :)
chubot 1y ago
That looks more like a SIMD problem than a multi-core problem You want bigger units of work for multiple cores, otherwise the coordination overhead will outweigh the work the application is doing I think the Erlang runtime is probably the best use of functional programming and multiple cores. Since Erlang processes are shared nothing, I think they will scale to 64 or 128 cores just fine Whereas the GC will be a bottleneck in most languages with shared memory ... you will stop scaling before using all your cores But I don't think Erlang is as fine-grained as your example ... Some related threads: https://news.ycombinator.com/item?id=40130079 https://news.ycombinator.com/item?id=40130079 https://news.ycombinator.com/item?id=31176264 https://news.ycombinator.com/item?id=31176264 AFAIU Erlang is not that fast an interpreter; I thought the Pony Language was doing something similar (shared nothing?) with compiled code, but I haven't heard about it in awhile
- juped 1y ago

malkia 1y ago

"./configure" has always been the wrong thing for a very long long time. Also slow...

codys 1y ago

I did something like the system described in this article a few years back. [1] Instead of splitting the "configure" and "make" steps though, I chose to instead fold much of the "configure" step into the "make". To clarify, this article describes a system where `./configure` runs a bunch of compilations in parallel, then `make` does stuff depending on those compilations. If one is willing to restrict what the configure can detect/do to writing to header files (rather than affecting variables examined/used in a Makefile), then instead one can have `./configure` generate a `Makefile` (or in my case, a ninja file), and then have the "run the compiler to see what defines to set" and "run compiler to build the executable" can be run in a single `make` or `ninja` invocation. The simple way here results in _almost_ the same behavior: all the "configure"-like stuff running and then all the "build" stuff running. But if one is a bit more careful/clever and doesn't depend on the entire "config.h" for every "<real source>.c" compilation, then one can start to interleave the work perceived as "configuration" with that seen as "build". (I did not get that fancy) [1]: https://github.com/codyps/cninja/tree/master/config_h https://github.com/codyps/cninja/tree/master/config_h

tavianator 1y ago
Nice! I used to do something similar, don't remember exactly why I had to switch but the two step process did become necessary at some point. Just from a quick peek at that repo, nowadays you can write #if __has_attribute(cold) and avoid the configure test entirely. Probably wasn't a thing 10 years ago though :)
- codys 1y ago
  yep. C's really come a long way with the special operators for checking if attributes exist, if builtins exist, if headers exist, etc. Covers a very large part of what is needed, making fewer and fewer things need to end up in configure scripts. I think most of what's left is checking for items (types, functions) existence and their shape, as you were doing :). I can dream about getting a nice special operator to check for fields/functions, would let us remove even more from configure time, but I suspect we won't because that requires type resolution and none of the existing special operators do that.

andreyv 1y ago

Autoconf can use cache files [1], which can greatly speed up repeated configures. With cache, a test is run at most once. [1] https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.72/html_node/Cache-Files.html https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/a...

fanf2 1y ago
Sadly the cache files don’t record enough about the environment to be usable if you change configure options. They are generally unreliable.

iforgotpassword 1y ago

The other issue is that people seem to just copy configure/autotools scripts over from older or other projects because either they are lazy or don't understand them enough to do it themselves. The result is that even with relatively modern code bases that only target something like x86, arm and maybe mips and only gcc/clang, you still get checks for the size of an int, or which header is needed for printf, or whether long long exists.... And then the entire code base never checks the generated macros in a single place, uses int64_t and never checks for stint.h in the configure script...

rbanffy 1y ago
It’s always wise to be specific about the sizes you want for your variables. You don’t want your ancient 64-bit code to act differently on your grandkids 128-bit laptops. Unless, of course, you want to let the compiler decide whether to leverage higher precision types that become available after you retire.
IshKebab 1y ago
I don't think it's fair to say "because they are lazy or don't understand". Who would want to understand that mess? It isn't a virtue. A fairer criticism would be that they have no sense to use a more sane build system. CMake is a mess but even that is faaaaar saner than autotools, and probably more popular at this point.
- xiaoyu2006 1y ago
  Autotools use M4 to meta-program a bash script that meta-programs a bunch of C(++) sources and generates C(++) sources that utilizes meta-programming for different configurations; after which the meta-programmed script, again, meta-programs monolithic makefiles. This is peak engineering.
  - krior 1y ago
    Sounds like a headache. Is there a nice Python lib to generate all this M4-mumbo-jumbo?

amelius 1y ago

What I'd like to see is a configure with guarantees that if the configure succeeds, then the build will succeed too.

Chocimier 1y ago

It is possible in theory to speed up existing configure scripts by switching interpreter from /bin/sh to something that scans file, splits it to independent blocks and runs them in parallel. Is there any such previous work?

tavianator 1y ago
I was looking for a relevant paper and found this one, which isn't what I was looking for but is related: https://sigops.org/s/conferences/hotos/2023/papers/liargkovas.pdf https://sigops.org/s/conferences/hotos/2023/papers/liargkova...

rbanffy 1y ago

I get the impression configure not only runs sequentially, but incrementally, where previous results can change the results of tests run later. Were it just sequential, running multiple tests as separate processes would be relatively simple. Also, you shouldn’t need to run ./configure every time you run make.

fmajid 1y ago
No, but if you are doing something like rebuilding a distro's worth of packages from source from scratch, the configure step starts to dominate. I build around 550, and it takes around 6 hours on a single node. Most checks are common, so what can help is having a shared cache for all configure scripts so if you have 400 packages to rebuild, it doesn't check 400 times if you should use flock or fcntl. This approach is described here: https://jmmv.dev/2022/06/autoconf-caching.html https://jmmv.dev/2022/06/autoconf-caching.html It doesn't help that autoconf is basically abandonware, with one forlorn maintainer trying to resuscitate it, but creating major regressions with new releases: https://lwn.net/Articles/834682/ https://lwn.net/Articles/834682/
- rbanffy 1y ago
  > It doesn't help that autoconf is basically abandonware A far too common tragedy of our age.
  - pdimitar 1y ago
    I don't disagree with that general premise but IMO autotools being (gradually?) abandoned is logical. It served its purpose. Not saying it's still not very useful in the darker shadows of technology but for a lot of stuff people choose Zig, Rust, Golang etc. today, with fairly good reasons too, and those PLs usually have fairly good packaging and dependency management and building subsystems built-in. Furthermore, there really has to be a better way to do what autotools is doing, no? Sure, there are some situations where you only have some bare sh shell and nothing much else but I'd venture to say that in no less than 90% of all cases you can very easily have much more stuff installed -- like the `just` task runner tool, for example, that solves most of the problems that `make` usually did. If we are talking in terms of our age, we also have to take into account that there's too much software everywhere! I believe some convergence has to start happening. There is such a thing as too much freedom. We are dispersing so much creative energy for almost no benefit of humankind...

tmtvl 1y ago

As a user I highly appreciate ./configure for the --help flag, which usually tells me how to build a program with or without particular functionalities which may or may not be applicable to my use-case.

saagarjha 1y ago

I actually think this is possible to improve if you have the autoconf files. You could parse it to find all the checks you know can run in parallel and run those.

mrrogot69 1y ago

Good idea!

tekknolagi 1y ago

This doesn't mention another use of configure which is manually enabling or disabling features via --with-X -- I might send in a PR for that

tavianator 1y ago
My actual "production" implementation of this concept does support that: https://github.com/tavianator/bfs/blob/main/configure https://github.com/tavianator/bfs/blob/main/configure But I wanted the blog post sized version to be simpler for exposition.
- tekknolagi 1y ago
  Very nice

gitroom 1y ago

Man, I've spent way too many hours wrestling with build systems like autotools and cmake and they both make me want to just toss my laptop sometimes - feels like it's way harder than it needs to be each time. You ever think we'll actually get to a point where building stuff just works, no endless config scripts or chasing weird cross-platform bugs?

pdimitar 1y ago
I asked myself that probably no less than 200 times. Thinking in terms of technological problems, that should be a 100% solved problem at this point! Build a DAG of all tasks and just solve it and invoke stuff, right? Well, not exactly. A lot of the build systems don't allow you to specify if something is safe to execute in parallel. And that's important because sometimes even though it seems three separate tasks are completely independent and can be executed in parallel they'd still share f.ex. a filesystem-level cache and would compete for it and likely corrupt it, so... not so fast. :( But I feel the entire tech sector is collectively dragging their feet on this. We should be able to devise a good build system that allows us to encode more constraints and requirements than the current ones, and then simply build a single solver for its DAG and be done with it. How frakkin difficult can that be? Of course the problem is actually social, not technical. Meaning that most programmers wouldn't ever migrate if the decision was left to them. I still would not care about them though; if I had the free time and energy then I would absolutely work on it. It's something that might swallow a lot of energy but if solved properly (meaning it has to be reasonably extensive without falling into the trap that no two projects would even use the same implementation -- let us NOT do that!) then it will be solved only once and never again. We can dream. But again, it very much does seem like a very solvable problem to me.

db48x 1y ago

Wrong solution. Just run ‘configure -C’ so that it caches the results, or reuses the cache if it already exists. And of course most of the time you don't need to rerun configure at all, just make.

kazinator 1y ago

I've implemented a configuration caching mechanism for myself (in one important project) which stores configuration artifacts in a cache directory, associated by the commit hash. It works as a git hook: $ git bisect good Bisecting: 7 revisions left to test after this (roughly 3 steps) restored cached configuration for 2f8679c346a88c07b81ea8e9854c71dae2ade167 [2f8679c346a88c07b81ea8e9854c71dae2ade167] expander: noexpand mechanism. The "restored cached configuration" message is from the git hook. What it's not saying is that it also saved the config for the commit it is navigating away from. I primed the cache by executing a "git checkout" for each of a range of commits. Going forward, it will populate itself. This is the only issue I would conceivably care about with regard to configure performance. When not navigating in git history, I do not often run configure. Downstream distros do not care; they keep their machines and cores busy by building multiple packages in parallel. It's not ideal because the cache from one host is not applicable to another; you can't port it. I could write an intelligent script to populate it, which basically identifies commits (within some specified range) that have touched the config system, and then assumes that for all in-between commits, it's the same. The hook could do this. When it notices that the current sha doesn't have a cached configuration, it could search backwards through history for the most recent commit which does have it. If the configure script (or something influencing it) has not been touched since that commit, then its cached material can be populated for all in-between commits right through the current one. That would take care of large swaths of commits in a typical bisect session.

kazinator 1y ago
The right way to do this is not to rely on the git hashes, but to hash the inputs into the configuration system (those that are in version control, not the implicit environmental inputs from the platform). For instance, if the only input to the configuration system is the body of the configure script, then we hash that. That is then our key to the generated materials.