14 ms·

Autoconf makes me think we stopped evolving too soon

213 points by Tomte 3y ago

jauntywundrkind 3y ago
This goes opposite of how I think of evolution. Admittedly, my feels are that we socially tend to overly emphasize the brutalist shades of Darwinianism, of natural selection as a competitive pressure. Evolution though is a much broader game of adaption, of finding niches. It's about capturing energy. And having diverse & creative means to adapt & succeed, less one finds oneself trapped only able to survive in a overly narrow environment. Evolutions greater story is more about over specialization being doom than it is about refining the exact perfect match; the perfect matches are amazing & elegant but rarely able to stick around. I don't really know my stance here. But it does feel like a call for authority. Why are we still diverse? Why still have complexity? Can't someone free us of the diversity of options and crown a champion, decree how it will be? That sort of resembles Darwinianism on the small, but in the big, we get change & evolution from having non-static environments, from having many ways to shape and be shaped, all about. That shift from dynamic to static systems, when evolution peaks, is usually not far from when that stasis turns to ossification & rot, decay & the fall. We probably should cleanup some & not support a million options, for our own sake, but this notion the past had of supporting a deeply rich & diverse technical ecosystem I find enormously beautiful. Anyhow, I forget how often autoconf needs to be run. I think often many projects would work fine for aomg time using the same generated conf across many many builds? There's focus here on the silliness of re-running config again and again and again, but I'm not sure when technically it's required; it seems more like habit & ritual than a need?
- smackeyacky 3y ago
  If you're building something really portable, you still need autoconf. Anything that has to run on a BSD or a Mac needs these tools. The article comes across as a bit of "blame the tools" where the tools aren't really at fault, it's the culture which has failed here. We seem to have never graduated from "wow it actually works" levels of Linux appreciation that isn't that different from the 1990s. Those distributions that most of us rely on (whether directly or indirectly) - Debian, RHEL - need to up their game somehow on vetting the packages they accept. We also need to stop just downloading .deb packages from the internet and installing them and preferring distros that look at security as a serious business including the packaging. Blaming autoconf is just silly. Sure it can be annoying, is mired in the past and perhaps seen as an anachronism now. But it's not to blame for supply chain attacks any more than a C compiler can be blamed for supply chain attacks.
  - chubot 3y ago
    Anything that has to run on a BSD or a Mac needs these tools Do QEMU, tcc, and OCaml run on BSD and Mac? Pretty sure they do https://news.ycombinator.com/item?id=39925033 https://news.ycombinator.com/item?id=39925033 https://github.com/lucasaiu/ocaml/blob/master/configure https://github.com/lucasaiu/ocaml/blob/master/configure They have hand-written configure scripts, and don't use autoconf. --- Also, https://www.oilshell.org/ https://www.oilshell.org/ runs on BSD and Mac, and it doesn't use autoconf. The configure script is a few hundred lines of "boring" shell.
    - oefrha 3y ago
      QEMU and tcc were created by Fabrice Bellard, they were written by a god! Us mortals use autoconf ;)
    - octachron 3y ago
      Note that OCaml has been using autoconf since OCaml 4.09.
- senectus1 3y ago
  also... what makes them think we stopped evolving? much like history, you don't actually notice it till after its done :-P
BoomerMoment 3y ago
Portability. A test to get the answer to "what the fuck am I running on and what does it support" is more portable and robust than thousands of "flavours" manually configured in /etc/whatamieven.conf The author misses that the buildtime magic for the xz exploit is not in the m4 file but in an obfuscated, compressed, encrypted, binary disguised as a test file that alters the build process at multiple stages (configure and build) A better argument can be made that the act of compiling a binary / obfuscating / minifying code instead of interpreting code directly is the fault.
- forrestthewoods 3y ago
  configure is a mistake. Building a project shouldn’t generate input files that are dependent on the system state. That’s what C projects from the 90s do and it’s genuinely awful.
  - matheusmoreira 3y ago
    What is the alternative? Even if you're very organized and abstract away all the stuff into neat platform-specific modules, you'll still have those system-dependent inputs in the build system. No matter what happens it will still have to pick which of those modules to include in the build and it will most likely do so via some target variable whose default value is whatever it autodetects the host system as.
- davisp 3y ago
  From the article: That means it's totally normal to ship all kinds of really crazy looking stuff, and so when someone noticed that and decided to use that as their mechanism for extracting some badness from a so-called "test file" that was actually laden with their binary code, is it so surprising that it happened? To me, it seems inevitable. Yeah, no. The author is well aware of how and why autotools are not awesome but also with the background of why they exist. A better argument can be made that the act of compiling a binary / obfuscating / minifying code instead of interpreting code directly is the fault. I can’t decide if you’ve never worked in systems software or trying to be hyperbolic. Given that it’s HN I’ll assume the best. But who do you think would do the interpreting? The priests at Delphi?
mid-kid 3y ago
You can pre-answer most autoconf sections using /etc/config.site, and a system distributor can create an /usr/share/config.site[1]. It's just really hard to get right, nobody uses it effectively. Autoconf also has a config.cache[2], which it uses to store answers to reuse across multiple of the same checks and reruns of the configure script. [1]: https://www.gnu.org/software/autoconf/manual/autoconf-2.63/html_node/Site-Defaults.html https://www.gnu.org/software/autoconf/manual/autoconf-2.63/h... [2]: https://www.gnu.org/software/autoconf/manual/autoconf-2.65/html_node/Cache-Files.html https://www.gnu.org/software/autoconf/manual/autoconf-2.65/h...
- mdaniel 3y ago
  and if I understand correctly, any such /etc/config.site falls prey to one of the more famous N problems in computer science, since the only truly stable things one could put in any such file would be the host triple and byte order. Maybe if there was a systemwide hook into deb or rpm to regenerate it after any package updates it would solve the cache invalidation part
  - bandrami 3y ago
    pkg-config was supposed to solve that, but, well, you know...
- fanf2 3y ago
  In my experience, config.cache fails to properly separate platform feature tests from build configuration options, so you need to delete config.cache if any build options change. It is extremely brittle and buggy. config.cache is definitely not suitable as a basis for for platforms to provide pre-cooked answers to platform feature tests.
  - professoretc 3y ago
    In other words, it's not just as simple as "figure out how to compile things on this system, once, and then reuse that information in the future". Different projects can be compiled with different options, cross-compilation is a thing, etc. As soon as you try to solve this with a simple global `site_config.h` you discover that that doesn't quite work.
chubot 3y ago
A better solution is just to write a plain ass shell script that tests if various C snippets compile. https://github.com/oilshell/oil/blob/master/configure https://github.com/oilshell/oil/blob/master/configure https://github.com/oilshell/oil/blob/master/build/detect-pwent.c https://github.com/oilshell/oil/blob/master/build/detect-pwe... Not an unholy mix of m4, shell, and C, all in the same file. --- These are the same style as a the configure scripts that Fabrice Bellard wrote for tcc and QEMU. They are plain ass shell scripts, because he actually understands the code he writes. https://github.com/qemu/qemu/blob/master/configure https://github.com/qemu/qemu/blob/master/configure https://github.com/TinyCC/tinycc/blob/mob/configure https://github.com/TinyCC/tinycc/blob/mob/configure OCaml’s configure script is also “normal”. You don’t have to copy and paste thousands of lines of GNU stuff that you don’t understand. (copy of lobste.rs comment)
- Gormo 3y ago
  > A better solution is just to write a plain ass shell script that tests if various C snippets compile. Then standardize those scripts and share them so others can benefit from code reuse. Then build a configuration tool to make it easy for everyone to define collections of tests applicable to their software. Then create a meta-level scripting system so all of the above can be integrated into the software build process. Then complain about how confusing and complicated the whole thing has become and start over from the beginning again.
  - chubot 3y ago
    If you redid all that work, and managed to avoid m4, it would still be a win! I believe m4 was used because some Unix shells at the time did not have functions. All shells have functions now -- it's in POSIX. If you have shell functions, then you can have modularity without concatenating blocks of code. You can make a library of functions, and then allow people to reuse them. It could just be source lib_configure.sh detect_foo detect_bar etc. POSIX shell can do everything that m4 can, and it's MORE deployed, not less. (Related: some people missed that m4 never runs on the end user machine which compiles the source code -- only on the machine that makes the tarball from the repo.)
evmar 3y ago
FWIW the sandbox-disabling bit in xz was in CMake logic, not autoconf. (Or at least the obvious one is in the CMake part; maybe the autoconf is separately backdoored.) https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2bbb81307644efdb58db2c422d9ba7 https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2...
- ajross 3y ago
  Yeah, this autotools thing is just a chance for people to whine about a favorite hated technology[1]. Clever attackers (and this was a very clever attacker) can hide stuff in anything. Anyone who thinks that CMake or Bazel or Python or whatever can't hide misbehavior is just fooling themselves. [1] And even then it's more hatred of M4 than it is anything in autoconf per se.
  - kergonath 3y ago
    > And even then it's more hatred of M4 than it is anything in autoconf per se. To be fair, M4 is rather…special. And when you add that in a Makefile template with shell thrown into the mix it gets _very_ hairy.
    - ajross 3y ago
      Oh sure, and that's true enough. Working with autoconf[1] is a giant pain for Kids Today weaned on entirely different idioms[2]. And it's the kind of thing you can't really avoid[3] if you're going to be doing work on these traditional Linux packages. And that sucks, and it makes it look like Linux is a weird kingdom filled with dinosaurs. And that sucks because you still have to work in Linux. So you hate the proximate cause. But the boring truth is that configuration management is just a hard problem and realistically you'd hate it anyway. [1] Not so much automake/libtool, which are configured by straightforward idioms and mostly harmless. [2] Though if you've got some C preprocessor fu, it doesn't look quite so alien. [3] Though it also needs to be pointed out that 90% of the problem autoconf was originally intended to solve (gratuitously incompatible Unix variants) is basically gone now. You either write to straight ISO-C/POSIX if you're doing simple stuff, work through well-established abstraction libraries with fixed APIs, or you include well-known platform headers (Linux or otherwise) for the complicated bits. No more nonsense testing like "Hm... what header do I need for strncat()?"
forrestthewoods 3y ago
A huge part of the problem is relying on the system. Project repos should include their dependencies. The Linux model of global shared libraries is, imho, bad and wrong. Building a project shouldn’t be complicated. There should be extremely minimal branching and if checks. My blog post where I argue this in more detail: https://www.forrestthewoods.com/blog/dependencies-belong-in-version-control/ https://www.forrestthewoods.com/blog/dependencies-belong-in-...
- nickelpro 3y ago
  vcpkg, conan, and Fetch_Content fix all the issues you discuss in that blog with the added benefit of not being insane. You dismiss package managers out of hand and you shouldn't.
  - forrestthewoods 3y ago
    > not being insane Insane? It’s what all of BigTech and every game dev does. It’s tried and true. > vcpkg, conan, and Fetch_Content fix all the issues Not really, no.
  - nalllar 3y ago
    conan makes it very hard to build a project offline and with a fixed set of inputs, it makes things worse in important ways if you care about reproducibility.
- jonstewart 3y ago
  Ah, the bad old days, when we checked DLLs into our cvs repos and every six months the greybeard on the team would bring a case of Mountain Dew into the office on a weekend and try upgrading the dependencies, sometimes successfully, before everyone else would arrive at work on Monday morning. Fun stuff.
- marcus_holmes 3y ago
  Ahh the eternal cycle of static vs dynamic linking swings around again. "I need common.lib, it gets linked into my application at compile-time" "why does every binary include this common.lib code? Let's pull that out to a shared module so we only have one version of it! See? that saved 35739475749Kb of storage! Everything is smaller and sleeker now!" "which version of common.lib does this application need? Hmm, but that other application needs an older version. I'll have to hack some way of having multiple versions of the common.lib available" "ahh, this application needs version 5.2.1 but that dependency needs version 4.11.6! Two incompatible versions in the same application! Who designed this thing?" "right, let's compile whatever versions of common.lib we need into our binary. It'll make life easier and to be honest there are so many versions of common.lib around that we might as well have a separate version for every application" "Ah, small problem, I have to compile a binary for my application for each variant of each platform because common.lib actually varies per platform... hmm, wonder if I can link to the system version of that lib at run-time..." <repeat until heat death of the universe>
andrewmcwatters 3y ago
There are super common scenarios with Autoconf, CMake, et al. which are just not provided as readily available examples or recipes and the whole time you work in these ecosystems you’re scratching your head asking why the hell no one has written these things in literal decades. No, I’m not switching build systems because everything people use as dependencies for Real Work™ relies on you using the incumbents. They don’t provide those templates either, anyway. Mind boggling.
elicksaur 3y ago
Is anyone doing and publicizing the git clone vs. tarball work in other projects? I suppose if something has been found, it might be too soon to disclose publicly.
- cperciva 3y ago
  Note that this sort of comparison isn't necessarily trivial. I have code where running `foo --version` out of the git repository will print "foo @@VERSION@@" because that gets substituted with the actual version number as part of the release process, for example.
  - elicksaur 3y ago
    A first step could be performance benchmarking a tarball release vs something built from a git clone, maybe? Just a guess since that would be similar to how this was found.
- opello 3y ago
  The major downside of using git clone build workflows from an autotools project is that your build environment and the developer's may have different enough versions of autotools that macro incompatibility introduces more problems for you to solve. I've noticed this less recently (say 5 years) but still has come up occasionally. While a distributed tarball will have the "developer blessed" autoreconf run and declared good by way of releasing the tarball. Without that process the options are committing generated files (boo) and burdening those that would build from a git clone instead of a distribution tarball with the demands of details from the development environment. That's intentionally a little vague because of the variety of things that can ultimately affect the autotools output. All of this is kind of the design of the autotools build process and could very well be a reason to take a harder look at it and the tradeoffs incumbent on its use.
  - TillE 3y ago
    Ultimately autotools isn't doing anything magical, so it's not unusual for cross-platform package managers like vcpkg to simply ignore all that and write their own clean CMakeLists.txt. In this particular case xz-utils actually did support CMake for building liblzma, so that's what you use if you don't care about the command line tools.
IAmNotACellist 3y ago
...But I like seeing hundreds of lines of "yes"es and success messages fly by at lightning speed before the inevitable compilation failure for the first line of actually incompatible (and non-trivial-to-fix) code. It makes me feel like I actually have a shot at compiling liboldafgnuglibxx3 from 2004 and that my efforts aren't futile.
titzer 3y ago
Autoconf is one of the absolutely hilarious things about UNIX. On the one hand, we've got people optimizing kernels down to the individual instructions (often doing very unsafe dirty C tricks underneath), sometimes with super-clunky and overly complex APIs as a result...and on the other hand you have all the shell script absolute nuttery like the behemoth heap of kludges that is autoconf. It's crazy to me the disconnect. And, oh by the way, underneath? Shells have some of the absolutely bonkersly dumb parsers and interpreters; absolutely embarrassingly dumb stuff like aliases that can override keywords and change the parsing of shell script. The fact that some (most) shells interpret a shell script one line at a time, and that "built-ins" like the syntax for conditions in ifs looks like syntax but might be little binaries underneath (look up how "[" and "]" is handled in some shells--it might not be!). What a wild irony that all the ickiest parts of UNIX--shells and scripts and autoconf and all that stringly-typed pipe stuff, ended up becoming (IMHO) the most reusable, pluggable programming environment yet devised...
- nabla9 3y ago
  Alan Perlis Epigram #9: "It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures." Unix settled for string as a data structure and it's little too low level. All Lisp programmers know that s-expressions would have been the right choice. JSON works also.
  - chubot 3y ago
    Yup, I call that the Perlis-Thompson Principle -- because Ken Thompson made a similar combinatorial argument about software composition: you should design it around "one thing". https://www.oilshell.org/blog/2021/07/blog-backlog-1.html#concepts https://www.oilshell.org/blog/2021/07/blog-backlog-1.html#co... Files had structure on pre-Unix OSes, but they don't on Unix, because it doesn't compose. The "Uniform Interface Constraint" of REST is the same thing, and resembles a file system -- everything is GET / POST on URLs. --- So the reason that shell/Unix and HTTP are so common is a MATHEMATICAL property of software growth. How do you make a Zig program talk to a Mojo program? Probably with a byte stream. What about a Clojure program and a Common Lisp program? Probably a byte stream. (Ironically, S-expressions have no commonly used "exterior" interchange format) Every time a new language is introduced, I think "well there's another reason you're going to need a shell script". --- The larger the system, the more heterogeneous it is. And software is larger now, which is why MORE GLUE is needed. This is why shell was the #6 fastest growing language on Github in 2022: https://octoverse.github.com/2022/top-programming-languages https://octoverse.github.com/2022/top-programming-languages And the #1 fastest growing language is HCL, which is a very closely related form of glue. It's basically "shell for distributed systems" --- https://www.oilshell.org/ https://www.oilshell.org/ has JSON as of a few months ago, and is now pure native code (no more Python) All the normal shell stuff works: osh$ ls */*.py | wc -l; whoami 327 andy But you can seamlessly create dicts and lists like JavaScript. They are fully GC'd data structures: osh$ var d = {name: 42} And then you can write them as JSON: osh$ json write (d) { "name": 42 } And you can also upgrade to ysh with one line: shopt --set ysh:upgrade (YSH isn't stable yet, but OSH is very stable) I think of the data languages as layered -- i.e. an entire json message literally is a string, and it can also contain strings: https://www.oilshell.org/blog/2023/06/ysh-design.html#exterior-designs-are-layered https://www.oilshell.org/blog/2023/06/ysh-design.html#exteri...
bradfitz 3y ago
Obligatory mention of this book title: Die GNU Autotools https://a.co/d/bQyvEYm https://a.co/d/bQyvEYm And classic tweet: https://x.com/timmartin2/status/23365017839599616?s=46 https://x.com/timmartin2/status/23365017839599616?s=46 ... "I saw a book entitled "Die GNU Autotools" and I thought "My feelings exactly". Turns out the book was in German."
wyldfire 3y ago
I'm finding myself in the unenviable position of defending autotools. > They were a reflection of the way the kernel, C libraries, APIs and userspace happened to work. Short of that changing, the results wouldn't change either. These things do change, especially as toolchain and OS vendors realize their quirks that cause portability friction.
- pas 3y ago
  there's absolutely nothing wrong with the functionality (it's essential in fact), it's that maybe, maybe, we should purge shell scripts from our hearts. (or at least reserve them for ad-hoc one-off in-situ time-conscious pragmatic programs, instead of letting a billion bash-lines beat its little shell-song behind every .deb,.rpm & etc.) // and of course 'should' is more like "it would be great if someone would rewrite it in Rust"
  - wewxjfq 3y ago
    Remember the serde_derive fiasco? https://news.ycombinator.com/item?id=37205647 https://news.ycombinator.com/item?id=37205647
wyldfire 3y ago
Does autotools really matter here? Did folks at the distros really audit the tainted package contents against the source repo and decide that the additional contents must be legitimate? In all likelihood there's no process to do such an audit? Let's assume such a process did exist, forcing attackers to take a different strategy. Could the rain have hidden in plain sight in the source repo anyways?
- pas 3y ago
  autotools matter just as much a the obfuscated bash, the tarball, email-only workflows, it's a symptom. we settle for bad tools out of convenience/necessity (status quo bias, time pressure, perverse incentives - ie. it's a hobby project, free and open source, yet half the world depends on it, so half the world prefers it to remain free even if it remains amateurish).
ahefner 3y ago
autoconf is a mess for developers, but when it's working it is great for users who want to build software from source. Fantastic feedback and help at the command line to enable/disable features and dependencies. Much friendlier than cmake, at least.
bandrami 3y ago
Younger people may not remember one of the early alternatives, libiberty (now at least somewhat improved as gnulib), but as an idea it was even worse: if your system doesn't provide a GNU compatible function call, it wrote one that hijacked whatever your system did have. It's still bumbling around there in the internals of gcc, silently patching every system it builds on.
- donatj 3y ago
  Maybe I'm crazy but that sounds like a huge improvement.
parasti 3y ago
Autoconf has never really made sense to me in the last twenty years or so. At what point does a Makefile-based project start thinking "hmm, we should upgrade to autoconf to solve this"? And what is "this" that they can't solve any other way? Seems like that point never occurs anymore. Either projects start with autoconf or they don't.
- kergonath 3y ago
  To me, the point is when something I write has tor each for BLAS or LAPACK and every single user seems to want it to work with a different library (ATLAS, OpenBLAS, MKL, Accelerate.framework, vanilla BLAS, cuBLAS)… Then, combine that with the various FFT libraries flavours, wherever the fuck hdf5 and NetCDF are, and which goddamned version of MPI they want to use. Not mentioning some computers where they insist on using Intel’s compilers. Or, dog saves us, Cray. Getting this to work is a major pain, we cannot expect the users to figure it out themselves, and we cannot list all the possible combinations in a Makefile. We tried several fancy build systems, and every single one breaks in more or less subtle ways.
  - parasti 3y ago
    That's a valid use case. Your users operate in their own diverse environments and you explicitly want to support them where they are.
deleted 3y ago
[deleted]
bitwize 3y ago
Autotools, for all its ugliness, solved a critical problem: getting GNU software up and going on a variety of Unix systems each of which supported a different mix of features. But it's 2024, people. If you port to Linux, macOS, and Windows, that's cross-platform enough to capture 99% of your userbase, and everything after that is diminishing returns. Still running DEC Ultrix on Alpha, a CPU architecture that's been out of manufacture for literal decades? Maintain the support yourself if it's so important. In conclusion, use cmake or Meson. Let autotools die. Kill them if you have to.
mediumsmart 3y ago
I am so grateful and happy I only have to write little programs for this one machine that only I run. Free cookies, fortune and fame for the real programmers who made this possible The sendmail configuration file is one of those files that looks like someone beat their head on the keyboard. After working with it... I can see why! -- Harry Skelton
kazinator 3y ago
One of the worst things about Autoconf is the way it is deployed in the GNU Project's own programs. Typically, GNU project programs assume that anyone working on the program (cloning the repo, looking for bugs, making changes, ...) has the Autotools installed. Not just any Autotools but the exact version the program wants. Only the "release tarball" if a GNU program contains the generated ./configure script that can run on a machine that doesn't have Autotools installed. You have to do some "make boostrap". So to make a source code release, they can't just tag a code baseline and serve that; they have to generate the release source! See the stupidity? The source code that is being released has to be compiled from the real source source. Now this makes life hell if you have to, say, git bisect to find a commit that introduces a problem. To be sure you're building each bisect step correctly, you have to disclean everything and re-run the autoconf bootstrapping, then ./configure and so on. The old code you're checking out may call for different Autotools versions from the trunk head. Working with some GNU programs is basically hell.
- NoGravitas 3y ago
  Everything you say is technically true, but I can't remember a time when just having the latest autotools from my distribution packages wasn't sufficient to build a GNU package.
  - kazinator 3y ago
    That's either a lucky coincidence, or use of a pre-generated ./configure script, not Autotools. If you're actually running the steps to regenerate the Makefile.in from Makefile.am, and configure from configure.ac, and all that, then you run into the version or version range checks. For instance, I have working repo of GNU Make. In the generate Makefile there are lines like: AUTOMAKE = ${SHELL} /home/kaz/make/build-aux/missing automake-1.16 The generated script missing is a wrapper for the program such as automate-1.16, which prints an error if it is missing. This version requirement comes from the file boostrap.conf: # Build prerequisites buildreq="\ autoconf 2.69 automake 1.16.1 " These prereuisite versions are also mentioned in configure.ac: AC_PREREQ([2.69]) # [ ... ] AM_INIT_AUTOMAKE([1.16.1 foreign -Werror -Wall]) Literally, the Autotools source is asserting it must be processed by a certain version of the tools. Needless to say, these lines change in the git history.
kazinator 3y ago
> So, okay, fine, at some point it made sense to run programs to empirically determine what was supported on a given system. That stopped making sense several decades ago, when FOSS began to be heavily used for embedded systems, where you are crosscompiling. Autoconf supports cross-compiling, but very badly. Programs not prepared with crosscompiling in mind will do stupid things at configure time that require a compiled program to be executed. For that reason, distros ended up using Qemu for cross builds. > What I don't understand is why we kept running those stupid little shell snippets and little bits of C code over and over. It's like, okay, we established that this particular system does <library function foobar> with two args, not three. So why the hell are we constantly testing for it over and over? Autoconf does have a cache system for this. I think it's only used within one package, not globally. There may be some knobs and levers to share autom4te.cache stuff between projects. Even when the configuration is not cached for the purposes of the test itself, obviously the information is cached in the generated config.h and config.make type files; once the program is configured, each time you recompile it, it's not testing for whether foobar has two args, not three; it's just referring to HAVE_TWO_ARG_FOO or whatever. There could be a global cache for autoconf tests, or at least the stanadard ones built into autoconf. Each test would have to have a unique ID, which changes whenever it changes between autoconf versions. The cache would have to distinguish targets. Packages building for the build machine itself must not share configuration results with packages built for the target. Even if it were done right in these regards, there is a risk of introducing bugs due to package build interaction through such a shared cache. A clean build of any package is not really clean if it refers to material from a previous build, possibly of another package. I can see distro builds opting out of something like this, and they would be the primary audience, since it is distro builds that are hurt by hundreds of packages executing essentially the same tests over again.
lamontcg 3y ago
The problem is that a configure script takes a look at the system, right now, as configured by the end user, even with their home directory and whatever they've shoved into env variables like LD_LIBRARY_PATH and then builds the software. At a minimum you need to have /etc, /usr/local/etc and ~/etc databases of "union mounted" layered configuration information, along with the ability to add and subtract arbitrary other locations where software might get installed. And it all needs to get updated with every new change to packages in those locations. As someone else pointed out in a comment, this is sort of the very unrealized promise of pkg-config. You can't do with just a static file/directory in /etc that defines the distro, that isn't even half the problem once you start installing stuff in /usr/local that you want to link against. This is a pretty good example of a Chesterton's Fence. You have to first fully understand why the fence is there before you knock it down.

213 points by Tomte 3y ago

This goes opposite of how I think of evolution. Admittedly, my feels are that we socially tend to overly emphasize the brutalist shades of Darwinianism, of natural selection as a competitive pressure. Evolution though is a much broader game of adaption, of finding niches. It's about capturing energy. And having diverse & creative means to adapt & succeed, less one finds oneself trapped only able to survive in a overly narrow environment. Evolutions greater story is more about over specialization being doom than it is about refining the exact perfect match; the perfect matches are amazing & elegant but rarely able to stick around. I don't really know my stance here. But it does feel like a call for authority. Why are we still diverse? Why still have complexity? Can't someone free us of the diversity of options and crown a champion, decree how it will be? That sort of resembles Darwinianism on the small, but in the big, we get change & evolution from having non-static environments, from having many ways to shape and be shaped, all about. That shift from dynamic to static systems, when evolution peaks, is usually not far from when that stasis turns to ossification & rot, decay & the fall. We probably should cleanup some & not support a million options, for our own sake, but this notion the past had of supporting a deeply rich & diverse technical ecosystem I find enormously beautiful. Anyhow, I forget how often autoconf needs to be run. I think often many projects would work fine for aomg time using the same generated conf across many many builds? There's focus here on the silliness of re-running config again and again and again, but I'm not sure when technically it's required; it seems more like habit & ritual than a need?

smackeyacky 3y ago
If you're building something really portable, you still need autoconf. Anything that has to run on a BSD or a Mac needs these tools. The article comes across as a bit of "blame the tools" where the tools aren't really at fault, it's the culture which has failed here. We seem to have never graduated from "wow it actually works" levels of Linux appreciation that isn't that different from the 1990s. Those distributions that most of us rely on (whether directly or indirectly) - Debian, RHEL - need to up their game somehow on vetting the packages they accept. We also need to stop just downloading .deb packages from the internet and installing them and preferring distros that look at security as a serious business including the packaging. Blaming autoconf is just silly. Sure it can be annoying, is mired in the past and perhaps seen as an anachronism now. But it's not to blame for supply chain attacks any more than a C compiler can be blamed for supply chain attacks.
- chubot 3y ago
  Anything that has to run on a BSD or a Mac needs these tools Do QEMU, tcc, and OCaml run on BSD and Mac? Pretty sure they do https://news.ycombinator.com/item?id=39925033 https://news.ycombinator.com/item?id=39925033 https://github.com/lucasaiu/ocaml/blob/master/configure https://github.com/lucasaiu/ocaml/blob/master/configure They have hand-written configure scripts, and don't use autoconf. --- Also, https://www.oilshell.org/ https://www.oilshell.org/ runs on BSD and Mac, and it doesn't use autoconf. The configure script is a few hundred lines of "boring" shell.
  - oefrha 3y ago
    QEMU and tcc were created by Fabrice Bellard, they were written by a god! Us mortals use autoconf ;)
  - octachron 3y ago
    Note that OCaml has been using autoconf since OCaml 4.09.
senectus1 3y ago
also... what makes them think we stopped evolving? much like history, you don't actually notice it till after its done :-P

BoomerMoment 3y ago

Portability. A test to get the answer to "what the fuck am I running on and what does it support" is more portable and robust than thousands of "flavours" manually configured in /etc/whatamieven.conf The author misses that the buildtime magic for the xz exploit is not in the m4 file but in an obfuscated, compressed, encrypted, binary disguised as a test file that alters the build process at multiple stages (configure and build) A better argument can be made that the act of compiling a binary / obfuscating / minifying code instead of interpreting code directly is the fault.

forrestthewoods 3y ago
configure is a mistake. Building a project shouldn’t generate input files that are dependent on the system state. That’s what C projects from the 90s do and it’s genuinely awful.
- matheusmoreira 3y ago
  What is the alternative? Even if you're very organized and abstract away all the stuff into neat platform-specific modules, you'll still have those system-dependent inputs in the build system. No matter what happens it will still have to pick which of those modules to include in the build and it will most likely do so via some target variable whose default value is whatever it autodetects the host system as.
davisp 3y ago
From the article: That means it's totally normal to ship all kinds of really crazy looking stuff, and so when someone noticed that and decided to use that as their mechanism for extracting some badness from a so-called "test file" that was actually laden with their binary code, is it so surprising that it happened? To me, it seems inevitable. Yeah, no. The author is well aware of how and why autotools are not awesome but also with the background of why they exist. A better argument can be made that the act of compiling a binary / obfuscating / minifying code instead of interpreting code directly is the fault. I can’t decide if you’ve never worked in systems software or trying to be hyperbolic. Given that it’s HN I’ll assume the best. But who do you think would do the interpreting? The priests at Delphi?

mid-kid 3y ago

You can pre-answer most autoconf sections using /etc/config.site, and a system distributor can create an /usr/share/config.site[1]. It's just really hard to get right, nobody uses it effectively. Autoconf also has a config.cache[2], which it uses to store answers to reuse across multiple of the same checks and reruns of the configure script. [1]: https://www.gnu.org/software/autoconf/manual/autoconf-2.63/html_node/Site-Defaults.html https://www.gnu.org/software/autoconf/manual/autoconf-2.63/h... [2]: https://www.gnu.org/software/autoconf/manual/autoconf-2.65/html_node/Cache-Files.html https://www.gnu.org/software/autoconf/manual/autoconf-2.65/h...

mdaniel 3y ago
and if I understand correctly, any such /etc/config.site falls prey to one of the more famous N problems in computer science, since the only truly stable things one could put in any such file would be the host triple and byte order. Maybe if there was a systemwide hook into deb or rpm to regenerate it after any package updates it would solve the cache invalidation part
- bandrami 3y ago
  pkg-config was supposed to solve that, but, well, you know...
fanf2 3y ago
In my experience, config.cache fails to properly separate platform feature tests from build configuration options, so you need to delete config.cache if any build options change. It is extremely brittle and buggy. config.cache is definitely not suitable as a basis for for platforms to provide pre-cooked answers to platform feature tests.
- professoretc 3y ago
  In other words, it's not just as simple as "figure out how to compile things on this system, once, and then reuse that information in the future". Different projects can be compiled with different options, cross-compilation is a thing, etc. As soon as you try to solve this with a simple global `site_config.h` you discover that that doesn't quite work.

chubot 3y ago

A better solution is just to write a plain ass shell script that tests if various C snippets compile. https://github.com/oilshell/oil/blob/master/configure https://github.com/oilshell/oil/blob/master/configure https://github.com/oilshell/oil/blob/master/build/detect-pwent.c https://github.com/oilshell/oil/blob/master/build/detect-pwe... Not an unholy mix of m4, shell, and C, all in the same file. --- These are the same style as a the configure scripts that Fabrice Bellard wrote for tcc and QEMU. They are plain ass shell scripts, because he actually understands the code he writes. https://github.com/qemu/qemu/blob/master/configure https://github.com/qemu/qemu/blob/master/configure https://github.com/TinyCC/tinycc/blob/mob/configure https://github.com/TinyCC/tinycc/blob/mob/configure OCaml’s configure script is also “normal”. You don’t have to copy and paste thousands of lines of GNU stuff that you don’t understand. (copy of lobste.rs comment)

Gormo 3y ago
> A better solution is just to write a plain ass shell script that tests if various C snippets compile. Then standardize those scripts and share them so others can benefit from code reuse. Then build a configuration tool to make it easy for everyone to define collections of tests applicable to their software. Then create a meta-level scripting system so all of the above can be integrated into the software build process. Then complain about how confusing and complicated the whole thing has become and start over from the beginning again.
- chubot 3y ago
  If you redid all that work, and managed to avoid m4, it would still be a win! I believe m4 was used because some Unix shells at the time did not have functions. All shells have functions now -- it's in POSIX. If you have shell functions, then you can have modularity without concatenating blocks of code. You can make a library of functions, and then allow people to reuse them. It could just be source lib_configure.sh detect_foo detect_bar etc. POSIX shell can do everything that m4 can, and it's MORE deployed, not less. (Related: some people missed that m4 never runs on the end user machine which compiles the source code -- only on the machine that makes the tarball from the repo.)

evmar 3y ago

FWIW the sandbox-disabling bit in xz was in CMake logic, not autoconf. (Or at least the obvious one is in the CMake part; maybe the autoconf is separately backdoored.) https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2bbb81307644efdb58db2c422d9ba7 https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2...

ajross 3y ago
Yeah, this autotools thing is just a chance for people to whine about a favorite hated technology[1]. Clever attackers (and this was a very clever attacker) can hide stuff in anything. Anyone who thinks that CMake or Bazel or Python or whatever can't hide misbehavior is just fooling themselves. [1] And even then it's more hatred of M4 than it is anything in autoconf per se.
- kergonath 3y ago
  > And even then it's more hatred of M4 than it is anything in autoconf per se. To be fair, M4 is rather…special. And when you add that in a Makefile template with shell thrown into the mix it gets _very_ hairy.
  - ajross 3y ago
    Oh sure, and that's true enough. Working with autoconf[1] is a giant pain for Kids Today weaned on entirely different idioms[2]. And it's the kind of thing you can't really avoid[3] if you're going to be doing work on these traditional Linux packages. And that sucks, and it makes it look like Linux is a weird kingdom filled with dinosaurs. And that sucks because you still have to work in Linux. So you hate the proximate cause. But the boring truth is that configuration management is just a hard problem and realistically you'd hate it anyway. [1] Not so much automake/libtool, which are configured by straightforward idioms and mostly harmless. [2] Though if you've got some C preprocessor fu, it doesn't look quite so alien. [3] Though it also needs to be pointed out that 90% of the problem autoconf was originally intended to solve (gratuitously incompatible Unix variants) is basically gone now. You either write to straight ISO-C/POSIX if you're doing simple stuff, work through well-established abstraction libraries with fixed APIs, or you include well-known platform headers (Linux or otherwise) for the complicated bits. No more nonsense testing like "Hm... what header do I need for strncat()?"

forrestthewoods 3y ago

A huge part of the problem is relying on the system. Project repos should include their dependencies. The Linux model of global shared libraries is, imho, bad and wrong. Building a project shouldn’t be complicated. There should be extremely minimal branching and if checks. My blog post where I argue this in more detail: https://www.forrestthewoods.com/blog/dependencies-belong-in-version-control/ https://www.forrestthewoods.com/blog/dependencies-belong-in-...

nickelpro 3y ago
vcpkg, conan, and Fetch_Content fix all the issues you discuss in that blog with the added benefit of not being insane. You dismiss package managers out of hand and you shouldn't.
- forrestthewoods 3y ago
  > not being insane Insane? It’s what all of BigTech and every game dev does. It’s tried and true. > vcpkg, conan, and Fetch_Content fix all the issues Not really, no.
- nalllar 3y ago
  conan makes it very hard to build a project offline and with a fixed set of inputs, it makes things worse in important ways if you care about reproducibility.
jonstewart 3y ago
Ah, the bad old days, when we checked DLLs into our cvs repos and every six months the greybeard on the team would bring a case of Mountain Dew into the office on a weekend and try upgrading the dependencies, sometimes successfully, before everyone else would arrive at work on Monday morning. Fun stuff.
marcus_holmes 3y ago
Ahh the eternal cycle of static vs dynamic linking swings around again. "I need common.lib, it gets linked into my application at compile-time" "why does every binary include this common.lib code? Let's pull that out to a shared module so we only have one version of it! See? that saved 35739475749Kb of storage! Everything is smaller and sleeker now!" "which version of common.lib does this application need? Hmm, but that other application needs an older version. I'll have to hack some way of having multiple versions of the common.lib available" "ahh, this application needs version 5.2.1 but that dependency needs version 4.11.6! Two incompatible versions in the same application! Who designed this thing?" "right, let's compile whatever versions of common.lib we need into our binary. It'll make life easier and to be honest there are so many versions of common.lib around that we might as well have a separate version for every application" "Ah, small problem, I have to compile a binary for my application for each variant of each platform because common.lib actually varies per platform... hmm, wonder if I can link to the system version of that lib at run-time..." <repeat until heat death of the universe>

andrewmcwatters 3y ago

There are super common scenarios with Autoconf, CMake, et al. which are just not provided as readily available examples or recipes and the whole time you work in these ecosystems you’re scratching your head asking why the hell no one has written these things in literal decades. No, I’m not switching build systems because everything people use as dependencies for Real Work™ relies on you using the incumbents. They don’t provide those templates either, anyway. Mind boggling.

elicksaur 3y ago

Is anyone doing and publicizing the git clone vs. tarball work in other projects? I suppose if something has been found, it might be too soon to disclose publicly.

cperciva 3y ago
Note that this sort of comparison isn't necessarily trivial. I have code where running `foo --version` out of the git repository will print "foo @@VERSION@@" because that gets substituted with the actual version number as part of the release process, for example.
- elicksaur 3y ago
  A first step could be performance benchmarking a tarball release vs something built from a git clone, maybe? Just a guess since that would be similar to how this was found.
opello 3y ago
The major downside of using git clone build workflows from an autotools project is that your build environment and the developer's may have different enough versions of autotools that macro incompatibility introduces more problems for you to solve. I've noticed this less recently (say 5 years) but still has come up occasionally. While a distributed tarball will have the "developer blessed" autoreconf run and declared good by way of releasing the tarball. Without that process the options are committing generated files (boo) and burdening those that would build from a git clone instead of a distribution tarball with the demands of details from the development environment. That's intentionally a little vague because of the variety of things that can ultimately affect the autotools output. All of this is kind of the design of the autotools build process and could very well be a reason to take a harder look at it and the tradeoffs incumbent on its use.
- TillE 3y ago
  Ultimately autotools isn't doing anything magical, so it's not unusual for cross-platform package managers like vcpkg to simply ignore all that and write their own clean CMakeLists.txt. In this particular case xz-utils actually did support CMake for building liblzma, so that's what you use if you don't care about the command line tools.

IAmNotACellist 3y ago

...But I like seeing hundreds of lines of "yes"es and success messages fly by at lightning speed before the inevitable compilation failure for the first line of actually incompatible (and non-trivial-to-fix) code. It makes me feel like I actually have a shot at compiling liboldafgnuglibxx3 from 2004 and that my efforts aren't futile.

titzer 3y ago

Autoconf is one of the absolutely hilarious things about UNIX. On the one hand, we've got people optimizing kernels down to the individual instructions (often doing very unsafe dirty C tricks underneath), sometimes with super-clunky and overly complex APIs as a result...and on the other hand you have all the shell script absolute nuttery like the behemoth heap of kludges that is autoconf. It's crazy to me the disconnect. And, oh by the way, underneath? Shells have some of the absolutely bonkersly dumb parsers and interpreters; absolutely embarrassingly dumb stuff like aliases that can override keywords and change the parsing of shell script. The fact that some (most) shells interpret a shell script one line at a time, and that "built-ins" like the syntax for conditions in ifs looks like syntax but might be little binaries underneath (look up how "[" and "]" is handled in some shells--it might not be!). What a wild irony that all the ickiest parts of UNIX--shells and scripts and autoconf and all that stringly-typed pipe stuff, ended up becoming (IMHO) the most reusable, pluggable programming environment yet devised...

nabla9 3y ago
Alan Perlis Epigram #9: "It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures." Unix settled for string as a data structure and it's little too low level. All Lisp programmers know that s-expressions would have been the right choice. JSON works also.
- chubot 3y ago
  Yup, I call that the Perlis-Thompson Principle -- because Ken Thompson made a similar combinatorial argument about software composition: you should design it around "one thing". https://www.oilshell.org/blog/2021/07/blog-backlog-1.html#concepts https://www.oilshell.org/blog/2021/07/blog-backlog-1.html#co... Files had structure on pre-Unix OSes, but they don't on Unix, because it doesn't compose. The "Uniform Interface Constraint" of REST is the same thing, and resembles a file system -- everything is GET / POST on URLs. --- So the reason that shell/Unix and HTTP are so common is a MATHEMATICAL property of software growth. How do you make a Zig program talk to a Mojo program? Probably with a byte stream. What about a Clojure program and a Common Lisp program? Probably a byte stream. (Ironically, S-expressions have no commonly used "exterior" interchange format) Every time a new language is introduced, I think "well there's another reason you're going to need a shell script". --- The larger the system, the more heterogeneous it is. And software is larger now, which is why MORE GLUE is needed. This is why shell was the #6 fastest growing language on Github in 2022: https://octoverse.github.com/2022/top-programming-languages https://octoverse.github.com/2022/top-programming-languages And the #1 fastest growing language is HCL, which is a very closely related form of glue. It's basically "shell for distributed systems" --- https://www.oilshell.org/ https://www.oilshell.org/ has JSON as of a few months ago, and is now pure native code (no more Python) All the normal shell stuff works: osh$ ls */*.py | wc -l; whoami 327 andy But you can seamlessly create dicts and lists like JavaScript. They are fully GC'd data structures: osh$ var d = {name: 42} And then you can write them as JSON: osh$ json write (d) { "name": 42 } And you can also upgrade to ysh with one line: shopt --set ysh:upgrade (YSH isn't stable yet, but OSH is very stable) I think of the data languages as layered -- i.e. an entire json message literally is a string, and it can also contain strings: https://www.oilshell.org/blog/2023/06/ysh-design.html#exterior-designs-are-layered https://www.oilshell.org/blog/2023/06/ysh-design.html#exteri...

bradfitz 3y ago

Obligatory mention of this book title: Die GNU Autotools https://a.co/d/bQyvEYm https://a.co/d/bQyvEYm And classic tweet: https://x.com/timmartin2/status/23365017839599616?s=46 https://x.com/timmartin2/status/23365017839599616?s=46 ... "I saw a book entitled "Die GNU Autotools" and I thought "My feelings exactly". Turns out the book was in German."

wyldfire 3y ago

I'm finding myself in the unenviable position of defending autotools. > They were a reflection of the way the kernel, C libraries, APIs and userspace happened to work. Short of that changing, the results wouldn't change either. These things do change, especially as toolchain and OS vendors realize their quirks that cause portability friction.

pas 3y ago
there's absolutely nothing wrong with the functionality (it's essential in fact), it's that maybe, maybe, we should purge shell scripts from our hearts. (or at least reserve them for ad-hoc one-off in-situ time-conscious pragmatic programs, instead of letting a billion bash-lines beat its little shell-song behind every .deb,.rpm & etc.) // and of course 'should' is more like "it would be great if someone would rewrite it in Rust"
- wewxjfq 3y ago
  Remember the serde_derive fiasco? https://news.ycombinator.com/item?id=37205647 https://news.ycombinator.com/item?id=37205647

wyldfire 3y ago

Does autotools really matter here? Did folks at the distros really audit the tainted package contents against the source repo and decide that the additional contents must be legitimate? In all likelihood there's no process to do such an audit? Let's assume such a process did exist, forcing attackers to take a different strategy. Could the rain have hidden in plain sight in the source repo anyways?

pas 3y ago
autotools matter just as much a the obfuscated bash, the tarball, email-only workflows, it's a symptom. we settle for bad tools out of convenience/necessity (status quo bias, time pressure, perverse incentives - ie. it's a hobby project, free and open source, yet half the world depends on it, so half the world prefers it to remain free even if it remains amateurish).

ahefner 3y ago

autoconf is a mess for developers, but when it's working it is great for users who want to build software from source. Fantastic feedback and help at the command line to enable/disable features and dependencies. Much friendlier than cmake, at least.

bandrami 3y ago

Younger people may not remember one of the early alternatives, libiberty (now at least somewhat improved as gnulib), but as an idea it was even worse: if your system doesn't provide a GNU compatible function call, it wrote one that hijacked whatever your system did have. It's still bumbling around there in the internals of gcc, silently patching every system it builds on.

donatj 3y ago
Maybe I'm crazy but that sounds like a huge improvement.

parasti 3y ago

Autoconf has never really made sense to me in the last twenty years or so. At what point does a Makefile-based project start thinking "hmm, we should upgrade to autoconf to solve this"? And what is "this" that they can't solve any other way? Seems like that point never occurs anymore. Either projects start with autoconf or they don't.

kergonath 3y ago
To me, the point is when something I write has tor each for BLAS or LAPACK and every single user seems to want it to work with a different library (ATLAS, OpenBLAS, MKL, Accelerate.framework, vanilla BLAS, cuBLAS)… Then, combine that with the various FFT libraries flavours, wherever the fuck hdf5 and NetCDF are, and which goddamned version of MPI they want to use. Not mentioning some computers where they insist on using Intel’s compilers. Or, dog saves us, Cray. Getting this to work is a major pain, we cannot expect the users to figure it out themselves, and we cannot list all the possible combinations in a Makefile. We tried several fancy build systems, and every single one breaks in more or less subtle ways.
- parasti 3y ago
  That's a valid use case. Your users operate in their own diverse environments and you explicitly want to support them where they are.

deleted 3y ago

[deleted]

bitwize 3y ago

Autotools, for all its ugliness, solved a critical problem: getting GNU software up and going on a variety of Unix systems each of which supported a different mix of features. But it's 2024, people. If you port to Linux, macOS, and Windows, that's cross-platform enough to capture 99% of your userbase, and everything after that is diminishing returns. Still running DEC Ultrix on Alpha, a CPU architecture that's been out of manufacture for literal decades? Maintain the support yourself if it's so important. In conclusion, use cmake or Meson. Let autotools die. Kill them if you have to.

mediumsmart 3y ago

I am so grateful and happy I only have to write little programs for this one machine that only I run. Free cookies, fortune and fame for the real programmers who made this possible The sendmail configuration file is one of those files that looks like someone beat their head on the keyboard. After working with it... I can see why! -- Harry Skelton

kazinator 3y ago

One of the worst things about Autoconf is the way it is deployed in the GNU Project's own programs. Typically, GNU project programs assume that anyone working on the program (cloning the repo, looking for bugs, making changes, ...) has the Autotools installed. Not just any Autotools but the exact version the program wants. Only the "release tarball" if a GNU program contains the generated ./configure script that can run on a machine that doesn't have Autotools installed. You have to do some "make boostrap". So to make a source code release, they can't just tag a code baseline and serve that; they have to generate the release source! See the stupidity? The source code that is being released has to be compiled from the real source source. Now this makes life hell if you have to, say, git bisect to find a commit that introduces a problem. To be sure you're building each bisect step correctly, you have to disclean everything and re-run the autoconf bootstrapping, then ./configure and so on. The old code you're checking out may call for different Autotools versions from the trunk head. Working with some GNU programs is basically hell.

NoGravitas 3y ago
Everything you say is technically true, but I can't remember a time when just having the latest autotools from my distribution packages wasn't sufficient to build a GNU package.
- kazinator 3y ago
  That's either a lucky coincidence, or use of a pre-generated ./configure script, not Autotools. If you're actually running the steps to regenerate the Makefile.in from Makefile.am, and configure from configure.ac, and all that, then you run into the version or version range checks. For instance, I have working repo of GNU Make. In the generate Makefile there are lines like: AUTOMAKE = ${SHELL} /home/kaz/make/build-aux/missing automake-1.16 The generated script missing is a wrapper for the program such as automate-1.16, which prints an error if it is missing. This version requirement comes from the file boostrap.conf: # Build prerequisites buildreq="\ autoconf 2.69 automake 1.16.1 " These prereuisite versions are also mentioned in configure.ac: AC_PREREQ([2.69]) # [ ... ] AM_INIT_AUTOMAKE([1.16.1 foreign -Werror -Wall]) Literally, the Autotools source is asserting it must be processed by a certain version of the tools. Needless to say, these lines change in the git history.

kazinator 3y ago

> So, okay, fine, at some point it made sense to run programs to empirically determine what was supported on a given system. That stopped making sense several decades ago, when FOSS began to be heavily used for embedded systems, where you are crosscompiling. Autoconf supports cross-compiling, but very badly. Programs not prepared with crosscompiling in mind will do stupid things at configure time that require a compiled program to be executed. For that reason, distros ended up using Qemu for cross builds. > What I don't understand is why we kept running those stupid little shell snippets and little bits of C code over and over. It's like, okay, we established that this particular system does <library function foobar> with two args, not three. So why the hell are we constantly testing for it over and over? Autoconf does have a cache system for this. I think it's only used within one package, not globally. There may be some knobs and levers to share autom4te.cache stuff between projects. Even when the configuration is not cached for the purposes of the test itself, obviously the information is cached in the generated config.h and config.make type files; once the program is configured, each time you recompile it, it's not testing for whether foobar has two args, not three; it's just referring to HAVE_TWO_ARG_FOO or whatever. There could be a global cache for autoconf tests, or at least the stanadard ones built into autoconf. Each test would have to have a unique ID, which changes whenever it changes between autoconf versions. The cache would have to distinguish targets. Packages building for the build machine itself must not share configuration results with packages built for the target. Even if it were done right in these regards, there is a risk of introducing bugs due to package build interaction through such a shared cache. A clean build of any package is not really clean if it refers to material from a previous build, possibly of another package. I can see distro builds opting out of something like this, and they would be the primary audience, since it is distro builds that are hurt by hundreds of packages executing essentially the same tests over again.

lamontcg 3y ago

The problem is that a configure script takes a look at the system, right now, as configured by the end user, even with their home directory and whatever they've shoved into env variables like LD_LIBRARY_PATH and then builds the software. At a minimum you need to have /etc, /usr/local/etc and ~/etc databases of "union mounted" layered configuration information, along with the ability to add and subtract arbitrary other locations where software might get installed. And it all needs to get updated with every new change to packages in those locations. As someone else pointed out in a comment, this is sort of the very unrealized promise of pkg-config. You can't do with just a static file/directory in /etc that defines the distro, that isn't even half the problem once you start installing stuff in /usr/local that you want to link against. This is a pretty good example of a Chesterton's Fence. You have to first fully understand why the fence is there before you knock it down.