4 ms·

is this really a big deal given you run ./configure once? it's like systemd trading off non-determinism for boot speed, when it takes 5 minutes to get through

is this really a big deal given you run ./configure once? it's like systemd trading off non-determinism for boot speed, when it takes 5 minutes to get through the POST

LegionMammal978 2y ago
If you do a lot of bisecting, or bootstrapping, or building compatibility matrices, or really anything that needs you to compile lots of old versions, the repeated ./configure steps really start feeling like a drag.
- kazinator 2y ago
  In a "reasonably well-behaved program", if you have the artifacts from a current configure, like a "config.h" header, they are compatible with older commits, even if configurations changed, as long as the configuration changes were additive: introducing some new test, along with a new symbol in "config.h". It's possible to skip some of the ./configure steps. Especially for someone who knows the program very well.
  - LegionMammal978 2y ago
    Perhaps you can get away with that for small, young, or self-contained projects. But for medium-to-large projects running more than a few years, the (different versions of) external or vendored dependencies tend to come and go, and they all have their own configurations. Long-running projects are also prone to internal reorganizations and overhauls to the build system. (Go back far enough, and you're having to wrangle patchsets for every few months' worth of versions since -fpermissive is no longer permissive enough to get it to build.)
    - deleted 1y ago
      [deleted]
    - kazinator 1y ago
      > Perhaps you can get away with that for small, young, or self-contained projects Only a small subset of which could just switch to the parallel configure being proposed.
      - LegionMammal978 1y ago
        Well, the hope would be that such a thing had already existed years ago. But failing that, it would be nice if a parallel configure could become solid enough for new C/C++ projects to use today.
        kazinator 1y ago
        I've rolled a successful caching mechanism for configure artifacts, in my biggest project. You install it as a git hook: post-checkout. The cache is keyed on a SHA256 sum taken over the configuration inputs. (The ones in the repo, not the environment. The host environment is assumed to be static.) These inputs do not change for many commits at a time when the configure system isn't touched. During a bisect operation, the cache can be expected to quickly warm up, reducing the number of configuration operations needed, even on the host where repository was just cleanly cloned and the cache is empty. It's not parallel config, but it's pretty cool. In some ways it's better because when there's a hit it's instant. It's quite synergistic with ccache as well. You navigate to a commit. You are informed that cash config info was retrieved successfully, so you skip configure and run make. All the objects are retrieved from ccache.
        LegionMammal978 1y ago
        Yes, I get it, your own projects are impeccably well-organized. But it's hardly helpful for the rest of us, working with projects that organically grow and mutate over time.
csdvrx 2y ago
> it's like systemd trading off non-determinism for boot speed, when it takes 5 minutes to get through the POST That's a bad analogy: if a given deterministic service ordering is needed for a service to correctly start (say because it doesn't start with the systemd unit), it means the non-deterministic systemd service units are not properly encoding the dependencies tree in the Before= and After= When done properly, both solutions should work the same. However, the solution properly encoding the dependency graph (instead of just projecting it on a 1-dimensional sequence of numbers) will be more flexible: it's the better solution, because it will give you more speed but also more flexibility: you can see the branches any leaf depends on, remove leaves as needed, then cull the useless branches. You could add determinism if you want, but why bother? It's like using the dependencies of linux packages, and leaving the job of resolving them to package managers (apt, pacman...): you can then remove the useless packages which are no longer required. Compare that to doing a `make install` of everything to /usr/local in a specific order, as specified by a script: when done properly, both solutions will work, but one solution is clearly better than the other as it encodes more finely the existing dependencies instead of projecting them to a sequence. You can add determinism if you want to follow a sequence (ex: `apt-get install make` before adding gcc, then add cuda...), or you can use meta package like build-essentials, but being restricted to a sequence gains you nothing.
- blibble 2y ago
  I don't think it is a bad analogy given how complicated the boot process is ([1]), and it occurs once a month, I'd rather it was as deterministic as possible vs. shaving 1% off the boot time [1]: distros continue to ship subtlety broken unit files, because the model is too complicated
  - Aurornis 2y ago
    Most systems do not have 5 minute POST times. That’s an extreme outlier. Linux runs all over, including embedded systems where boot time is important. Optimizing for edge cases on outliers isn’t a priority. If you need specific boot ordering, configure it that way. It doesn’t make sense for the entire Linux world to sacrifice boot speed.
Aurornis 2y ago
> is this really a big deal given you run ./configure once I end up running it dozens of times when changing versions, checking out different branches, chasing dependencies. It’s a big deal. > it's like systemd trading off non-determinism for boot speed, when it takes 5 minutes to get through the POST 5 minute POST time is a bad analogy. systemd is used in many places, from desktops (that POST quickly) to embedded systems where boot time is critical. If deterministic boot is important then you would specify it explicitly. Relying on emergent behavior for consistent boot order is bad design. The number of systems that have 5 minute POST times and need deterministic boot is an edge case of an edge case.
- mschuster91 2y ago
  > I end up running it dozens of times when changing versions, checking out different branches, chasing dependencies. Yeah... but neither of that is going to change stuff like the size of a data type, the endianness of the architecture you're running on, or the features / build configuration of some library the project depends on. Parallelization is a bandaid (although a sorely needed!) IMHO, C/C++ libraries desperately need to develop some sort of standard that doesn't require a full gcc build for each tiny test. I'd envision something like nodejs's package.json, just with more specific information about the build details themselves. And for the stuff like datatype sizes, that should be provided by gcc/llvm in a fast-parseable way so that autotools can pick it up.
  - o11c 2y ago
    There is the `-C` option of course. It's supposedly good for the standard tests that waste all the time, but not so much for the ad-hoc tests various projects use, which have an unfortunate chance of being buggy or varying across time. ... I wonder if it's possible to manually seed a cache file with only known-safe test results and let it still perform the unsafe tests? Be sure to copy the cache file to a temporary name ... --- I've thought about rewriting `./configure` in C (I did it in Python once but Python's portability turned out to be poor - Python2 was bug-free but killed; Python3 was unfixably buggy for a decade or so). Still have a stub shell script that reads HOSTCC etc. then quickly builds and executes `./configure.bin`.
TacticalCoder 2y ago
[dead]
asah 2y ago
For postgresql development, you run configure over and over...

is this really a big deal given you run ./configure once? it's like systemd trading off non-determinism for boot speed, when it takes 5 minutes to get through the POST

LegionMammal978 2y ago

If you do a lot of bisecting, or bootstrapping, or building compatibility matrices, or really anything that needs you to compile lots of old versions, the repeated ./configure steps really start feeling like a drag.

kazinator 2y ago
In a "reasonably well-behaved program", if you have the artifacts from a current configure, like a "config.h" header, they are compatible with older commits, even if configurations changed, as long as the configuration changes were additive: introducing some new test, along with a new symbol in "config.h". It's possible to skip some of the ./configure steps. Especially for someone who knows the program very well.
- LegionMammal978 2y ago
  Perhaps you can get away with that for small, young, or self-contained projects. But for medium-to-large projects running more than a few years, the (different versions of) external or vendored dependencies tend to come and go, and they all have their own configurations. Long-running projects are also prone to internal reorganizations and overhauls to the build system. (Go back far enough, and you're having to wrangle patchsets for every few months' worth of versions since -fpermissive is no longer permissive enough to get it to build.)
  - deleted 1y ago
    [deleted]
  - kazinator 1y ago
    > Perhaps you can get away with that for small, young, or self-contained projects Only a small subset of which could just switch to the parallel configure being proposed.
    - LegionMammal978 1y ago
      Well, the hope would be that such a thing had already existed years ago. But failing that, it would be nice if a parallel configure could become solid enough for new C/C++ projects to use today.
      - kazinator 1y ago
        I've rolled a successful caching mechanism for configure artifacts, in my biggest project. You install it as a git hook: post-checkout. The cache is keyed on a SHA256 sum taken over the configuration inputs. (The ones in the repo, not the environment. The host environment is assumed to be static.) These inputs do not change for many commits at a time when the configure system isn't touched. During a bisect operation, the cache can be expected to quickly warm up, reducing the number of configuration operations needed, even on the host where repository was just cleanly cloned and the cache is empty. It's not parallel config, but it's pretty cool. In some ways it's better because when there's a hit it's instant. It's quite synergistic with ccache as well. You navigate to a commit. You are informed that cash config info was retrieved successfully, so you skip configure and run make. All the objects are retrieved from ccache.
        LegionMammal978 1y ago
        Yes, I get it, your own projects are impeccably well-organized. But it's hardly helpful for the rest of us, working with projects that organically grow and mutate over time.

csdvrx 2y ago

> it's like systemd trading off non-determinism for boot speed, when it takes 5 minutes to get through the POST That's a bad analogy: if a given deterministic service ordering is needed for a service to correctly start (say because it doesn't start with the systemd unit), it means the non-deterministic systemd service units are not properly encoding the dependencies tree in the Before= and After= When done properly, both solutions should work the same. However, the solution properly encoding the dependency graph (instead of just projecting it on a 1-dimensional sequence of numbers) will be more flexible: it's the better solution, because it will give you more speed but also more flexibility: you can see the branches any leaf depends on, remove leaves as needed, then cull the useless branches. You could add determinism if you want, but why bother? It's like using the dependencies of linux packages, and leaving the job of resolving them to package managers (apt, pacman...): you can then remove the useless packages which are no longer required. Compare that to doing a `make install` of everything to /usr/local in a specific order, as specified by a script: when done properly, both solutions will work, but one solution is clearly better than the other as it encodes more finely the existing dependencies instead of projecting them to a sequence. You can add determinism if you want to follow a sequence (ex: `apt-get install make` before adding gcc, then add cuda...), or you can use meta package like build-essentials, but being restricted to a sequence gains you nothing.

blibble 2y ago
I don't think it is a bad analogy given how complicated the boot process is ([1]), and it occurs once a month, I'd rather it was as deterministic as possible vs. shaving 1% off the boot time [1]: distros continue to ship subtlety broken unit files, because the model is too complicated
- Aurornis 2y ago
  Most systems do not have 5 minute POST times. That’s an extreme outlier. Linux runs all over, including embedded systems where boot time is important. Optimizing for edge cases on outliers isn’t a priority. If you need specific boot ordering, configure it that way. It doesn’t make sense for the entire Linux world to sacrifice boot speed.

Aurornis 2y ago

> is this really a big deal given you run ./configure once I end up running it dozens of times when changing versions, checking out different branches, chasing dependencies. It’s a big deal. > it's like systemd trading off non-determinism for boot speed, when it takes 5 minutes to get through the POST 5 minute POST time is a bad analogy. systemd is used in many places, from desktops (that POST quickly) to embedded systems where boot time is critical. If deterministic boot is important then you would specify it explicitly. Relying on emergent behavior for consistent boot order is bad design. The number of systems that have 5 minute POST times and need deterministic boot is an edge case of an edge case.

mschuster91 2y ago
> I end up running it dozens of times when changing versions, checking out different branches, chasing dependencies. Yeah... but neither of that is going to change stuff like the size of a data type, the endianness of the architecture you're running on, or the features / build configuration of some library the project depends on. Parallelization is a bandaid (although a sorely needed!) IMHO, C/C++ libraries desperately need to develop some sort of standard that doesn't require a full gcc build for each tiny test. I'd envision something like nodejs's package.json, just with more specific information about the build details themselves. And for the stuff like datatype sizes, that should be provided by gcc/llvm in a fast-parseable way so that autotools can pick it up.
- o11c 2y ago
  There is the `-C` option of course. It's supposedly good for the standard tests that waste all the time, but not so much for the ad-hoc tests various projects use, which have an unfortunate chance of being buggy or varying across time. ... I wonder if it's possible to manually seed a cache file with only known-safe test results and let it still perform the unsafe tests? Be sure to copy the cache file to a temporary name ... --- I've thought about rewriting `./configure` in C (I did it in Python once but Python's portability turned out to be poor - Python2 was bug-free but killed; Python3 was unfixably buggy for a decade or so). Still have a stub shell script that reads HOSTCC etc. then quickly builds and executes `./configure.bin`.

TacticalCoder 2y ago

[dead]

asah 2y ago

For postgresql development, you run configure over and over...