6 ms·

This is great. Short of having Electric Make and Electric Insight it's the best thing you can get in open source and I think shuffling should really be the defa

by t43562 4y ago

This is great. Short of having Electric Make and Electric Insight it's the best thing you can get in open source and I think shuffling should really be the default to stop people from making horrendous parallel unsafe makefiles in the first place - so that the rest of us never have to fix them.

gt743268 4y ago
Pretty sure the make maintainers would be drawn and quartered if they did that. So many things - probably most of the things - would stop building. "Thy build shall be nondeterministic!" is quite a curse to spring on someone :)
- t43562 4y ago
  These problems come out anyhow when people run the build on a different machine with more or less cores or other issues then the people who are just users of the build have a problem. The worst problems are when nobody realises that a parallelism issue has happened and the binary for some tool on your phone OS gets generated from the old rule instead of the new one and it's just an invisible random bug. Not sure shuffling would help that exact problem of course.
  - pixl97 4y ago
    Invisible until it opens up a security exploit, oops that library doesn't have its security flags set.
- Arainach 4y ago
  If you want to argue this could break legacy products, that's a valid reason why they shouldn't do this. For new development, however, a nondeterministic default ensures no one relies on accidental behavior and is good practice. Golang did this with the order of traversing a map and it's wonderful in practice - if you want to depend on an ordering, sort it yourself.
  - blauditore 4y ago
    I disagree: The default should be deterministic but slow, and one should have to go out of their way to e.g. parallelize stuff, at least just enough to understand the risk of it becoming non-deterministic if done wrong.
    - Arainach 4y ago
      Determinism is an implementation that limits your future development. Is a faster sort available? You can't use it. Could you do some of your internal implementation in parallel? You can't do that either, since some customer is relying on exactly how things work today. Backwards compatibility is one of the most important things to do to keep people using your product - if I have to debug and change my code to use version 2 of your product, it's not much more work to switch to a competitor. It's also incredibly difficult to deliver backwards compatibility and requires intentional planning in advance.
    - TeMPOraL 4y ago
      > The default should be deterministic but slow That all but ensures nobody will care then. Performance is, sadly, not a big motivator in most projects.
      - ProblemFactory 4y ago
        Developers will care if they are spending their time waiting for a build. Both could be satisfied with a single-threaded make keeps the standard order to ensure backwards compatibility, but `make -j` that also turns on --shuffle. Which is also backwards compatible, because timing of each target might rearrange them.
- licebmi__at__ 4y ago
  I would argue that builds would break are already non deterministic, they just appear to be deterministic on "your" machine.
jgrahamc 4y ago
Thanks for mentioning Electric Make, I did a lot of work on that :-)
- deng 4y ago
  While we're at it: Thanks for writing your book on GNU Make, it was a godsend years ago when I had to convert a big recursive Make build to be non-recursive (and that new shuffle feature would definitely have been a big help).
  - jgrahamc 4y ago
    That's kind. That book came about because I wrote the Electric Make implementation of GNU Make and learnt a lot about it and then I worked with our customers on massive Makefiles and saw a lot of things.
- natrys 4y ago
  I love your book on GNU Make :D
  - jgrahamc 4y ago
    Thanks!
- t43562 4y ago
  I know! :-D I did a small bit of work on it but long after you! I now selfishly wish that it was open source so I could still use it. :-)
jart 4y ago
What does Electric Make do? Have you considered Landlock Make? https://github.com/jart/landlock-make https://github.com/jart/landlock-make It's a GNU Make fork that uses pledge() and unveil() which is based on the Landlock security module which was introduced into the Linux Kernel two years ago. https://justine.lol/make/ https://justine.lol/make/ Using this tool, it becomes impossible to specify build rules that fail to declare dependencies, because userspace sandboxing prevents it. This sandboxing goes 2x faster than Bazel. There's also an mkdeps.com program for C/C++ projects that can generate dependencies for 1m+ LOC projects in ~100ms.
- jgrahamc 4y ago
  Runs the make job in parallel on remote machines but maintains state about file accesses in order to detect order problems and dynamically reruns the build healing the problem in real time. Then writes out the correct build order so that subsequent builds can be arbitrarily parallel. It was/is a clever combination of remotely running jobs, understand the Makefile DAG of what needs to be built and observing file system accesses to detect when the DAG doesn't reflect the files that are actually accessed (and therefore cause parallelism problems). The initial version was built by a small group of people led by me and John Ousterhout.
- t43562 4y ago
  Those look interesting! I'll try them out. Electric Make is the large scale makefile maintainers dream basically. You're not going to get that much out of it for a small linux package but once you have a build that can manage a lot of parallelism it makes life easy. It makes your build run across a cluster of machines but presents a filesystem on each machine that makes your build think it's on one computer. It has a filesystem module that does this and which also perfectly spots all your dependencies even when they're not C or C++ header files - just any file that got used when a target was being made. It's so smart that it spots parallelism issues as they happen and reruns everything in the correct ordering that they would have got run in if the build was running on a single core - so it can make horrible old makefiles work properly in parallel without you needing to update them. It remembers what happened so that it gets everything right first time in the next build. It remembers what build times things had and schedules longer items earlier. Its dependency mechanism lets it cache build objects with great certainty that you'll never pull a stale item out of a the cache. And the icing on the cake is that it has a visualisation tool that lets you look at your build across all CPUs/computers and see almost at a glance why it is slow and how to fix it. It's just the last word. But it is not free at all and again my opinion is that it's mostly going to pay off on larger builds. I also think that if you're building something where the developers take build time very seriously and fix problems a lot then it won't help as much.
m463 4y ago
I remember checking out electric make years ago for a project. It was from electric cloud and I believe it was somehow related to the creator of tcl. It would hook into the filesystem to understand everything that was being built. If there was a hidden dependency it would record it and could restart steps to build in the correct order. It was pretty smart - I suspect you could write a really crappy makefile and coast. this was probably 20 years ago.

by t43562 4y ago

gt743268 4y ago

Pretty sure the make maintainers would be drawn and quartered if they did that. So many things - probably most of the things - would stop building. "Thy build shall be nondeterministic!" is quite a curse to spring on someone :)

t43562 4y ago
These problems come out anyhow when people run the build on a different machine with more or less cores or other issues then the people who are just users of the build have a problem. The worst problems are when nobody realises that a parallelism issue has happened and the binary for some tool on your phone OS gets generated from the old rule instead of the new one and it's just an invisible random bug. Not sure shuffling would help that exact problem of course.
- pixl97 4y ago
  Invisible until it opens up a security exploit, oops that library doesn't have its security flags set.
Arainach 4y ago
If you want to argue this could break legacy products, that's a valid reason why they shouldn't do this. For new development, however, a nondeterministic default ensures no one relies on accidental behavior and is good practice. Golang did this with the order of traversing a map and it's wonderful in practice - if you want to depend on an ordering, sort it yourself.
- blauditore 4y ago
  I disagree: The default should be deterministic but slow, and one should have to go out of their way to e.g. parallelize stuff, at least just enough to understand the risk of it becoming non-deterministic if done wrong.
  - Arainach 4y ago
    Determinism is an implementation that limits your future development. Is a faster sort available? You can't use it. Could you do some of your internal implementation in parallel? You can't do that either, since some customer is relying on exactly how things work today. Backwards compatibility is one of the most important things to do to keep people using your product - if I have to debug and change my code to use version 2 of your product, it's not much more work to switch to a competitor. It's also incredibly difficult to deliver backwards compatibility and requires intentional planning in advance.
  - TeMPOraL 4y ago
    > The default should be deterministic but slow That all but ensures nobody will care then. Performance is, sadly, not a big motivator in most projects.
    - ProblemFactory 4y ago
      Developers will care if they are spending their time waiting for a build. Both could be satisfied with a single-threaded make keeps the standard order to ensure backwards compatibility, but `make -j` that also turns on --shuffle. Which is also backwards compatible, because timing of each target might rearrange them.
licebmi__at__ 4y ago
I would argue that builds would break are already non deterministic, they just appear to be deterministic on "your" machine.

jgrahamc 4y ago

Thanks for mentioning Electric Make, I did a lot of work on that :-)

deng 4y ago
While we're at it: Thanks for writing your book on GNU Make, it was a godsend years ago when I had to convert a big recursive Make build to be non-recursive (and that new shuffle feature would definitely have been a big help).
- jgrahamc 4y ago
  That's kind. That book came about because I wrote the Electric Make implementation of GNU Make and learnt a lot about it and then I worked with our customers on massive Makefiles and saw a lot of things.
natrys 4y ago
I love your book on GNU Make :D
- jgrahamc 4y ago
  Thanks!
t43562 4y ago
I know! :-D I did a small bit of work on it but long after you! I now selfishly wish that it was open source so I could still use it. :-)

jart 4y ago

What does Electric Make do? Have you considered Landlock Make? https://github.com/jart/landlock-make https://github.com/jart/landlock-make It's a GNU Make fork that uses pledge() and unveil() which is based on the Landlock security module which was introduced into the Linux Kernel two years ago. https://justine.lol/make/ https://justine.lol/make/ Using this tool, it becomes impossible to specify build rules that fail to declare dependencies, because userspace sandboxing prevents it. This sandboxing goes 2x faster than Bazel. There's also an mkdeps.com program for C/C++ projects that can generate dependencies for 1m+ LOC projects in ~100ms.

jgrahamc 4y ago
Runs the make job in parallel on remote machines but maintains state about file accesses in order to detect order problems and dynamically reruns the build healing the problem in real time. Then writes out the correct build order so that subsequent builds can be arbitrarily parallel. It was/is a clever combination of remotely running jobs, understand the Makefile DAG of what needs to be built and observing file system accesses to detect when the DAG doesn't reflect the files that are actually accessed (and therefore cause parallelism problems). The initial version was built by a small group of people led by me and John Ousterhout.
t43562 4y ago
Those look interesting! I'll try them out. Electric Make is the large scale makefile maintainers dream basically. You're not going to get that much out of it for a small linux package but once you have a build that can manage a lot of parallelism it makes life easy. It makes your build run across a cluster of machines but presents a filesystem on each machine that makes your build think it's on one computer. It has a filesystem module that does this and which also perfectly spots all your dependencies even when they're not C or C++ header files - just any file that got used when a target was being made. It's so smart that it spots parallelism issues as they happen and reruns everything in the correct ordering that they would have got run in if the build was running on a single core - so it can make horrible old makefiles work properly in parallel without you needing to update them. It remembers what happened so that it gets everything right first time in the next build. It remembers what build times things had and schedules longer items earlier. Its dependency mechanism lets it cache build objects with great certainty that you'll never pull a stale item out of a the cache. And the icing on the cake is that it has a visualisation tool that lets you look at your build across all CPUs/computers and see almost at a glance why it is slow and how to fix it. It's just the last word. But it is not free at all and again my opinion is that it's mostly going to pay off on larger builds. I also think that if you're building something where the developers take build time very seriously and fix problems a lot then it won't help as much.

m463 4y ago

I remember checking out electric make years ago for a project. It was from electric cloud and I believe it was somehow related to the creator of tcl. It would hook into the filesystem to understand everything that was being built. If there was a hidden dependency it would record it and could restart steps to build in the correct order. It was pretty smart - I suspect you could write a really crappy makefile and coast. this was probably 20 years ago.