3 ms·

What does Electric Make do? Have you considered Landlock Make? https://github.com/jart/landlock-make https://github.com/jart/landlock-make It's a GNU Make fork

by jart 4y ago

What does Electric Make do? Have you considered Landlock Make? https://github.com/jart/landlock-make https://github.com/jart/landlock-make It's a GNU Make fork that uses pledge() and unveil() which is based on the Landlock security module which was introduced into the Linux Kernel two years ago. https://justine.lol/make/ https://justine.lol/make/ Using this tool, it becomes impossible to specify build rules that fail to declare dependencies, because userspace sandboxing prevents it. This sandboxing goes 2x faster than Bazel. There's also an mkdeps.com program for C/C++ projects that can generate dependencies for 1m+ LOC projects in ~100ms.

jgrahamc 4y ago
Runs the make job in parallel on remote machines but maintains state about file accesses in order to detect order problems and dynamically reruns the build healing the problem in real time. Then writes out the correct build order so that subsequent builds can be arbitrarily parallel. It was/is a clever combination of remotely running jobs, understand the Makefile DAG of what needs to be built and observing file system accesses to detect when the DAG doesn't reflect the files that are actually accessed (and therefore cause parallelism problems). The initial version was built by a small group of people led by me and John Ousterhout.
t43562 4y ago
Those look interesting! I'll try them out. Electric Make is the large scale makefile maintainers dream basically. You're not going to get that much out of it for a small linux package but once you have a build that can manage a lot of parallelism it makes life easy. It makes your build run across a cluster of machines but presents a filesystem on each machine that makes your build think it's on one computer. It has a filesystem module that does this and which also perfectly spots all your dependencies even when they're not C or C++ header files - just any file that got used when a target was being made. It's so smart that it spots parallelism issues as they happen and reruns everything in the correct ordering that they would have got run in if the build was running on a single core - so it can make horrible old makefiles work properly in parallel without you needing to update them. It remembers what happened so that it gets everything right first time in the next build. It remembers what build times things had and schedules longer items earlier. Its dependency mechanism lets it cache build objects with great certainty that you'll never pull a stale item out of a the cache. And the icing on the cake is that it has a visualisation tool that lets you look at your build across all CPUs/computers and see almost at a glance why it is slow and how to fix it. It's just the last word. But it is not free at all and again my opinion is that it's mostly going to pay off on larger builds. I also think that if you're building something where the developers take build time very seriously and fix problems a lot then it won't help as much.

by jart 4y ago

jgrahamc 4y ago

Runs the make job in parallel on remote machines but maintains state about file accesses in order to detect order problems and dynamically reruns the build healing the problem in real time. Then writes out the correct build order so that subsequent builds can be arbitrarily parallel. It was/is a clever combination of remotely running jobs, understand the Makefile DAG of what needs to be built and observing file system accesses to detect when the DAG doesn't reflect the files that are actually accessed (and therefore cause parallelism problems). The initial version was built by a small group of people led by me and John Ousterhout.

t43562 4y ago

Those look interesting! I'll try them out. Electric Make is the large scale makefile maintainers dream basically. You're not going to get that much out of it for a small linux package but once you have a build that can manage a lot of parallelism it makes life easy. It makes your build run across a cluster of machines but presents a filesystem on each machine that makes your build think it's on one computer. It has a filesystem module that does this and which also perfectly spots all your dependencies even when they're not C or C++ header files - just any file that got used when a target was being made. It's so smart that it spots parallelism issues as they happen and reruns everything in the correct ordering that they would have got run in if the build was running on a single core - so it can make horrible old makefiles work properly in parallel without you needing to update them. It remembers what happened so that it gets everything right first time in the next build. It remembers what build times things had and schedules longer items earlier. Its dependency mechanism lets it cache build objects with great certainty that you'll never pull a stale item out of a the cache. And the icing on the cake is that it has a visualisation tool that lets you look at your build across all CPUs/computers and see almost at a glance why it is slow and how to fix it. It's just the last word. But it is not free at all and again my opinion is that it's mostly going to pay off on larger builds. I also think that if you're building something where the developers take build time very seriously and fix problems a lot then it won't help as much.