3 ms·
Do you know why isn't it easier to offload compilation to the cloud yet? Compilation is an embarrassingly parallel problem and shouldn't need take longer than t
by chrisseaton 7y ago
Do you know why isn't it easier to offload compilation to the cloud yet? Compilation is an embarrassingly parallel problem and shouldn't need take longer than than slowest file plus linking. Compilation is also a pure function (I know it's not quite set up like that in practice but it is in theory) so it's stateless and cacheable. I would have thought developers could share a massive cluster and have thousand-way compilation instantly. Doesn't seem to be a thing people are building or using in practice for some reason.
- saurik 7y agoI nearly have this working, using AWS Lambda (a few months ago I got super angry at the compile times for a project I was working on and spent a week getting a proof of concept ready that I have been refining since). Have sadly been busy with some other tasks I have to get done (age old story), but intend to push to the point where you could easily add it to your own project soon (as long as your build system isn't stupid; a lot of people think they are using super advanced build systems, but in practice they handle toolchain hacks extremely poorly :/).
- chrisseaton 7y agoIf I was writing a new compiler from scratch today I think I'd make it client-server from the start, even locally, so that the compiler could be transparently made remote. The client would be responsible for doing all IO on behalf of the server so that the server was completely separated from the system it ran on.
- deleted 7y ago[deleted]
- scott_s 7y agodistcc does this for C and C++: https://distcc.github.io/ https://distcc.github.io/
- milkey_mouse 7y agohttps://github.com/StanfordSNR/gg https://github.com/StanfordSNR/gg can do builds in the cloud. Their USENIX talk is really interesting
- saghm 7y agoDevelopers at my company who work on a large shared C++ codebase all use Icecream[0] to distribute the builds across their workstations. I've only hooked into it to build once since I don't work in that codebase, but it seems to work pretty well scaling it (I think on the order of 50-60 servers and about the same number of clients). As for caching, I think sccache[1] is designed to handle that in a scalable way, although I've personally only used it locally for my Rust builds. [0]: https://github.com/icecc/icecream https://github.com/icecc/icecream [1]: https://github.com/mozilla/sccache https://github.com/mozilla/sccache