34 ms·

Distcc – distribute builds across multiple machines simultaneously (2006)

39 points by Scoobs 6y ago

prussian 6y ago
If you like distcc, perhaps you'd also like icecream https://github.com/icecc/icecream https://github.com/icecc/icecream which I think is a bit easier to use.
- baybal2 6y ago
  I'm using it. It's reaction to network degradation, even one just moderately worse off than LAN with a single switch really hampers usage on cloud servers.
- aidanhs 6y ago
  I worked on adding distributed compilation to sccache [0]. Docs at [1] and [2]. Compared to existing tools, sccache supports: - local caching (like ccache) - remote caching e.g. to S3, or a LAN redis instance (unique afaik) - distributed compilation of C/C++ code (like distcc, icecream) - distributed compilation of Rust (unique afaik) - distributed compilation on Windows by cross compiling on Linux machines (unique afaik) Note that I think bazel also does a bunch of these, but you need to use bazel as your build system. [0] https://github.com/mozilla/sccache https://github.com/mozilla/sccache [1] quickstart - https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md https://github.com/mozilla/sccache/blob/master/docs/Distribu... [2] reference docs - https://github.com/mozilla/sccache/blob/master/docs/Distributed.md https://github.com/mozilla/sccache/blob/master/docs/Distribu...
  - prussian 6y ago
    I've been contemplating how to add ccache to my work's current icecream cluster and I also considered the possibility of using it for non-release branch dev builds in CI. I'll have to experiment with this instead. Thank you.
arcticbull 6y ago
Is distcc actually any faster than running the builds locally on a half decent machine? If I recall the latency associated with getting the file to the build machines often exceeded the local compilation duration. I remember being really excited about the idea, and really disappointed with the results.
- derekp7 6y ago
  One use that I can think of (never tried it myself, though in theory it should work) is if you are trying to cross compile for a processor architecture that doesn't have decent speed (many embedded chips), and the package you are compiling is difficult to cross compile (where the build script makes decisions based on the local environment or is otherwise to get to play nice with a cross compiler). In that case you can run the build on the target CPU, but instead of calling gcc locally it calls distcc pointing to a cross compiler installed on a fast machine. This can be useful if you are compiling a bunch of packages from a distribution.
  - LukeShu 6y ago
    Indeed, in my use (with the Parabola GNU/Linux-libre distro), this is the killer use-case of distcc. This is how most of Parabola's packages for MIPS were built, and how many for ARM are built.
  - rvdginste 6y ago
    I've used distcc in that way long time ago. I had an powerpc iBook (G4) and a x86 desktop (Pentium4). The desktop helped compile software on the iBook using distcc and a cross-compiler. It did work pretty well. I also used ccache in addition to that to cache build output. That was probably around 2005 when most (all?) cpus were still single-core.
- hansdieter1337 6y ago
  Running it locally will always be faster as long as your machine is not a bottleneck (#cores, ram, ...). I think the use-case for distcc et al is to enable less-powerful machines to run builds faster by levering other machines. That’s exactly what we use it for at work. Our developers have not-so-powerful laptops and with distcc/icecc they can utilize the power of our build agents in the server room. Also interesting to read: https://github.com/StanfordSNR/gg https://github.com/StanfordSNR/gg
new_realist 6y ago
Just use Bazel with remote execution. It can distribute your unit tests, too.
- hansdieter1337 6y ago
  And cache build outputs: https://docs.bazel.build/versions/master/remote-caching.html https://docs.bazel.build/versions/master/remote-caching.html
- silasdb 6y ago
  Not the same thing. distcc is a wrapper to a compiler that can work with any build system. Just use CC=distcc.
  - klodolph 6y ago
    I don’t think anyone is claiming that Bazel is “the same thing” as distcc. But they both solve the problem of distributed builds.
    - emelski 6y ago
      But unless you're already using Bazel, it's impractical to suggest "just use Bazel to distribute your builds" as doing so requires that you first migrate your build to Bazel. distcc, as noted, can be applied to an existing build regardless of the build tool used.
      - klodolph 6y ago
        So, you’re saying that it’s impractical to switch build systems, and that it’s inappropriate to suggest alternative build systems? I think that if you’re reading a suggestion to “just use X” on Hacker News, the sensible thing to do is to evaluate the suggestion for your own. You don’t need to put a long list of caveats in front of every software recommendation. If your problem is, “My builds are slow, I want distributed builds,” then you should probably at least consider the two most popular solutions to that problem, which are Bazel and Distcc. Both with their drawbacks.
OskarS 6y ago
Does this cache build artifacts in any way? So if several developers are working on the same code-base, they can share build artifacts based on, like, the hash of the inputs? That would require reproducible builds, but that's probably not too tricky to achieve.
- hansdieter1337 6y ago
  It doesn’t IIRC. But, you can combine it with ccache which can use a NFS share. If you use clang, that might be of interest: https://github.com/yrnkrn/zapcc https://github.com/yrnkrn/zapcc
- uvatbc 6y ago
  We've built Crave.io to share build cache between developers without resorting to customer supplied NFS or other storage bottlenecks.
  - raxrb 6y ago
    This looks interesting. Up for a chat sometime?
Twirrim 6y ago
I think I last used distcc back in the mid 2000s, when I was mucking around with Gentoo a bit more frequently, on a single core machine. I had two machines, one running Gentoo, one running Windows, and I would use distcc to enable me to use the Windows machine to help boost the compilation speed of Gentoo. I can't remember where the documentation was, but http://wikigentoo.ksiezyc.pl/HOWTO_Distcc_server_on_Windows.htm http://wikigentoo.ksiezyc.pl/HOWTO_Distcc_server_on_Windows.... seems to do a good job of capturing the general concept.

39 points by Scoobs 6y ago

If you like distcc, perhaps you'd also like icecream https://github.com/icecc/icecream https://github.com/icecc/icecream which I think is a bit easier to use.

baybal2 6y ago
I'm using it. It's reaction to network degradation, even one just moderately worse off than LAN with a single switch really hampers usage on cloud servers.
aidanhs 6y ago
I worked on adding distributed compilation to sccache [0]. Docs at [1] and [2]. Compared to existing tools, sccache supports: - local caching (like ccache) - remote caching e.g. to S3, or a LAN redis instance (unique afaik) - distributed compilation of C/C++ code (like distcc, icecream) - distributed compilation of Rust (unique afaik) - distributed compilation on Windows by cross compiling on Linux machines (unique afaik) Note that I think bazel also does a bunch of these, but you need to use bazel as your build system. [0] https://github.com/mozilla/sccache https://github.com/mozilla/sccache [1] quickstart - https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md https://github.com/mozilla/sccache/blob/master/docs/Distribu... [2] reference docs - https://github.com/mozilla/sccache/blob/master/docs/Distributed.md https://github.com/mozilla/sccache/blob/master/docs/Distribu...
- prussian 6y ago
  I've been contemplating how to add ccache to my work's current icecream cluster and I also considered the possibility of using it for non-release branch dev builds in CI. I'll have to experiment with this instead. Thank you.

arcticbull 6y ago

Is distcc actually any faster than running the builds locally on a half decent machine? If I recall the latency associated with getting the file to the build machines often exceeded the local compilation duration. I remember being really excited about the idea, and really disappointed with the results.

derekp7 6y ago
One use that I can think of (never tried it myself, though in theory it should work) is if you are trying to cross compile for a processor architecture that doesn't have decent speed (many embedded chips), and the package you are compiling is difficult to cross compile (where the build script makes decisions based on the local environment or is otherwise to get to play nice with a cross compiler). In that case you can run the build on the target CPU, but instead of calling gcc locally it calls distcc pointing to a cross compiler installed on a fast machine. This can be useful if you are compiling a bunch of packages from a distribution.
- LukeShu 6y ago
  Indeed, in my use (with the Parabola GNU/Linux-libre distro), this is the killer use-case of distcc. This is how most of Parabola's packages for MIPS were built, and how many for ARM are built.
- rvdginste 6y ago
  I've used distcc in that way long time ago. I had an powerpc iBook (G4) and a x86 desktop (Pentium4). The desktop helped compile software on the iBook using distcc and a cross-compiler. It did work pretty well. I also used ccache in addition to that to cache build output. That was probably around 2005 when most (all?) cpus were still single-core.
hansdieter1337 6y ago
Running it locally will always be faster as long as your machine is not a bottleneck (#cores, ram, ...). I think the use-case for distcc et al is to enable less-powerful machines to run builds faster by levering other machines. That’s exactly what we use it for at work. Our developers have not-so-powerful laptops and with distcc/icecc they can utilize the power of our build agents in the server room. Also interesting to read: https://github.com/StanfordSNR/gg https://github.com/StanfordSNR/gg

new_realist 6y ago

Just use Bazel with remote execution. It can distribute your unit tests, too.

hansdieter1337 6y ago
And cache build outputs: https://docs.bazel.build/versions/master/remote-caching.html https://docs.bazel.build/versions/master/remote-caching.html
silasdb 6y ago
Not the same thing. distcc is a wrapper to a compiler that can work with any build system. Just use CC=distcc.
- klodolph 6y ago
  I don’t think anyone is claiming that Bazel is “the same thing” as distcc. But they both solve the problem of distributed builds.
  - emelski 6y ago
    But unless you're already using Bazel, it's impractical to suggest "just use Bazel to distribute your builds" as doing so requires that you first migrate your build to Bazel. distcc, as noted, can be applied to an existing build regardless of the build tool used.
    - klodolph 6y ago
      So, you’re saying that it’s impractical to switch build systems, and that it’s inappropriate to suggest alternative build systems? I think that if you’re reading a suggestion to “just use X” on Hacker News, the sensible thing to do is to evaluate the suggestion for your own. You don’t need to put a long list of caveats in front of every software recommendation. If your problem is, “My builds are slow, I want distributed builds,” then you should probably at least consider the two most popular solutions to that problem, which are Bazel and Distcc. Both with their drawbacks.

OskarS 6y ago

Does this cache build artifacts in any way? So if several developers are working on the same code-base, they can share build artifacts based on, like, the hash of the inputs? That would require reproducible builds, but that's probably not too tricky to achieve.

hansdieter1337 6y ago
It doesn’t IIRC. But, you can combine it with ccache which can use a NFS share. If you use clang, that might be of interest: https://github.com/yrnkrn/zapcc https://github.com/yrnkrn/zapcc
uvatbc 6y ago
We've built Crave.io to share build cache between developers without resorting to customer supplied NFS or other storage bottlenecks.
- raxrb 6y ago
  This looks interesting. Up for a chat sometime?

Twirrim 6y ago

I think I last used distcc back in the mid 2000s, when I was mucking around with Gentoo a bit more frequently, on a single core machine. I had two machines, one running Gentoo, one running Windows, and I would use distcc to enable me to use the Windows machine to help boost the compilation speed of Gentoo. I can't remember where the documentation was, but http://wikigentoo.ksiezyc.pl/HOWTO_Distcc_server_on_Windows.htm http://wikigentoo.ksiezyc.pl/HOWTO_Distcc_server_on_Windows.... seems to do a good job of capturing the general concept.