6 ms·

Nocc – A Distributed C++ Compiler

63 points by gjvc 2y ago

7e 2y ago
Why not just use Bazel?
- jcelerier 2y ago
  How do I do if I want to make something with bazel that is going to use libraries on the host computer? Say, I want to make an app that builds against system-provided Qt6, does Bazel finally allow this?
  - malkia 2y ago
    Haven't tried it, but you can always `cc_import` a locally installed library and expose it - though not sure how this works across the wire... Then again, this might be all asking for trouble - what if your colleague has different installed version. Best is to have these artifacts (recompiled and/or in source form) some other way.
    - rfoo 2y ago
      > though not sure how this works across the wire You send all header files to 1,000 boxes. Same as nocc. Oh, and if you really want, you can make Bazel ship your /usr/bin/gcc and co, too. It is just slow so nobody would like that. Or maybe in 2025 it does not feel slow any more (though still wasteful)?
      - malkia 2y ago
        I think at some point, but could be misremembering, IncrediBuild did local preprocessing, and then all you need on the other end is compiler. It might be still an option if pre-processing (for your own codebase) is insignificant vs compilation, but hard to say. Though it greatly simplifies things with little massage (e.g. all these #pragma line - and make them deterministic using some path mapping, etc.)
    - jcelerier 2y ago
      > what if your colleague has different installed version. "colleagues" does not make much sense for an open-source project built by people all around the world with various environments and where the tool should accomodate all of them
      - malkia 2y ago
        Right, I'm giving my thoughts based on where I work, and actually we do enforce the same compiler for everyone, even where it's installed, but still..
MathMonkeyMan 2y ago
Neat! I remember using [something similar][1] at a previous job. [1]: https://github.com/icecc/icecream https://github.com/icecc/icecream
- o11c 2y ago
  That appears to only handle the distcc half. It looks like the main advantage of nocc is that it integrates the equivalent of ccache, in a way that works even when there are both multiple clients and multiple servers, without the ludicrous complexity of bazel-like tools.
  - 7e 2y ago
    Did the authors also write their own operating system to avoid the “ludicrous complexity” of the Linux kernel? This project just looks like NIH syndrome and an unwillingness to read documentation.
  - menaerus 2y ago
    Not true. icecream handles that case very well. Any machine in the network can trigger the build. And all the machines in the network can register themselves as build nodes. It also integrates with the ccache.
    - o11c 2y ago
      `ccache icecream` (like `ccache distcc`) only handles the "single client, multiple servers" case. `icecream ccache` (like `distcc ccache`) only handles the "multiple client, single server" case. `ccache icecream ccache` (like `ccache distcc ccache`) just means you're wasting even more disk space. Now, if you configure `ccache` to use remote storage, and you can make it actually work, then that combination will compete with `nocc` (though admittedly it follows different models). But that's a lot more moving parts.
      - menaerus 2y ago
        I am not sure I follow your argument. You said that icecream cannot work in multi-client multi-server case. I said that it certainly can, and it can. Perhaps you're arguing that in that multi-case scenario ccache is not usable? That I don't know but it definitely works when ccache is setup on the client machine and this client machine is using whatever number of server machines for the build jobs.
malkia 2y ago
Some other alternatives, also haven't hearded about this one: - SN DBS - https://www.snsystems.com/ - Used by a lot of game developers, to spread mostly compilation (but also shader compile, or custom jobs). - IncrediBuild - https://www.incredibuild.com/ - Fast build - https://www.fastbuild.org/ - icecream - https://github.com/icecc/icecream - Goma - https://chromium.googlesource.com/infra/goma/client/ - Bazel / buck / like with various RBE back ends - https://bazel.build/remote/rbe - distcc - https://www.distcc.org/ - ElectricAccelerator - https://docs.cloudbees.com/docs/cloudbees-build-acceleration/11.0/ - and many others... I've had mostly experience with IncrediBuild in the past, currently SN-DBS, but colleagues are looking into FASTBuild. Though my personal favorite is bazel
- jcelerier 2y ago
  From the README this tool has been built because distcc was too slow
  - gjvc 2y ago
    Yes. This makes sense.
  - menaerus 2y ago
    distcc was really slow IME as well. It did not know how to optimally distribute the work across different build nodes. AFAIR distcc simply assumed that all build nodes are created equal and it also disregarded the load on the nodes resulting in over-saturation with build jobs. However, this is something that icecream aimed to solve, and in my case it really did. I can't remember the actual numbers but it provided a major gain and it was super easy to set up. So I guess it would be more nteresting to see how nocc compares to icecream instead of distcc.
- bobmcnamara 2y ago
  We discovered that while incredibuild did trap the system calls needed to stream and cache files from the building computer, at the time they did not trap Beep()!
o11c 2y ago
It looks like there are some infelicities documented that really should be fixed: * fix that silly "got another sha256" bug, it really shouldn't be hard * optimal job count should be infinite from the perspective of the client's `make`, but the daemon should only send jobs to the servers based on how many jobs they are willing to perform in parallel. Possibly split out the "upload blobs" part from the "actually run the build" part? * You should detect compiler version differences, which is essential for reliability. This is nontrivial (wrapper scripts, internal headers) but good solutions exist if you're willing to hard-code what compilers are supported.
deleted 2y ago
[deleted]
nasretdinov 2y ago
The funny thing about the name of the tool is that it's essentially named after a PHP linter I wrote while working at VK, called NoVerify. I did an internal poll for the best name, and noverify won, because the way to skip the checks done that linter was "git push --no-verify"...
- mappu 2y ago
  Thank you for noverify! We used it at $DAYJOB in CI.
  - nasretdinov 2y ago
    No problem! If you don't mind sharing, what are you using it for? I've built NoVerify primarily because the VK PHP code base was quite old-school, so it relied a lot on global functions, global constants, global variables, etc, so I needed a way to quickly make an index of all of what's defined, and it required pararallelism with shared memory which PHP doesn't support. It turned out to be quite fast in general as well, but it lacks more advanced features that other PHP linters have (mostly because I spent a few months on it vs. a few years for others :))
sixthDot 2y ago
Would you say that this tool is required because cpp is slow to compile or is it still useful even with a module system (or whatever would make it a fast-to-compile-proglang) ?
octocop 2y ago
Distributed C++ Build System? It's not a compiler, like it's stated in the readme, it uses g++..
rurban 2y ago
(2022) They use KPHP which creates huge and a lot of files, similar to meta's php compiler or my B::C perl compiler in use at cPanel. In my case the outstanding work was to avoid compilation at all, split it up into many modules/libs, and only compile those which changed. The mnain advantage is instant startup and less memory. With many shared libs the startup advantage is less, but you can still use static libs and use a modern fast linker. And avoid -g and use -Os

63 points by gjvc 2y ago

7e 2y ago

Why not just use Bazel?

jcelerier 2y ago
How do I do if I want to make something with bazel that is going to use libraries on the host computer? Say, I want to make an app that builds against system-provided Qt6, does Bazel finally allow this?
- malkia 2y ago
  Haven't tried it, but you can always `cc_import` a locally installed library and expose it - though not sure how this works across the wire... Then again, this might be all asking for trouble - what if your colleague has different installed version. Best is to have these artifacts (recompiled and/or in source form) some other way.
  - rfoo 2y ago
    > though not sure how this works across the wire You send all header files to 1,000 boxes. Same as nocc. Oh, and if you really want, you can make Bazel ship your /usr/bin/gcc and co, too. It is just slow so nobody would like that. Or maybe in 2025 it does not feel slow any more (though still wasteful)?
    - malkia 2y ago
      I think at some point, but could be misremembering, IncrediBuild did local preprocessing, and then all you need on the other end is compiler. It might be still an option if pre-processing (for your own codebase) is insignificant vs compilation, but hard to say. Though it greatly simplifies things with little massage (e.g. all these #pragma line - and make them deterministic using some path mapping, etc.)
  - jcelerier 2y ago
    > what if your colleague has different installed version. "colleagues" does not make much sense for an open-source project built by people all around the world with various environments and where the tool should accomodate all of them
    - malkia 2y ago
      Right, I'm giving my thoughts based on where I work, and actually we do enforce the same compiler for everyone, even where it's installed, but still..

MathMonkeyMan 2y ago

Neat! I remember using [something similar][1] at a previous job. [1]: https://github.com/icecc/icecream https://github.com/icecc/icecream

o11c 2y ago
That appears to only handle the distcc half. It looks like the main advantage of nocc is that it integrates the equivalent of ccache, in a way that works even when there are both multiple clients and multiple servers, without the ludicrous complexity of bazel-like tools.
- 7e 2y ago
  Did the authors also write their own operating system to avoid the “ludicrous complexity” of the Linux kernel? This project just looks like NIH syndrome and an unwillingness to read documentation.
- menaerus 2y ago
  Not true. icecream handles that case very well. Any machine in the network can trigger the build. And all the machines in the network can register themselves as build nodes. It also integrates with the ccache.
  - o11c 2y ago
    `ccache icecream` (like `ccache distcc`) only handles the "single client, multiple servers" case. `icecream ccache` (like `distcc ccache`) only handles the "multiple client, single server" case. `ccache icecream ccache` (like `ccache distcc ccache`) just means you're wasting even more disk space. Now, if you configure `ccache` to use remote storage, and you can make it actually work, then that combination will compete with `nocc` (though admittedly it follows different models). But that's a lot more moving parts.
    - menaerus 2y ago
      I am not sure I follow your argument. You said that icecream cannot work in multi-client multi-server case. I said that it certainly can, and it can. Perhaps you're arguing that in that multi-case scenario ccache is not usable? That I don't know but it definitely works when ccache is setup on the client machine and this client machine is using whatever number of server machines for the build jobs.

malkia 2y ago

Some other alternatives, also haven't hearded about this one: - SN DBS - https://www.snsystems.com/ - Used by a lot of game developers, to spread mostly compilation (but also shader compile, or custom jobs). - IncrediBuild - https://www.incredibuild.com/ - Fast build - https://www.fastbuild.org/ - icecream - https://github.com/icecc/icecream - Goma - https://chromium.googlesource.com/infra/goma/client/ - Bazel / buck / like with various RBE back ends - https://bazel.build/remote/rbe - distcc - https://www.distcc.org/ - ElectricAccelerator - https://docs.cloudbees.com/docs/cloudbees-build-acceleration/11.0/ - and many others... I've had mostly experience with IncrediBuild in the past, currently SN-DBS, but colleagues are looking into FASTBuild. Though my personal favorite is bazel

jcelerier 2y ago
From the README this tool has been built because distcc was too slow
- gjvc 2y ago
  Yes. This makes sense.
- menaerus 2y ago
  distcc was really slow IME as well. It did not know how to optimally distribute the work across different build nodes. AFAIR distcc simply assumed that all build nodes are created equal and it also disregarded the load on the nodes resulting in over-saturation with build jobs. However, this is something that icecream aimed to solve, and in my case it really did. I can't remember the actual numbers but it provided a major gain and it was super easy to set up. So I guess it would be more nteresting to see how nocc compares to icecream instead of distcc.
bobmcnamara 2y ago
We discovered that while incredibuild did trap the system calls needed to stream and cache files from the building computer, at the time they did not trap Beep()!

o11c 2y ago

It looks like there are some infelicities documented that really should be fixed: * fix that silly "got another sha256" bug, it really shouldn't be hard * optimal job count should be infinite from the perspective of the client's `make`, but the daemon should only send jobs to the servers based on how many jobs they are willing to perform in parallel. Possibly split out the "upload blobs" part from the "actually run the build" part? * You should detect compiler version differences, which is essential for reliability. This is nontrivial (wrapper scripts, internal headers) but good solutions exist if you're willing to hard-code what compilers are supported.

deleted 2y ago

[deleted]

nasretdinov 2y ago

The funny thing about the name of the tool is that it's essentially named after a PHP linter I wrote while working at VK, called NoVerify. I did an internal poll for the best name, and noverify won, because the way to skip the checks done that linter was "git push --no-verify"...

mappu 2y ago
Thank you for noverify! We used it at $DAYJOB in CI.
- nasretdinov 2y ago
  No problem! If you don't mind sharing, what are you using it for? I've built NoVerify primarily because the VK PHP code base was quite old-school, so it relied a lot on global functions, global constants, global variables, etc, so I needed a way to quickly make an index of all of what's defined, and it required pararallelism with shared memory which PHP doesn't support. It turned out to be quite fast in general as well, but it lacks more advanced features that other PHP linters have (mostly because I spent a few months on it vs. a few years for others :))

sixthDot 2y ago

Would you say that this tool is required because cpp is slow to compile or is it still useful even with a module system (or whatever would make it a fast-to-compile-proglang) ?

octocop 2y ago

Distributed C++ Build System? It's not a compiler, like it's stated in the readme, it uses g++..

rurban 2y ago

(2022) They use KPHP which creates huge and a lot of files, similar to meta's php compiler or my B::C perl compiler in use at cPanel. In my case the outstanding work was to avoid compilation at all, split it up into many modules/libs, and only compile those which changed. The mnain advantage is instant startup and less memory. With many shared libs the startup advantage is less, but you can still use static libs and use a modern fast linker. And avoid -g and use -Os