9 ms·

Sccache, Mozilla’s distributed compiler cache, now written in Rust

243 points by AndrewDucker 10y ago

jononor 10y ago
No tests? :( EDIT: my bad, they were inside src/
- cesarb 10y ago
  In Rust, it's very common to put tests right next to the code; all functions tagged with #[test] will be run as tests, and examples within the function documentation comments will also be run as tests by default. For instance, randomly looking at the first file in that repository, cache/cache.rs, you can see at the end a submodule tagged with #[cfg(test)] (meaning it will be compiled only when running the tests), and within it several functions tagged with #[test].
  - christophilus 10y ago
    I can't figure out whether or not I like this convention. I used to work with a guy who did the same thing with C++ and C# tests: right there alongside the code they tested. On the one hand, it is nice to have all of the relevant code in one place for context. On the other hand, it muddied up the actual business logic with tests, so you had to perform some mental work to figure out whether the code you were looking at was for production use or only for testing.
    - ryeguy 10y ago
      It's worth noting this convention is only for unit tests. Integration tests traditionally have their own test directory and files. Because of this, the file-inlined test code you see is normally relatively simple and shouldn't have a ton of extra helper functions and setup and such. The majority of the code should just be test cases themselves, which are clearly marked with #[test].
    - Yoric 10y ago
      Note that Rust lets you put tests in tests/ instead of src/. So typically, you put short unit tests in src/ and longer/integration tests in tests/. Oh, and examples that show up in the documentation are also run alongside tests. Also, the compiler will tell you if you have code that's not labelled with #[cfg(test)] but is designed to run only in tests.
      - tyoverby 10y ago
        The big difference with `tests/` is that they have to be integration tests. This means that you can't test private functions from there.
        Manishearth 10y ago
        Not .. exactly. An integration test usually tests an entire application at once. `tests/` can still test things in the public API (indeed, we do this in Servo a lot) without being an integration test. The point is that some unit tests cannot be written in tests/, whereas all integration tests can be.
    - Sean1708 10y ago
      It's so that you can do unit testing without having to do weird things like airing your privates in tests, since private modules/functions/etc. aren't visible to your integration tests (which live in tests/ and basically just use your code as a library).
    - Tuna-Fish 10y ago
      I started liking this convention when I started considering the tests documentation. In my code, the first unit test for any function is always the simple common case, and having that right next to the function usually makes the code easier to read.
      - steveklabnik 10y ago
        You should move it to a documentation block, given that doc examples are also run as tests. Then your users can see them too. Win-win!
- tedmielczarek 10y ago
  It does have a bunch of tests, although probably not enough. I've been meaning to add code coverage stats to its CI, although I've only used kcov for Rust code coverage so far, and that only works on Linux it wouldn't reflect reality very well. I've done a lot of work on test automation at Mozilla over the years so automated testing is pretty important to me. :) I've tried to write unit tests where they made sense, they're generally placed within the relevant source file in the tree, like the cache tests here: https://github.com/mozilla/sccache/blob/6eadccc9d6752747766d550388590199d87f23a5/src/cache/cache.rs#L263 https://github.com/mozilla/sccache/blob/6eadccc9d6752747766d... I also wrote some higher-level tests that test the server functionality: https://github.com/mozilla/sccache/blob/master/src/test/tests.rs https://github.com/mozilla/sccache/blob/master/src/test/test... I rolled my own set of traits and structs to mock process execution for those tests so they would run independent of the compilers installed on the system: https://github.com/mozilla/sccache/blob/master/src/mock_command.rs https://github.com/mozilla/sccache/blob/master/src/mock_comm... I actually like how that code turned out, I've thought about cleaning it up and publishing it as a standalone crate. Rust doesn't have a real test mocking solution yet, and even if it did this is a special case since the standard library's process execution types don't implement a trait that could be mocked. As development went on I realized I didn't have automated tests that tested the whole program, especially running against real compilers (which is important given that the tool is a compiler wrapper), so I wrote some "system" tests that run the actual binary with an actual compiler and local disk cache and verify that it works as expected: https://github.com/mozilla/sccache/blob/master/src/test/system.rs https://github.com/mozilla/sccache/blob/master/src/test/syst...
bpicolo 10y ago
That's some tiny font.
- tedmielczarek 10y ago
  Sorry, I don't remember if I've ever actually configured the Wordpress settings on that blog.
- Matthias247 10y ago
  Yep, nearly to impossible to read without a big zoom. The culprit: font-size: 62.5%;
- nilved 10y ago
  It's using one of Phu Ly's WordPress themes from 10 years ago so that may be why. I'd recognize his themes anywhere. Back then 8pt was standard and flexible units were just catching on. I feel old.
- kibwen 10y ago
  A good time to try out your browser's Reader Mode. :P
Ericson2314 10y ago
Parts of Mozilla also use Nix so...
- gmfawcett 10y ago
  ...so what, in particular? Don't leave us hanging!
  - Ericson2314 10y ago
    It would be nice to use Nix in a more fine-grained way (e.g. per make rule rather than per package).
jtille 10y ago
I've been using distcc for quicker distributed builds without issue for many years. What extra features does sccache bring to the table other than it being rewritten in Rust?
- Sean1708 10y ago
  This comment[1] goes into some of the reasoning. [1]: https://www.reddit.com/r/rust/comments/5e654a/sccache_mozillas_distributed_compiler_cache_now/daa2uen/ https://www.reddit.com/r/rust/comments/5e654a/sccache_mozill...
  - jtille 10y ago
    Their reddit comments are not consistent with my experience using distcc. I can attest it is indeed a distributed ccache and works quite well. For a while I was even using distcc with a GCC cross compiler on Windows nodes to speed up my Linux app builds. If there's a sccache maintainer reading this thread please consider explaining the advantages of using sccache over distcc in the README.
    - Manishearth 10y ago
      > I can attest it is indeed a distributed ccache and works quite well. The distcc readme says nothing of the sort? It seems to specify that it shares header files and compiles cpp files (which are all presumably sent back to the source as object files, which links them together) It seems to work with ccache, but if you're spinning up fresh instances all the time ccache will need to be synchronized somehow, and it doesn't seem to drive that (whereas sccache seems to do so)
    - codyps 10y ago
      distcc is not a cache. It doesn't keep the output of the compiler around after it builds things. It (only) distributes running the compiler around (and to do that handles some details of preprocessing, in certain modes) and then gets the output back to the requester of the compilation task. At that point, the requester could cache that output, should they want to.
luckydude 10y ago
I didn't understand his comment on Rust's match expression. Other than it returning a value (and insisting a default clause) it's just like C's switch, right?
- fjh 10y ago
  match also lets you destructure values, which is useful when you're dealing with enums. It also doesn't have C's fall-through behaviour. Edit: match doesn't insist on a default clause, it enforces exhaustiveness. Which can be achieved by having a default clause, but quite often you just have an explicit branch for every possible case.
- Tarean 10y ago
  It is much more like haskell's pattern matching. Notably it handles tagged unions (in rust enum). Here is a good example from the [book](https://doc.rust-lang.org/book/match.html https://doc.rust-lang.org/book/match.html): enum Message { Quit, ChangeColor(i32, i32, i32), Move { x: i32, y: i32 }, Write(String), } fn quit() { /* ... */ } fn change_color(r: i32, g: i32, b: i32) { /* ... */ } fn move_cursor(x: i32, y: i32) { /* ... */ } fn process_message(msg: Message) { match msg { Message::Quit => quit(), Message::ChangeColor(r, g, b) => change_color(r, g, b), Message::Move { x: x, y: y } => move_cursor(x, y), Message::Write(s) => println!("{}", s), }; }
- kibwen 10y ago
  Differences from C's switch: 1. No fallthrough-by-default. 2. Checked for exhaustiveness. This is different from "insisting on a default clause", because if you see a match with no default clause, then you know that the compiler is verifying that every possible case is being handled. 3. The cases of the match can be arbitrary patterns, not just integers. This allows you to perform very natural and powerful conditional control flow, especially when using tagged unions. 4. Can return a value. Finally, remember the context of this post: Python doesn't have a C-style switch statement at all. :P
- deleted 10y ago
  [deleted]
zeotroph 10y ago
Running this without amazon seems to require replacing the hardcoded "{}://{}.s3{}.amazonaws.com" url with the location where the s3 compatible service is running on the local network. Now if that was just a default and could be overwritten with an env var as well... :) And minio [1] seems to be the easiest pseudo-s3 there is ($ ./minio server CacheDir/, done?), or are there better alternatives by now? 1: https://github.com/minio/minio https://github.com/minio/minio
- buckhx 10y ago
  Rust's format macro only takes string literals as a template, so you can't provide a runtime string template. The solution seems to be adding an external templating lib, which seems like overkill for a single template.
  - steveklabnik 10y ago
    https://crates.io/crates/strfmt https://crates.io/crates/strfmt works too.
    - chc 10y ago
      Wouldn't that be an external templating lib?
      - deleted 10y ago
        [deleted]
      - steveklabnik 10y ago
        In some sense, yes. In another, it's not some sort of different kind of templating, it's the same stuff, just calculated at runtime instead of compile time. I assume the lack of wanting a library for this is due to its weight and learning some other form of templating; this crate is pretty small, and isn't different in that way.
  - Groxx 10y ago
    Seems like it could be replaced with just string concatenation? Slightly less pretty, perhaps, but that wouldn't require any dependencies.
throwaway40483 10y ago
When I see these rewrites, I'm always left wondering about the speedup achieved. How much was due to: 1) Going from programming language X to Y 2) Rewriting the algorithm in language Y I suspect (without any proof) that too much is attributed to 1) and not enough to 2).
- sangnoir 10y ago
  3) Removing "unnecessary cruft" during the rewrite, measuring the speedup and then gradually adding the "cruft" (features) back one by one as previously unknown boundary conditions are encountered. Restart the process again after a few years of years, as is tradition.
  - quotemstr 10y ago
    This effect is the reason my default is to be skeptical of benchmarks of new software. It's easy to be fast if you're willing to be incorrect or incomplete.
  - tedmielczarek 10y ago
    This is not what happened in this case (although I have seen this pattern in other situations in the past). The Rust version is a faithful port of the feature set of the Python version. (It had to be, we were using essentially the entire feature set in production.)
- kibwen 10y ago
  I'm not sure what this comment is implying. The author isn't complaining about Python's performance, rather, it's noting that concurrency in Python isn't as painless as it is in Rust (which isn't a controversial statement, Rust is explicitly designed for robust concurrency). It also isn't controversial that Rust code should end up faster than Python, considering that Rust is designed to prioritize runtime performance. This isn't a case of comparing the performance of two dynamic languages (as we so often had with all the "I switched to Ruby" or "I switched to Node" posts in prior years); nobody is going to hold up this blog post as proof that Rust is generally faster than Python, because nobody in the world argues otherwise (and I say this as a prolific Python user, not just as a Rust user).
actuallyalys 10y ago
Congrats to Mozilla for not only creating a programming language research project that engaged the community, but also growing it into a successful language that's useful and robust for real-world projects. Mozilla sometimes gets flak for its experiments (or abandoning them), sometimes deservedly, but by doing so many of them and not being afraid to cancel them, they occasionally get big wins like Rust.
tedmielczarek 10y ago
Just as a neat data point, we hadn't been using sccache for Firefox builds in our new CI infrastructure (Taskcluster) for various reasons. After rolling out the sccache rewrite I fixed that, and build times in that environment dropped by 32-38%: https://treeherder.mozilla.org/perf.html#/alerts?id=4335 https://treeherder.mozilla.org/perf.html#/alerts?id=4335

243 points by AndrewDucker 10y ago

jononor 10y ago

No tests? :( EDIT: my bad, they were inside src/

cesarb 10y ago
In Rust, it's very common to put tests right next to the code; all functions tagged with #[test] will be run as tests, and examples within the function documentation comments will also be run as tests by default. For instance, randomly looking at the first file in that repository, cache/cache.rs, you can see at the end a submodule tagged with #[cfg(test)] (meaning it will be compiled only when running the tests), and within it several functions tagged with #[test].
- christophilus 10y ago
  I can't figure out whether or not I like this convention. I used to work with a guy who did the same thing with C++ and C# tests: right there alongside the code they tested. On the one hand, it is nice to have all of the relevant code in one place for context. On the other hand, it muddied up the actual business logic with tests, so you had to perform some mental work to figure out whether the code you were looking at was for production use or only for testing.
  - ryeguy 10y ago
    It's worth noting this convention is only for unit tests. Integration tests traditionally have their own test directory and files. Because of this, the file-inlined test code you see is normally relatively simple and shouldn't have a ton of extra helper functions and setup and such. The majority of the code should just be test cases themselves, which are clearly marked with #[test].
  - Yoric 10y ago
    Note that Rust lets you put tests in tests/ instead of src/. So typically, you put short unit tests in src/ and longer/integration tests in tests/. Oh, and examples that show up in the documentation are also run alongside tests. Also, the compiler will tell you if you have code that's not labelled with #[cfg(test)] but is designed to run only in tests.
    - tyoverby 10y ago
      The big difference with `tests/` is that they have to be integration tests. This means that you can't test private functions from there.
      - Manishearth 10y ago
        Not .. exactly. An integration test usually tests an entire application at once. `tests/` can still test things in the public API (indeed, we do this in Servo a lot) without being an integration test. The point is that some unit tests cannot be written in tests/, whereas all integration tests can be.
  - Sean1708 10y ago
    It's so that you can do unit testing without having to do weird things like airing your privates in tests, since private modules/functions/etc. aren't visible to your integration tests (which live in tests/ and basically just use your code as a library).
  - Tuna-Fish 10y ago
    I started liking this convention when I started considering the tests documentation. In my code, the first unit test for any function is always the simple common case, and having that right next to the function usually makes the code easier to read.
    - steveklabnik 10y ago
      You should move it to a documentation block, given that doc examples are also run as tests. Then your users can see them too. Win-win!
tedmielczarek 10y ago
It does have a bunch of tests, although probably not enough. I've been meaning to add code coverage stats to its CI, although I've only used kcov for Rust code coverage so far, and that only works on Linux it wouldn't reflect reality very well. I've done a lot of work on test automation at Mozilla over the years so automated testing is pretty important to me. :) I've tried to write unit tests where they made sense, they're generally placed within the relevant source file in the tree, like the cache tests here: https://github.com/mozilla/sccache/blob/6eadccc9d6752747766d550388590199d87f23a5/src/cache/cache.rs#L263 https://github.com/mozilla/sccache/blob/6eadccc9d6752747766d... I also wrote some higher-level tests that test the server functionality: https://github.com/mozilla/sccache/blob/master/src/test/tests.rs https://github.com/mozilla/sccache/blob/master/src/test/test... I rolled my own set of traits and structs to mock process execution for those tests so they would run independent of the compilers installed on the system: https://github.com/mozilla/sccache/blob/master/src/mock_command.rs https://github.com/mozilla/sccache/blob/master/src/mock_comm... I actually like how that code turned out, I've thought about cleaning it up and publishing it as a standalone crate. Rust doesn't have a real test mocking solution yet, and even if it did this is a special case since the standard library's process execution types don't implement a trait that could be mocked. As development went on I realized I didn't have automated tests that tested the whole program, especially running against real compilers (which is important given that the tool is a compiler wrapper), so I wrote some "system" tests that run the actual binary with an actual compiler and local disk cache and verify that it works as expected: https://github.com/mozilla/sccache/blob/master/src/test/system.rs https://github.com/mozilla/sccache/blob/master/src/test/syst...

bpicolo 10y ago

That's some tiny font.

tedmielczarek 10y ago
Sorry, I don't remember if I've ever actually configured the Wordpress settings on that blog.
Matthias247 10y ago
Yep, nearly to impossible to read without a big zoom. The culprit: font-size: 62.5%;
nilved 10y ago
It's using one of Phu Ly's WordPress themes from 10 years ago so that may be why. I'd recognize his themes anywhere. Back then 8pt was standard and flexible units were just catching on. I feel old.
kibwen 10y ago
A good time to try out your browser's Reader Mode. :P

Ericson2314 10y ago

Parts of Mozilla also use Nix so...

gmfawcett 10y ago
...so what, in particular? Don't leave us hanging!
- Ericson2314 10y ago
  It would be nice to use Nix in a more fine-grained way (e.g. per make rule rather than per package).

jtille 10y ago

I've been using distcc for quicker distributed builds without issue for many years. What extra features does sccache bring to the table other than it being rewritten in Rust?

Sean1708 10y ago
This comment[1] goes into some of the reasoning. [1]: https://www.reddit.com/r/rust/comments/5e654a/sccache_mozillas_distributed_compiler_cache_now/daa2uen/ https://www.reddit.com/r/rust/comments/5e654a/sccache_mozill...
- jtille 10y ago
  Their reddit comments are not consistent with my experience using distcc. I can attest it is indeed a distributed ccache and works quite well. For a while I was even using distcc with a GCC cross compiler on Windows nodes to speed up my Linux app builds. If there's a sccache maintainer reading this thread please consider explaining the advantages of using sccache over distcc in the README.
  - Manishearth 10y ago
    > I can attest it is indeed a distributed ccache and works quite well. The distcc readme says nothing of the sort? It seems to specify that it shares header files and compiles cpp files (which are all presumably sent back to the source as object files, which links them together) It seems to work with ccache, but if you're spinning up fresh instances all the time ccache will need to be synchronized somehow, and it doesn't seem to drive that (whereas sccache seems to do so)
  - codyps 10y ago
    distcc is not a cache. It doesn't keep the output of the compiler around after it builds things. It (only) distributes running the compiler around (and to do that handles some details of preprocessing, in certain modes) and then gets the output back to the requester of the compilation task. At that point, the requester could cache that output, should they want to.

luckydude 10y ago

I didn't understand his comment on Rust's match expression. Other than it returning a value (and insisting a default clause) it's just like C's switch, right?

fjh 10y ago
match also lets you destructure values, which is useful when you're dealing with enums. It also doesn't have C's fall-through behaviour. Edit: match doesn't insist on a default clause, it enforces exhaustiveness. Which can be achieved by having a default clause, but quite often you just have an explicit branch for every possible case.
Tarean 10y ago
It is much more like haskell's pattern matching. Notably it handles tagged unions (in rust enum). Here is a good example from the [book](https://doc.rust-lang.org/book/match.html https://doc.rust-lang.org/book/match.html): enum Message { Quit, ChangeColor(i32, i32, i32), Move { x: i32, y: i32 }, Write(String), } fn quit() { /* ... */ } fn change_color(r: i32, g: i32, b: i32) { /* ... */ } fn move_cursor(x: i32, y: i32) { /* ... */ } fn process_message(msg: Message) { match msg { Message::Quit => quit(), Message::ChangeColor(r, g, b) => change_color(r, g, b), Message::Move { x: x, y: y } => move_cursor(x, y), Message::Write(s) => println!("{}", s), }; }
kibwen 10y ago
Differences from C's switch: 1. No fallthrough-by-default. 2. Checked for exhaustiveness. This is different from "insisting on a default clause", because if you see a match with no default clause, then you know that the compiler is verifying that every possible case is being handled. 3. The cases of the match can be arbitrary patterns, not just integers. This allows you to perform very natural and powerful conditional control flow, especially when using tagged unions. 4. Can return a value. Finally, remember the context of this post: Python doesn't have a C-style switch statement at all. :P
deleted 10y ago
[deleted]

zeotroph 10y ago

Running this without amazon seems to require replacing the hardcoded "{}://{}.s3{}.amazonaws.com" url with the location where the s3 compatible service is running on the local network. Now if that was just a default and could be overwritten with an env var as well... :) And minio [1] seems to be the easiest pseudo-s3 there is ($ ./minio server CacheDir/, done?), or are there better alternatives by now? 1: https://github.com/minio/minio https://github.com/minio/minio

buckhx 10y ago
Rust's format macro only takes string literals as a template, so you can't provide a runtime string template. The solution seems to be adding an external templating lib, which seems like overkill for a single template.
- steveklabnik 10y ago
  https://crates.io/crates/strfmt https://crates.io/crates/strfmt works too.
  - chc 10y ago
    Wouldn't that be an external templating lib?
    - deleted 10y ago
      [deleted]
    - steveklabnik 10y ago
      In some sense, yes. In another, it's not some sort of different kind of templating, it's the same stuff, just calculated at runtime instead of compile time. I assume the lack of wanting a library for this is due to its weight and learning some other form of templating; this crate is pretty small, and isn't different in that way.
- Groxx 10y ago
  Seems like it could be replaced with just string concatenation? Slightly less pretty, perhaps, but that wouldn't require any dependencies.

throwaway40483 10y ago

When I see these rewrites, I'm always left wondering about the speedup achieved. How much was due to: 1) Going from programming language X to Y 2) Rewriting the algorithm in language Y I suspect (without any proof) that too much is attributed to 1) and not enough to 2).

sangnoir 10y ago
3) Removing "unnecessary cruft" during the rewrite, measuring the speedup and then gradually adding the "cruft" (features) back one by one as previously unknown boundary conditions are encountered. Restart the process again after a few years of years, as is tradition.
- quotemstr 10y ago
  This effect is the reason my default is to be skeptical of benchmarks of new software. It's easy to be fast if you're willing to be incorrect or incomplete.
- tedmielczarek 10y ago
  This is not what happened in this case (although I have seen this pattern in other situations in the past). The Rust version is a faithful port of the feature set of the Python version. (It had to be, we were using essentially the entire feature set in production.)
kibwen 10y ago
I'm not sure what this comment is implying. The author isn't complaining about Python's performance, rather, it's noting that concurrency in Python isn't as painless as it is in Rust (which isn't a controversial statement, Rust is explicitly designed for robust concurrency). It also isn't controversial that Rust code should end up faster than Python, considering that Rust is designed to prioritize runtime performance. This isn't a case of comparing the performance of two dynamic languages (as we so often had with all the "I switched to Ruby" or "I switched to Node" posts in prior years); nobody is going to hold up this blog post as proof that Rust is generally faster than Python, because nobody in the world argues otherwise (and I say this as a prolific Python user, not just as a Rust user).

actuallyalys 10y ago

Congrats to Mozilla for not only creating a programming language research project that engaged the community, but also growing it into a successful language that's useful and robust for real-world projects. Mozilla sometimes gets flak for its experiments (or abandoning them), sometimes deservedly, but by doing so many of them and not being afraid to cancel them, they occasionally get big wins like Rust.

tedmielczarek 10y ago

Just as a neat data point, we hadn't been using sccache for Firefox builds in our new CI infrastructure (Taskcluster) for various reasons. After rolling out the sccache rewrite I fixed that, and build times in that environment dropped by 32-38%: https://treeherder.mozilla.org/perf.html#/alerts?id=4335 https://treeherder.mozilla.org/perf.html#/alerts?id=4335