21 ms·

Rust in Large Organizations

248 points by lalaithion 7y ago

awestroke 7y ago
Super interesting read. The geometric growth of deps problem seems like a natural consequence of having a good package manager. In languages like C/C++, libraries often include all their code - there are no transitive dependencies.
- vlovich123 7y ago
  My reading of that isn't that the problem is the large set of transitive deps but that it's opaque to the native build system at these companies (Buck/blaze) which means rebuilding one requires rebuilding all which is time-consuming. I infer this from the discussion about being able to query inputs/outputs of the build so that integration can be done more meaningfully into these systems to rebuild just changes.
kccqzy 7y ago
The build system issue isn't inherent to Rust or to large organizations. Even in a startup, we have a polyglot development with a separate build tool in addition to the language-specific build tool. We ended up doing much of the same things: use the language-specific build tool to generate artifacts which are then consumed by the normal build tool.
- monocasa 7y ago
  A slight difference is that Rust really wants to be the top level compiler. More than a lot of languages, even C++, it really relies on link time optimization to get the perf you'd expect.
  - kccqzy 7y ago
    In that case just print the command line cargo would've used to do the linking and let a different consume that and make changes. That's an easy case.
  - hope-striker 7y ago
    How this is more true of Rust than modern C++?
    - SamReidHughes 7y ago
      Most "modern" C++ involves template metaprogramming of a sort that is in header file, so the modern aspect doesn't stand to increase the need for LTO that much. I would imagine that Rust programs benefit from LTO by removing static assertions necessary for memory safety that C++ programs simply omit, and by unraveling functional style code that Rust programs often get shoehorned into. Or maybe not, but that's one non-LTO that's needed.
- bsder 7y ago
  > The build system issue isn't inherent to Rust or to large organizations. That may be true, but even the big boys can't converge, and they all have their own build systems. Attempting to add/remove features from cargo/rust when even the big companies can't agree as to what is most important in a build system seems like a path to disaster. The big boys aren't always right (see: Gradle/Groovy).
dtolnay 7y ago
Here is some context for the event at which these notes were taken, and a summary writeup: https://users.rust-lang.org/t/rust-in-large-organizations-meeting/32059?u=dtolnay https://users.rust-lang.org/t/rust-in-large-organizations-me... This was a meeting between a few employees of large organizations using Rust (Facebook, Google, Microsoft, Mozilla) at RustConf two weeks ago to discuss the common needs of enterprise users and how to collaborate effectively on the language and ecosystem going forward.
ausjke 7y ago
I tried to learn those languages, my IQ level can handle java/golang/python/c just fine, but c++ and rust is beyond my comfort zone for sure, for c++, getting familiar with it is OK, but getting good at it is hard, really hard, mastering it is impossible for me. similar can be said to rust as far as I can tell. even python has some "tricks", try "x=256; y=256; x is y" and "x=257, y=257, x is y" under python, or 'x=[], y=[], x is y; x=y=[], x is y' you will know what I mean, you actually need know some internal designs to use python properly. At the moment I feel golang has the right trade off for programmers, be it beginners or senior developers.
- kbp 7y ago
  I don't think that's a great example for Python; that's just knowing what "is" is for. The fact that it works the same as == for small integers in CPython is an optimisation showing through, but only in a place where it doesn't really matter. You still need to have a grasp on the difference between reference equality and value equality without getting into anything anyone would call tricks or implementation details (eg, after `x = []; y = []; x.append(1)`, how many elements does y have?).
  - ausjke 7y ago
    x=x.append(1) and guess what, now x is "None", yes it makes perfect sense for those who coded python for a while, but it's a disaster for newcomers. too many similar demos by the way I actually like python and use it often, just saying it can surprise you when you start, not intuitive per se until you become good at it.
    - gridspy 7y ago
      Sure, x.append doesn't return a value. So x=x.append is None. You learn that one pretty quick. Perhaps use x = x + [1] instead? But yeah, surprises like this (unexpected return values) are actually a pretty strong case for using a language like Rust or C++ where the compiler will tell you this before you run the program.
epx 7y ago
I'd really like to engage in a job or project that uses Rust.
- gaogao 7y ago
  Try asking in the r/rust weekly status update thread, in the Rust discord, or going to a local meeting! (Here is pretty off-topic)
est31 7y ago
It's interesting how many organizations had problems with build.rs. Other build systems like Make, scons, or autotools have the ability to run arbitrary things in any step in the process compared to Rust which has a clear separation. There is a proposed RFC to reduce the need for build.rs scripts in many cases [1], will likely be merged within the next 2 weeks. [1]: https://github.com/rust-lang/rfcs/pull/2523 https://github.com/rust-lang/rfcs/pull/2523
- epage 7y ago
  For code-gen, I'm experimenting with checking in the results and having a "is the code-gen correct" check in the CI See https://github.com/crate-ci/imperative/pull/4 https://github.com/crate-ci/imperative/pull/4 for an example I hadn't even considered the benefit of reducing build.rs usage. I was focused on speeding up people's builds by reducing the number of dependencies involved and making my crate do less work when being built. Tempted to turn this pattern into a crate to reduce the hump to overcome to adopt it.
- saghm 7y ago
  > It's interesting how many organizations had problems with build.rs. Other build systems like Make, scons, or autotools have the ability to run arbitrary things in any step in the process compared to Rust which has a clear separation. I think the history behind this is that Cargo was built with only Rust in mind; when your world is 100% Rust, you generally don't need much extensibility in your build process. Make, scons, and autotools all seem like they were designed to be more language agnostic from the start. It'll be interesting to see how cargo needs to evolve to be able to support interior with other ecosystems at the level that larger companies need.
  - ChrisSD 7y ago
    The way I heard it was that Cargo was intentionally avoiding being a complete build system. The idea being that Make et al would still be used when necessary so build.rs (and cargo in general) could be free to concentrate on managing the Rust parts exclusively.
  - deleted 7y ago
lacker 7y ago
I was entertained by this exchange: ms: would like to know how to control use of unsafe in codebase google: grep
- layoutIfNeeded 7y ago
  They obviously meant ripgrep
- swsieber 7y ago
  You can totally hook into the linting system to index all usages of unsafe, even (and especially) in dependencies.
- mijoharas 7y ago
  As I got to that point I started laughing and then explained it to a non-programmer that was with me. My fumbled metaphor of "Find in page across a lot of pages" didn't really land...
  - heavenlyblue 7y ago
    So what do you do if “unsafe” is in a string or a comment? How do you automate the linting of it? I believe that’s what MS is speaking about. Google probably simply puts this on the shoulders of the developer.
mmastrac 7y ago
Glad to see some thinking going on in this area. We're using Rust at FullStory and it's about 1/3 source-wise for one of our projects. A few of these concerns echo conversations that teammates have had on Slack. FullStory itself is somewhat polyglot (Go on the server, TypeScript/react in the browser, Java/ObjJ/Rust on native mobile) and we've felt some of the pain around having a consistent and reliable build environment. We've also had internal discussions along these lines - ie: which deps can we trust from various package managers (npm has interesting arbitrary code execution as well!). Building tooling to manage all these languages in a common system is a _huge_ commitment. We're actively hiring for the role of productivity engineers which is something that I see becoming more important in orgs as they scale up, the same way that devops/SRE split out in the 2000s.
jeffrallen 7y ago
Oh good lord. Reading that made me happy that (1) I am no longer in a large organisation, and (2) I program in Go. Good luck guys, you'll need it. You've got interesting problems.
vardump 7y ago
Some sort of "how-to" for Rust in large organizations would be very nice.
nunchuckninja 7y ago
Can any rust experts tell me that unsafe rust is powerful and expressive enough to let me do anything? I have a use case where I have two iterators on a linked list in nested loops and the second iterators deletes some elements as it goes. Is this something I can express in unsafe rust?
- unrealhoang 7y ago
  I’m no expert but yes, you can do anything in unsafe as in C can do anything. Unsafe Rust provides you access to raw pointers.
  - nunchuckninja 7y ago
    I'll add more details to see if it helps- C: ``` node prime= list.head; while () { node probe = prime.next; while() { if () //condition on prime.data and probe.data { probe.next = probe.next.next; //& drop stuff } } ```
    - pitaj 7y ago
      You need to use four space indent to create code blocks, triple-ticks don't work here.
amrx431 7y ago
I have to learn Rust for my new job starting next month. Any recommendations?
- mlindner 7y ago
  How did you get hired into a place that uses Rust when you don't know any? Also where do you work because I want that.

248 points by lalaithion 7y ago

Super interesting read. The geometric growth of deps problem seems like a natural consequence of having a good package manager. In languages like C/C++, libraries often include all their code - there are no transitive dependencies.

vlovich123 7y ago
My reading of that isn't that the problem is the large set of transitive deps but that it's opaque to the native build system at these companies (Buck/blaze) which means rebuilding one requires rebuilding all which is time-consuming. I infer this from the discussion about being able to query inputs/outputs of the build so that integration can be done more meaningfully into these systems to rebuild just changes.

kccqzy 7y ago

The build system issue isn't inherent to Rust or to large organizations. Even in a startup, we have a polyglot development with a separate build tool in addition to the language-specific build tool. We ended up doing much of the same things: use the language-specific build tool to generate artifacts which are then consumed by the normal build tool.

monocasa 7y ago
A slight difference is that Rust really wants to be the top level compiler. More than a lot of languages, even C++, it really relies on link time optimization to get the perf you'd expect.
- kccqzy 7y ago
  In that case just print the command line cargo would've used to do the linking and let a different consume that and make changes. That's an easy case.
- hope-striker 7y ago
  How this is more true of Rust than modern C++?
  - SamReidHughes 7y ago
    Most "modern" C++ involves template metaprogramming of a sort that is in header file, so the modern aspect doesn't stand to increase the need for LTO that much. I would imagine that Rust programs benefit from LTO by removing static assertions necessary for memory safety that C++ programs simply omit, and by unraveling functional style code that Rust programs often get shoehorned into. Or maybe not, but that's one non-LTO that's needed.
bsder 7y ago
> The build system issue isn't inherent to Rust or to large organizations. That may be true, but even the big boys can't converge, and they all have their own build systems. Attempting to add/remove features from cargo/rust when even the big companies can't agree as to what is most important in a build system seems like a path to disaster. The big boys aren't always right (see: Gradle/Groovy).

dtolnay 7y ago

Here is some context for the event at which these notes were taken, and a summary writeup: https://users.rust-lang.org/t/rust-in-large-organizations-meeting/32059?u=dtolnay https://users.rust-lang.org/t/rust-in-large-organizations-me... This was a meeting between a few employees of large organizations using Rust (Facebook, Google, Microsoft, Mozilla) at RustConf two weeks ago to discuss the common needs of enterprise users and how to collaborate effectively on the language and ecosystem going forward.

ausjke 7y ago

I tried to learn those languages, my IQ level can handle java/golang/python/c just fine, but c++ and rust is beyond my comfort zone for sure, for c++, getting familiar with it is OK, but getting good at it is hard, really hard, mastering it is impossible for me. similar can be said to rust as far as I can tell. even python has some "tricks", try "x=256; y=256; x is y" and "x=257, y=257, x is y" under python, or 'x=[], y=[], x is y; x=y=[], x is y' you will know what I mean, you actually need know some internal designs to use python properly. At the moment I feel golang has the right trade off for programmers, be it beginners or senior developers.

kbp 7y ago
I don't think that's a great example for Python; that's just knowing what "is" is for. The fact that it works the same as == for small integers in CPython is an optimisation showing through, but only in a place where it doesn't really matter. You still need to have a grasp on the difference between reference equality and value equality without getting into anything anyone would call tricks or implementation details (eg, after `x = []; y = []; x.append(1)`, how many elements does y have?).
- ausjke 7y ago
  x=x.append(1) and guess what, now x is "None", yes it makes perfect sense for those who coded python for a while, but it's a disaster for newcomers. too many similar demos by the way I actually like python and use it often, just saying it can surprise you when you start, not intuitive per se until you become good at it.
  - gridspy 7y ago
    Sure, x.append doesn't return a value. So x=x.append is None. You learn that one pretty quick. Perhaps use x = x + [1] instead? But yeah, surprises like this (unexpected return values) are actually a pretty strong case for using a language like Rust or C++ where the compiler will tell you this before you run the program.

epx 7y ago

I'd really like to engage in a job or project that uses Rust.

gaogao 7y ago
Try asking in the r/rust weekly status update thread, in the Rust discord, or going to a local meeting! (Here is pretty off-topic)

est31 7y ago

It's interesting how many organizations had problems with build.rs. Other build systems like Make, scons, or autotools have the ability to run arbitrary things in any step in the process compared to Rust which has a clear separation. There is a proposed RFC to reduce the need for build.rs scripts in many cases [1], will likely be merged within the next 2 weeks. [1]: https://github.com/rust-lang/rfcs/pull/2523 https://github.com/rust-lang/rfcs/pull/2523

epage 7y ago
For code-gen, I'm experimenting with checking in the results and having a "is the code-gen correct" check in the CI See https://github.com/crate-ci/imperative/pull/4 https://github.com/crate-ci/imperative/pull/4 for an example I hadn't even considered the benefit of reducing build.rs usage. I was focused on speeding up people's builds by reducing the number of dependencies involved and making my crate do less work when being built. Tempted to turn this pattern into a crate to reduce the hump to overcome to adopt it.
saghm 7y ago
> It's interesting how many organizations had problems with build.rs. Other build systems like Make, scons, or autotools have the ability to run arbitrary things in any step in the process compared to Rust which has a clear separation. I think the history behind this is that Cargo was built with only Rust in mind; when your world is 100% Rust, you generally don't need much extensibility in your build process. Make, scons, and autotools all seem like they were designed to be more language agnostic from the start. It'll be interesting to see how cargo needs to evolve to be able to support interior with other ecosystems at the level that larger companies need.
- ChrisSD 7y ago
  The way I heard it was that Cargo was intentionally avoiding being a complete build system. The idea being that Make et al would still be used when necessary so build.rs (and cargo in general) could be free to concentrate on managing the Rust parts exclusively.
- deleted 7y ago

lacker 7y ago

I was entertained by this exchange: ms: would like to know how to control use of unsafe in codebase google: grep

layoutIfNeeded 7y ago
They obviously meant ripgrep
swsieber 7y ago
You can totally hook into the linting system to index all usages of unsafe, even (and especially) in dependencies.
mijoharas 7y ago
As I got to that point I started laughing and then explained it to a non-programmer that was with me. My fumbled metaphor of "Find in page across a lot of pages" didn't really land...
- heavenlyblue 7y ago
  So what do you do if “unsafe” is in a string or a comment? How do you automate the linting of it? I believe that’s what MS is speaking about. Google probably simply puts this on the shoulders of the developer.

mmastrac 7y ago

Glad to see some thinking going on in this area. We're using Rust at FullStory and it's about 1/3 source-wise for one of our projects. A few of these concerns echo conversations that teammates have had on Slack. FullStory itself is somewhat polyglot (Go on the server, TypeScript/react in the browser, Java/ObjJ/Rust on native mobile) and we've felt some of the pain around having a consistent and reliable build environment. We've also had internal discussions along these lines - ie: which deps can we trust from various package managers (npm has interesting arbitrary code execution as well!). Building tooling to manage all these languages in a common system is a _huge_ commitment. We're actively hiring for the role of productivity engineers which is something that I see becoming more important in orgs as they scale up, the same way that devops/SRE split out in the 2000s.

jeffrallen 7y ago

Oh good lord. Reading that made me happy that (1) I am no longer in a large organisation, and (2) I program in Go. Good luck guys, you'll need it. You've got interesting problems.

vardump 7y ago

Some sort of "how-to" for Rust in large organizations would be very nice.

nunchuckninja 7y ago

Can any rust experts tell me that unsafe rust is powerful and expressive enough to let me do anything? I have a use case where I have two iterators on a linked list in nested loops and the second iterators deletes some elements as it goes. Is this something I can express in unsafe rust?

unrealhoang 7y ago
I’m no expert but yes, you can do anything in unsafe as in C can do anything. Unsafe Rust provides you access to raw pointers.
- nunchuckninja 7y ago
  I'll add more details to see if it helps- C: ``` node prime= list.head; while () { node probe = prime.next; while() { if () //condition on prime.data and probe.data { probe.next = probe.next.next; //& drop stuff } } ```
  - pitaj 7y ago
    You need to use four space indent to create code blocks, triple-ticks don't work here.

amrx431 7y ago

I have to learn Rust for my new job starting next month. Any recommendations?

mlindner 7y ago
How did you get hired into a place that uses Rust when you don't know any? Also where do you work because I want that.