6 ms·
> When crashes do occur an engineer needs to spend time to diagnose how it happened and what caused it. Since Pingora's inception we’ve served a few hundred tri
by nicoburns 4y ago
> When crashes do occur an engineer needs to spend time to diagnose how it happened and what caused it. Since Pingora's inception we’ve served a few hundred trillion requests and have yet to crash due to our service code.
> In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible.
That's quite the endorsement of Rust. A lot of people focus on the fact that Rust can't absolutely guarantee freedom from crashes and memory safety issues. Which I think misses the point that this kind of experience of running high traffic Rust services in production for months almost without a single issue is common in practice.
- brink 4y agoI had the same experience when I wrote a camera capture / motion detection / video logging service for a commercial smart refrigerator in Rust. The Swift component crashed at least twice a week, the Rust component ran for months and months without issue.
- jhgg 4y agoWe had the same experience at work deploying rust services that serve many billions of requests a day as well.
- drogus 4y agoI had a very similar experience. Much smaller scale, but the service was keeping internal state and clients were connecting with a WebSocket. It could handle up to a million clients on one server and it practically never crashed. While I was writing it I had only hobby-level experience with Rust and I was also mentoring a colleague, so he wrote a big chunk of code as a total Rust noob.
- victor106 4y agoIs this using Async Rust?
- brink 4y agoYes, if you want to handle that many clients, you'll need to use async. It's not too bad to learn.
- Aperocky 4y agoCurses! now I need to learn yet another language!
- WesolyKubeczek 4y agoI have a theory that people who are noobs in a certain technology, but are overall experienced, tend to produce very solid code, because they actively avoid doing clever things.
- 4m1rk 4y agoNice one. I believe we should avoid doing unnecessary clever things deliberately and actively.
- rkagerer 4y agoWhich aspect(s) of Rust do you think are most responsible for this? (e.g. borrow checker, memory safety, culture that attracts devs who care about reliability, etc)
- gpm 4y agoNot the person you're asking, but the culture around data representation strikes me as the biggest factor: 1. Only making valid states possible, to the greatest extent reasonably possible. 2. Treating error as regular data, not an afterthought - with language features to make that not too painful. 3. Not returning placeholder values (i.e. if you get back a parsed value from a parse function, it means it parsed correctly, not that either there's an error somewhere else or it parsed correctly). Language features, in particular "enums" (aka algebraic data types, aka tagged unions), make this approach possible. You couldn't do it in go, for instance, even if there was a cultural decision to.
- pornel 4y agoIt is of course a combination of all these aspects. A type system that can express thread safety (Send/Sync traits) is incredibly valuable when building multi-threaded systems. Universal definition of what is safe, and standard traits and borrowing rules, make APIs more predictable. Just from function's signature you know a lot about its behavior, without having to look for gotchas in the manual. Mandatory error handling prevents cutting corners. Unit testing is built-in. Generics, good inlining, and Cargo help split code into libraries without a performance or usability hit, which helps make focused, well-tested components. Most of these things aren't groundbreaking, but Rust being new had a luxury of picking current best practices and sensible defaults.
- nicoburns 4y agoA few things: - I think memory safety is a baseline. You'll note that memory safe languages already tend to be much more reliable than non-memory-safe languages in general. - Then you have the error handling. A lot of unreliability in my code in other languages comes from unhandled exceptions that only occur rarely. Rust generally puts all possible error conditions in the type signature of the function. Meaning it's actually feasible to handle every failure case. - Speaking of unhandled exceptions, a lot of those in typed languages tend to be caused by null. Rust does not have null. Instead it has Option, and it is impossible to access the contents of an option without doing the equivalent of a null check. So that entire class of errors is gone. - Both Result (used for error handling) and Option (used instead of null) are what Rust calls enums, and what are more generally called Sum Types. I think these are a huge deal. They allow you to safely represent data that may be one thing or another with very strict type checking. These are broadly very useful in API design, and in my experience lead to much more robust code than the class hierarchies you need in OOP languages or unions which lack the safety checks. (Aside: sum types would be quite simple to add to other languages. I have no idea why they haven't been added yet). - Speaking of classes, inheritance is not supported. So that's a bunch of confusing code that just isn't possible to write. This can add a bit of boilerplate to Rust code, but it makes it more straightforward and less bug prone. - You mentioned the borrow checker. That definitely helps. It's yet another tool that allows you to write APIs that cannot be misused. A great example would be Rust's Mutex type. It can prove at compile time that code does not hold on to references to the protected data beyond the duration that the lock is held. - Speaking of Mutex. Rust's Send and Sync traits provide very good thread safety. You almost don't need to worry about thread safety at all in Rust. Most concurrency bugs are prevented by the compiler (you can still do things like cause data races). - Newtypes allow you to check invariants once and then have the fact that they remain satisfied enforced by the type system. - All type casts are explicit. - Lots of other little things One final thing that I think is often overlooked. Rust is strict, and all of these checks apply not only to the code your write, but to all of your dependencies. That means that Rust libraries tend to be much more reliable than libraries from other ecosystems. That probably is partly because of a culture of reliability. But it's also because the language itself makes it hard to write sloppy code. And that the code you are building on is likely to be reliable makes it both less effort and more worthwhile to make your own code reliable (including for library authors), leading to virtuous circle of reliable code.
- dcow 4y agoCame here to post this quote too. This is my experience (albeit at smaller scale) exactly. If this wasn't the case, Rust would be snake oil and I'd be the first to admit it. Until my most recent endeavor with a rust backend, among other things, I simply didn't know it was possible to not have to debug crashes. Seriously. The pleasure of maintaining rust software in production is so wildly different from anything I've ever experienced in the past that I gladly submit to the compiler over and over and over again. It's worth the investment 100 fold.
- ellieh 4y agoSimilar experience over here too, I've been running a Rust server in production for >1yr now and had absolutely no crashes. Resource usage has been nice and constant too. It's fantastic :)