4 ms·

In real world software, you need lots of freedom in order to design high performance scalable systems. This includes using non-trivial data structures, and bein

by afr0ck 4y ago

In real world software, you need lots of freedom in order to design high performance scalable systems. This includes using non-trivial data structures, and being able to access and organise memory in subtle ways. The Linux kernel is an example of this. Look at the reclaim subsystem in the kernel, for instance, and how it interacts in subtle and very complex ways with various other subsystems, the page cache, the generic block layer, the memory control groups, the VFS and the filesystems. All of them are multi-threaded and run on complex high performance CPUs performing memory re-ordering, speculative execution, branch perdiction and parallel execution of both processes and instructions. You really need extremely smart smart data structures and organisation and very smart algorithms to manage the concurrency. C is a language where you have all that freedom to do whatever you want leading to high performance and low latency. In Rust, you can't even safely implement a doubly linked list. Now, if you have to use unsafe every time you need high performance or scalable data structures, then Rust is as unsafe as C and it's just making your life harder for no reason, except hype.

deleted 4y ago
[deleted]
lalaithion 4y ago
The key is that Rust lets you define safe interfaces, so you can separate your data races and union juggling, which need to be unsafe, from your business logic, which basically never needs to be unsafe.
- nyanpasu64 4y ago
  "Object graph" architectures are common in C++ and sometimes necessary in Rust for business logic, when building GUI applications or emulators. But Rust doesn't allow mutating through a &/Rc, and throws a compile error if you create multiple &mut, and the workarounds are unreasonably boilerplate-heavy and RefCell carries runtime overhead, whereas C++ doesn't get in the way of making your code work. I've put together a playground at https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=3b5066c3f091b34b981d7d0ca8748c4d https://play.rust-lang.org/?version=stable&mode=debug&editio.... It's not just dereferencing invalid */& that's illegal in Rust, but constructing and dereferencing valid &/&mut in ways that don't respect tree-shaped mutability. Rust's pointer aliasing rules invalidate otherwise-correct code, placing roadblocks in the way of writing correct code. There's so much creation of &mut (which invalidates aliasing pointers for the duration of the &mut, and invalidates sibling &mut and all pointers constructed from them), that's so implicit I don't know what's legal and what's not by auditing code. (Box<T> used to also invalidate aliasing pointers, but this may be changed. The current plan for enabling self-reference is Pin<&mut T>, but the exact semantics for how and when putting a &mut T in a wrapper struct makes it not invalidate self-reference and incoming pointers, is still not specified.) (I've elaborated further at https://news.ycombinator.com/item?id=33658253 https://news.ycombinator.com/item?id=33658253.)
  - zozbot234 4y ago
    You can use UnsafeCell https://doc.rust-lang.org/std/cell/struct.UnsafeCell.html https://doc.rust-lang.org/std/cell/struct.UnsafeCell.html if you commit to manually enforcing RefCell's borrowing rules.
alkonaut 4y ago
> You really need extremely smart smart data structures and organisation and very smart algorithms to manage the concurrency. C is a language where you have all that freedom to do whatever you want leading to high performance and low latency. But it leaves that work to the developer, thousands of hours of testing and reviewing to ensure no little corner case is missed. I think the argument that unsafe is used making the language have “holes” is somewhat misdirected. When I see a rust implementation of a doubly linked list with unsafe I know exactly where I’m on my own (a few lines) and where the compiler does the job for me (the rest of it). It’s not as if that means safety or flexibility is out the window. It means “do the manual safety review on these two lines”.
jolux 4y ago
> Now, if you have to use unsafe every time you need high performance or scalable data structures, then Rust is as unsafe as C That's not true, because you can wrap unsafe code in safe interfaces and use it from safe code.
vore 4y ago
As someone who has done kernel development work (though not Linux), not every part of a kernel is as filled with dragons as you make it out to be. Why would my e.g. filesystem driver need to punch a lot of deep unsafe holes down to the internals of memory management? Why can't I make use of a safe abstraction implemented using unsafe code to free myself from caring about those details? With C, you can only do these kinds of safety guarantees with a lot of discipline. With Rust, you can offload much of your discipline to the compiler.
- afr0ck 4y ago
  Sometimes, you can't make hard walls between subsystems. Some filesystems need very close interaction with memory management to have very good performance. Some objects are accessed and manipulated in a racy way from logically independent places for scalability reasons. I am not against Rust, modularity, etc. But building very high performance systems is just something different.
  - bombela 4y ago
    You would encode that into types and expose a safe API. The usafe would still be local in scope. For example you might build an abstraction named ReadWriteMutex that guarantees safe access at runtime. I don't see a scenario where this would impede performances. The invariants to validate are the same. But rust would guarantee that it is memory safe.
  - vore 4y ago
    But do you need this in the 99% of cases? Probably not. You can write productive, performant, and safe code in safe Rust fine. If you do need to do something this exotic, the fundamentals don't change: you still need to take care of memory lifetimes, you still need to ensure thread safety, etc. I really don't think it's as all-or-nothing as you believe it to be.
zozbot234 4y ago
The problem with "subtle and very complex" interactions is that they don't really scale to larger systems. They're inherently anti-modular, since they depend on the state of the program as a whole. This is exactly what's avoided in Safe Rust.
- afr0ck 4y ago
  If you want blazing fast and very scalable performance, you have to do that. Multiple independent contexts in the kernels can have a reference on certain objects and manipulate them concurrently with relaxed consistency. It would be very impractical to hard-protect certain parts of those objects and guarantee serial consistency. For scalability and performance reasons, certain data structures, objects or certain parts of certain objects could be accessed in a inherently racy manner, but guaranteeing correctness in the existence of races is extremely hard. Any attempt for modularisation, message passing or serial consistency would make your kernel look like a joke compared to Linux. It would make the interaction between subsystems easy but would not provide the opportunity for very fast low latency execution.
josephg 4y ago
> Rust is as unsafe as C and it's just making your life harder for no reason, except hype. I hear what you’re saying, but that hasn’t been my experience with rust. I love C, but I’ve been writing rust for the last couple of years. And rust has stolen my heart. You’re right that unsafe rust is a bit less ergonomic than just writing C. Sometimes I miss void pointers. I definitely miss how fast C compiles. But most rust isn’t unsafe rust. Even deep in my custom in-memory btree implementation I think more than half of my methods are safe. And safe rust is a fabulous language when you can use it. There’s all these bugs you just can’t write. Early on with rust I had this magical experience. My program segfaulted in one of my tests because of memory corruption. It was “spooky action at a distance” where the segfault happened well after the buggy code was executed. Bugs like this are awful to track down in C because the bug could be literally anywhere. But when I looked at the trace, there was only one unsafe function which could be causing the problem. (Since I could rule out all my safe code). Sure enough, 10 minutes later I had a fix. I have a skip list for large strings that I ported from C to rust. I still have no idea why, but the rust code ran about 20% faster out of the box than the original optimized C code. And yet, the code is significantly smaller and easier to work with. I’ve added a few more optimizations since then - it’s about 10x faster than the C code now. The new optimizations are all from tricks I never got around to adding in C because I was afraid I might break something. And then there’s the things rust does well that aren’t in C at all. Cargo is incredible. Monomorphization is the right tool for a lot of problems, like custom collections. (Higher performance and types? Yes please!) Parametric enums & match statements are so much better to work with than C enums and unions. Then there's rust's std, which is fantastic. I use Option, Result, Vec, BtreeMap, PriorityQueue, the OS-agnostic filesystem API, and so on daily. So yeah, I hear you about C being lovely. But I think rust is even better. Rust is a bit of an ordeal to learn but I'm really glad I learned it. Its a joy to use.

by afr0ck 4y ago

deleted 4y ago

[deleted]

lalaithion 4y ago

The key is that Rust lets you define safe interfaces, so you can separate your data races and union juggling, which need to be unsafe, from your business logic, which basically never needs to be unsafe.

nyanpasu64 4y ago
"Object graph" architectures are common in C++ and sometimes necessary in Rust for business logic, when building GUI applications or emulators. But Rust doesn't allow mutating through a &/Rc, and throws a compile error if you create multiple &mut, and the workarounds are unreasonably boilerplate-heavy and RefCell carries runtime overhead, whereas C++ doesn't get in the way of making your code work. I've put together a playground at https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=3b5066c3f091b34b981d7d0ca8748c4d https://play.rust-lang.org/?version=stable&mode=debug&editio.... It's not just dereferencing invalid */& that's illegal in Rust, but constructing and dereferencing valid &/&mut in ways that don't respect tree-shaped mutability. Rust's pointer aliasing rules invalidate otherwise-correct code, placing roadblocks in the way of writing correct code. There's so much creation of &mut (which invalidates aliasing pointers for the duration of the &mut, and invalidates sibling &mut and all pointers constructed from them), that's so implicit I don't know what's legal and what's not by auditing code. (Box<T> used to also invalidate aliasing pointers, but this may be changed. The current plan for enabling self-reference is Pin<&mut T>, but the exact semantics for how and when putting a &mut T in a wrapper struct makes it not invalidate self-reference and incoming pointers, is still not specified.) (I've elaborated further at https://news.ycombinator.com/item?id=33658253 https://news.ycombinator.com/item?id=33658253.)
- zozbot234 4y ago
  You can use UnsafeCell https://doc.rust-lang.org/std/cell/struct.UnsafeCell.html https://doc.rust-lang.org/std/cell/struct.UnsafeCell.html if you commit to manually enforcing RefCell's borrowing rules.

alkonaut 4y ago

> You really need extremely smart smart data structures and organisation and very smart algorithms to manage the concurrency. C is a language where you have all that freedom to do whatever you want leading to high performance and low latency. But it leaves that work to the developer, thousands of hours of testing and reviewing to ensure no little corner case is missed. I think the argument that unsafe is used making the language have “holes” is somewhat misdirected. When I see a rust implementation of a doubly linked list with unsafe I know exactly where I’m on my own (a few lines) and where the compiler does the job for me (the rest of it). It’s not as if that means safety or flexibility is out the window. It means “do the manual safety review on these two lines”.

jolux 4y ago

> Now, if you have to use unsafe every time you need high performance or scalable data structures, then Rust is as unsafe as C That's not true, because you can wrap unsafe code in safe interfaces and use it from safe code.

vore 4y ago

As someone who has done kernel development work (though not Linux), not every part of a kernel is as filled with dragons as you make it out to be. Why would my e.g. filesystem driver need to punch a lot of deep unsafe holes down to the internals of memory management? Why can't I make use of a safe abstraction implemented using unsafe code to free myself from caring about those details? With C, you can only do these kinds of safety guarantees with a lot of discipline. With Rust, you can offload much of your discipline to the compiler.

afr0ck 4y ago
Sometimes, you can't make hard walls between subsystems. Some filesystems need very close interaction with memory management to have very good performance. Some objects are accessed and manipulated in a racy way from logically independent places for scalability reasons. I am not against Rust, modularity, etc. But building very high performance systems is just something different.
- bombela 4y ago
  You would encode that into types and expose a safe API. The usafe would still be local in scope. For example you might build an abstraction named ReadWriteMutex that guarantees safe access at runtime. I don't see a scenario where this would impede performances. The invariants to validate are the same. But rust would guarantee that it is memory safe.
- vore 4y ago
  But do you need this in the 99% of cases? Probably not. You can write productive, performant, and safe code in safe Rust fine. If you do need to do something this exotic, the fundamentals don't change: you still need to take care of memory lifetimes, you still need to ensure thread safety, etc. I really don't think it's as all-or-nothing as you believe it to be.

zozbot234 4y ago

The problem with "subtle and very complex" interactions is that they don't really scale to larger systems. They're inherently anti-modular, since they depend on the state of the program as a whole. This is exactly what's avoided in Safe Rust.

afr0ck 4y ago
If you want blazing fast and very scalable performance, you have to do that. Multiple independent contexts in the kernels can have a reference on certain objects and manipulate them concurrently with relaxed consistency. It would be very impractical to hard-protect certain parts of those objects and guarantee serial consistency. For scalability and performance reasons, certain data structures, objects or certain parts of certain objects could be accessed in a inherently racy manner, but guaranteeing correctness in the existence of races is extremely hard. Any attempt for modularisation, message passing or serial consistency would make your kernel look like a joke compared to Linux. It would make the interaction between subsystems easy but would not provide the opportunity for very fast low latency execution.

josephg 4y ago

> Rust is as unsafe as C and it's just making your life harder for no reason, except hype. I hear what you’re saying, but that hasn’t been my experience with rust. I love C, but I’ve been writing rust for the last couple of years. And rust has stolen my heart. You’re right that unsafe rust is a bit less ergonomic than just writing C. Sometimes I miss void pointers. I definitely miss how fast C compiles. But most rust isn’t unsafe rust. Even deep in my custom in-memory btree implementation I think more than half of my methods are safe. And safe rust is a fabulous language when you can use it. There’s all these bugs you just can’t write. Early on with rust I had this magical experience. My program segfaulted in one of my tests because of memory corruption. It was “spooky action at a distance” where the segfault happened well after the buggy code was executed. Bugs like this are awful to track down in C because the bug could be literally anywhere. But when I looked at the trace, there was only one unsafe function which could be causing the problem. (Since I could rule out all my safe code). Sure enough, 10 minutes later I had a fix. I have a skip list for large strings that I ported from C to rust. I still have no idea why, but the rust code ran about 20% faster out of the box than the original optimized C code. And yet, the code is significantly smaller and easier to work with. I’ve added a few more optimizations since then - it’s about 10x faster than the C code now. The new optimizations are all from tricks I never got around to adding in C because I was afraid I might break something. And then there’s the things rust does well that aren’t in C at all. Cargo is incredible. Monomorphization is the right tool for a lot of problems, like custom collections. (Higher performance and types? Yes please!) Parametric enums & match statements are so much better to work with than C enums and unions. Then there's rust's std, which is fantastic. I use Option, Result, Vec, BtreeMap, PriorityQueue, the OS-agnostic filesystem API, and so on daily. So yeah, I hear you about C being lovely. But I think rust is even better. Rust is a bit of an ordeal to learn but I'm really glad I learned it. Its a joy to use.