5 ms·

I invested a lot of time porting some parsing code I had written to Rust, with the vision that Rust is the memory-safe future. The code I was porting from used

I invested a lot of time porting some parsing code I had written to Rust, with the vision that Rust is the memory-safe future. The code I was porting from used arenas, so I tried to use arenas in Rust also. Using arenas required a bunch of lifetime annotations everywhere, but I was happy to do it if I could get provable memory safety. I got everything working, but the moment I tried to wrap it in Python, it failed. The lifetime annotation on my struct was a problem. I tried to work around this by using ouroboros and the self-referencing struct pattern. But then I ran into another problem: the Rust arena I was using (Bumpalo) is not Sync, which means references to the arena are not Send. All of my arena-aware containers were storing references to the Arena and therefore were not Send, but wrapping in Python requires it to be Send. I wrote more about these challenges here: https://blog.reverberate.org/2021/12/19/arenas-and-rust.html https://blog.reverberate.org/2021/12/19/arenas-and-rust.html You might say "well don't use an arena, use Box, Rc, etc." But now you're telling me to write more complicated and less efficient code, just to make it work with Rust. That is a hard pill to swallow for what is supposed to be a systems language.

himujjal 4y ago
I did the Crafting Interpreters book to learn Rust after I did The Rust book. I faced the same problem as you. Somehow it felt like Arenas went beyond Rust's philosophy and added huge amounts of complexity to the interpreter. Tree traversal and mutating the environment of a `block` internally was an issue I spent like 2-3 days on. I was porting Java code to Rust after all. Somehow got it working in a Rust way. I used unsafe at one place. But I was left heavily unsatisfied. Something about graphs/trees and Rust don't match up.
himujjal 4y ago
I understand where you are coming from. Rust and Graphs/Trees is a hard problem. Somehow it goes beyond the way we think about Graphs.
kaba0 4y ago
I’ve written a JVM in Rust where I was pretty much unable to work without unsafe, and while I know the general consensus is that it is a sin to use, I managed to get away with a few usages wrapped tightly inside a safe API. Sure, I had memory problems during writing, but with MIRI and some debug assertions they were not hard to hunt down and I really only had to get that few lines of code inside unsafe blocks right. What I’m trying to say with all that, do not be afraid to use unsafes in Rust. It is part of the language for a reason. Sure, do use ARC for some non-performance critical whatever, because it frankly doesn’t really matter. But where it matters, and you decided to use a low-level language, then go for unsafe if that’s the only reasonable way. The result will still be much safer than the other low-level languages. I believe the problem here is the same what C++ tried to achieve: making people believe it is a high level language. That is just dishonest, and really should not be the goal of Rust.
avgcorrection 4y ago
I respect the effort. I won’t argue against such hard-earned experience.
staticassertion 4y ago
You can tell pyo3 that your type isn't Send and then it'll panic if the object is accessed from multiple threads. Given that that's the only safe option, that seems fine? You say that that's not acceptable for a production library but I don't see the issue. You have the same restrictions in C++ except with worse consequences.
- haberman 4y ago
  > You have the same restrictions in C++ except with worse consequences. In C++ it is safe because the arena is only used from one thread at a time. To model this C++ pattern in Rust, what I would really want is: 1. Arena should be Sync, and not use interior mutability. 2. Arena::alloc() should do a dual borrow: (a) a mut borrow of the Arena metadata, only for the duration of the alloc() call, and (b) a non-mut (shared) borrow of the Arena data. Because this kind of split borrow cannot be expressed in Rust AFAICS, (2) is not possible, so (1) is not feasible. This forces Bumpalo to be !Sync, which makes a direct Rust port of the C++ pattern impossible. I've heard this called the "factory problem" for Rust: you cannot easily make a factory type in Rust that returns references, because if the create() operation mutates the factory, then the returned reference will have a mutable borrow on the factory. The alternative would be to make a truly thread-safe arena/factory, which could be Sync with interior mutability, but that is an efficiency compromise due to synchronization overhead.

himujjal 4y ago

I did the Crafting Interpreters book to learn Rust after I did The Rust book. I faced the same problem as you. Somehow it felt like Arenas went beyond Rust's philosophy and added huge amounts of complexity to the interpreter. Tree traversal and mutating the environment of a `block` internally was an issue I spent like 2-3 days on. I was porting Java code to Rust after all. Somehow got it working in a Rust way. I used unsafe at one place. But I was left heavily unsatisfied. Something about graphs/trees and Rust don't match up.

himujjal 4y ago

I understand where you are coming from. Rust and Graphs/Trees is a hard problem. Somehow it goes beyond the way we think about Graphs.

kaba0 4y ago

I’ve written a JVM in Rust where I was pretty much unable to work without unsafe, and while I know the general consensus is that it is a sin to use, I managed to get away with a few usages wrapped tightly inside a safe API. Sure, I had memory problems during writing, but with MIRI and some debug assertions they were not hard to hunt down and I really only had to get that few lines of code inside unsafe blocks right. What I’m trying to say with all that, do not be afraid to use unsafes in Rust. It is part of the language for a reason. Sure, do use ARC for some non-performance critical whatever, because it frankly doesn’t really matter. But where it matters, and you decided to use a low-level language, then go for unsafe if that’s the only reasonable way. The result will still be much safer than the other low-level languages. I believe the problem here is the same what C++ tried to achieve: making people believe it is a high level language. That is just dishonest, and really should not be the goal of Rust.

avgcorrection 4y ago

I respect the effort. I won’t argue against such hard-earned experience.

staticassertion 4y ago

You can tell pyo3 that your type isn't Send and then it'll panic if the object is accessed from multiple threads. Given that that's the only safe option, that seems fine? You say that that's not acceptable for a production library but I don't see the issue. You have the same restrictions in C++ except with worse consequences.

haberman 4y ago
> You have the same restrictions in C++ except with worse consequences. In C++ it is safe because the arena is only used from one thread at a time. To model this C++ pattern in Rust, what I would really want is: 1. Arena should be Sync, and not use interior mutability. 2. Arena::alloc() should do a dual borrow: (a) a mut borrow of the Arena metadata, only for the duration of the alloc() call, and (b) a non-mut (shared) borrow of the Arena data. Because this kind of split borrow cannot be expressed in Rust AFAICS, (2) is not possible, so (1) is not feasible. This forces Bumpalo to be !Sync, which makes a direct Rust port of the C++ pattern impossible. I've heard this called the "factory problem" for Rust: you cannot easily make a factory type in Rust that returns references, because if the create() operation mutates the factory, then the returned reference will have a mutable borrow on the factory. The alternative would be to make a truly thread-safe arena/factory, which could be Sync with interior mutability, but that is an efficiency compromise due to synchronization overhead.