8 ms·

You can’t “turn off the borrow checker” in Rust

152 points by ngaut 8y ago

kbwt 8y ago
I thought this was going to be a response to Jonathan Blow's video about how doing your own memory management is effectively turning off the borrow checker: https://www.youtube.com/watch?v=4t1K66dMhWk https://www.youtube.com/watch?v=4t1K66dMhWk The takeaway being that the borrow checker doesn't magically prevent the use-after-free class of bugs. Although you will never experience a segmentation fault in safe Rust, the bug is still there and your program keeps running in an invalid state. The symptoms are changed, but no less dangerous. To make the problem even more obvious, think of allocating a large array to be used as a heap and handing out indices to implement your own malloc. You have bounds checking to prevent indexing outside the bounds of the heap, but it doesn't really help when the elements have logically different lifetimes and occupy different parts of the array. I don't think this is a contrived example either. A less obvious version of this can easily creep into large or complex systems, as evidenced by the Entity Component System in Rust example.
- rectang 8y ago
  What I'd like to understand better is what idioms are emerging as best practices thanks to the (good!) pressure that Rust puts on us. For memory management, some of the major algorithms available are global allocation, stack allocation, malloc/free, reference counting, and tracing garbage collection. (The Entity Component System model is doing your own allocation so it's a subtype of either malloc/free or ref counting.) Stack allocation and global allocation with borrow checking seem very attractive in terms of safety, but you have to know more in advance about how the memory is going to be used because growing the allocation while you are deeper in the stack is not possible. Are there any idioms or algorithms which are particularly friendly to this model? For instance, traversing a preliminary object graph to establish max allocation requirements before allocating a large block on the stack?
  - steveklabnik 8y ago
    > What I'd like to understand better is what idioms are emerging as best practices thanks to the (good!) pressure that Rust puts on us. We all would. They're still emerging! I think the generational index idea is one of them, though.
- sdegutis 8y ago
  So it’s the difference between crash early (C) and don’t crash but run wrong (Rust), inherent by design in the main selling point of Rust (borrow checker)?
  - Tuna-Fish 8y ago
    No. Most modern C++ ECS take the same approach as the Rust ECS that is being discussed. Also, since I started using generational indexes, I have never had a bug caused by using stale data, so at least for me it hasn't been a real issue.
  - andrewflnr 8y ago
    Broadly speaking, C is the king of running wrong without crashing, so no. The borrow checker isn't really making anything worse, it's just failing to save you from this particular problem. In C you could still write the exact same bug, or you could even write it with actual pointers. I gather a lot of browser exploits start that way these days.
  - gliptic 8y ago
    Why is everyone assuming C is "crash early" as opposed to "undefined behaviour will crash if you're lucky and cause RCE in the worst case, or any weird thing in between". The selling point of the borrow checker (at least one of them) is that it doesn't allow undefined behaviour unless you explicitly enable unsafe operations. If there was a way to detect UB and predictably crash in a performant way, you could probably implement that in Rust as well. In fact, that's often what is done with generational indexes and similar.
    - shawn 8y ago
      https://doc.rust-lang.org/nomicon/print.html https://doc.rust-lang.org/nomicon/print.html Rust is otherwise quite permissive with respect to other dubious operations. Rust considers it "safe" to: Deadlock Have a race condition Leak memory Fail to call destructors Overflow integers Abort the program Delete the production database However any program that actually manages to do such a thing is probably incorrect. Rust provides lots of tools to make these things rare, but these problems are considered impractical to categorically prevent.
- zrm 8y ago
  > The symptoms are changed, but no less dangerous. Potentially even more dangerous. Having the program segfault immediately is much preferable to, say, Heartbleed.
  - barrkel 8y ago
    If only we could guarantee immediate segfaults.
    - kibwen 8y ago
      That's literally what Rust compiler errors are doing. :P
    - zrm 8y ago
      > If only we could guarantee immediate segfaults. It should be possible to create a malloc implementation that does that by making the minimum allocation size a page and then not reusing virtual addresses for new allocations. Then once an allocation is freed, any access to it is permanently a segfault. That may not be practical on existing architectures with 48-bit virtual addressing though, since you could plausibly exhaust the address space. The full 64 bits might be sufficient for most things at least. You could also get most of the benefit by not reusing virtual addresses until you run out.
      - steveklabnik 8y ago
        It's not, because that relies on actually doing the de-reference. Thanks to UB, that may never actually happen, the code may get removed entirely.
        zrm 8y ago
        If the code is removed entirely then what memory is being improperly accessed?
        steveklabnik 8y ago
- sidlls 8y ago
  I think the primary argument for Rust advocates here is that what you describe is only possible by using explicitly "unsafe" code (i.e. code using the "unsafe" block that is required). The weakness of this argument is exposed in your comment also: in libraries "unsafe" is often hidden behind an abstraction. To take a simple example, consider the standard Rust Vec: plenty of "unsafe" code resides in this object, but you'll rarely if ever see "unsafe" wrap typical vector operations ("unsafe { myvec[0] }"). Partial mitigation is that this does reduce the surface area where one might have to look if odd behaviors start to appear in a Rust program. Overall it's still a net benefit.
  - steveklabnik 8y ago
    > ("unsafe { myvec[0] }") Part of the point in my post is that this does not remove the bounds check. This unsafe block does nothing. This actually might be a better example than the one I picked...
    - kibwen 8y ago
      And just as in the OP, the compiler makes it clear that this unsafe block does nothing: warning: unnecessary `unsafe` block --> src/main.rs:4:3 | 4 | unsafe { myvec[0]; } | ^^^^^^ unnecessary `unsafe` block |
  - saghm 8y ago
    I might be in the minority here (and I don't do game development, which is the topic of this thread), but I tend to start by using `vec.get(i)` (which returns an option) and then only switch to the index notation if I end up checking the length immediately beforehand and want to do an early return or something to avoid extra indentation in the code where I use the accessed element.
    - majewsky 8y ago
      I think modern C++ compilers are smart enough to eliminate the bounds check if they can prove that it never fails.
- steveklabnik 8y ago
  So, I had been thinking about this post for a while, and Blow's video caused some more discussion that made me post it. But it's not a direct response, I still haven't watched the video, and so I don't know what he actually said. If I wanted it to be a response, I would have linked to it. > The symptoms are changed, but no less dangerous. I would take issue with this sentiment. There's a world of difference between "logic error and/or panic" and "undefined behavior". Yes, Rust doesn't fix all bugs. But it's still an improvement here.
  - kbwt 8y ago
    Thanks for the response. I wasn't trying to speculate on your intentions for publishing the blog post. > There's a world of difference between "logic error and/or panic" and "undefined behavior". Is is really so different for the programmer who wrote the bug? If you have undefined behavior, the language implementation can do whatever it wants. It won't actively work against you, but the implementer is given permission to ignore what would happen if you violate their assumptions. With a logic error in custom memory management, the program execution will still be following well-defined rules but the invariants assumed by the programmer will no longer hold. The resulting behavior appears effectively undefined to the programmer, because the point of invariants is to ignore what would happen when they are broken. Defensive coding with panics/asserts will definitely help catch some of these mistakes during development. > Yes, Rust doesn't fix all bugs. But it's still an improvement here. I applaud your efforts with Rust, it's great to see someone actually trying to improve the state of programming languages.
    - steveklabnik 8y ago
      It’s all good, it’s a totally reasonable thing, which was also brought up in all the other threads :) > it won’t actively work against you I guess it depends on what you mean by “active.” Consider the Option<NonNull<T>> case. We can do the null check in safe Rust. We know the check is done. Now consider the case with UB: https://blogs.msdn.microsoft.com/oldnewthing/20140627-00/?p=633 https://blogs.msdn.microsoft.com/oldnewthing/20140627-00/?p=... These kinds of things can cause lots of subtle issues. The rust code won’t.
    - 8y ago
- pcwalton 8y ago
  > The symptoms are changed, but no less dangerous. The symptoms of a use-after-free-style logic error are less dangerous in Rust, because it's much harder to get RCE.
mlevental 8y ago
not that steve isn't correct but Rc and Arc effectively (in exchange for runtime overhead) "turn off" the borrow checker. i'm sure i'll get yelled but it's just this week i had the borrow checker yelling at me for something and i realized that the appropriate thing to do was use Arc (yes of course i'm not advocating for ref counting instead of being more precise).
- masklinn 8y ago
  Rc and Arc are about (shared) ownership not borrowing. If they're turning off anything, it's ownership. Borrowing works the same way as ever. In fact that's an advantage of Rust here: because of the borrow checker you can safely get a reference to an Rc's contents and hand it off to something without having to alter the refcount, so you get significantly less refcount traffic than in other languages where such a thing is unsafe (or not under your control). That's especially important for Arc. It also provides for nice safe optimisations like "move out of this Rc if I'm the only owner" (Rc/Arc::try_unwrap).
- danieldk 8y ago
  I think you are confusing Rc/Arc with RefCell, since they are often used together. In order to provide interior mutability, the borrow_mut method of RefCell takes &self rather than &mut self. So, it is possible to obtain two mutable references. The following compiles: let c = RefCell::new(42usize); let mut mut_ref1 = c.borrow_mut(); let mut mut_ref2 = c.borrow_mut(); *mut_ref1 += 1; *mut_ref2 += 2; RefCell uses UnsafeCell to subvert the borrows checker. However, instead RefCell implements run-time borrow checks. So, the code above will panic while attempting to do the second mutable borrow. It is a trade-off. You lose catching errors during compilation, but you can have interior mutability. This is one of the goals of unsafe, which UnsafeCell requires. Sometimes, you need to do something that safe Rust does not permit, but that you can prove to be safe (such as RefCell). That said (and it depends a bit on your domain), I have used Rust for probably 1.5 years without using Arc, Rc, and RefCell. The first time I needed these types when was writing a Gtk+ application and later in some multi-threading programs. These three types are very easy to overuse when you are coming from other languages. IMO it is better to avoid them and try to work with the borrows checker until you have sufficient experience with Rust to identify when interior mutability and reference counting are really necessary.
kibwen 8y ago
I think there's mental mismatch between groups of people who talk about "turning off the borrow checker". The borrow checker is a tool to validate references. Sometimes people using references in Rust might feel like the borrow checker makes using references too cumbersome in a certain situation, so they switch to using a tool other than references (Rc, indexes into a Vec, etc). But this isn't bypassing the borrow checker; it's bypassing references themselves. The same phenomenon happens in C++; if references start to be a pain, you might switch to using something else (shared_ptr, indexes into a vector, etc.). When this happens in C++, we don't call this "bypassing the borrow checker". You don't need a borrow checker to know that references aren't always the right tool for a given job. It's the same in Rust.
- steveklabnik 8y ago
  Yup. Part of why I wrote this post is for exactly this reason. This phrase is used colloquially, but I think it misleads a lot of people on how Rust actually works.
orf 8y ago
> This means that we can combine it with Option<T>, and the option will use the null case for None: This sounds interesting, can anyone elaborate on this?
- tux3 8y ago
  It's a really cool optimization, and it doesn't just apply to Option. Here's a StackOverflow thread: https://stackoverflow.com/questions/46557608/what-is-the-null-pointer-optimization-in-rust https://stackoverflow.com/questions/46557608/what-is-the-nul...
- kibwen 8y ago
  One of the design goals of Option is to be a library-level replacement for the language-level null value found in languages like C. Furthermore, one of the design goals for Rust itself is to have its abstractions be zero-overhead. With a naive implementation of Option, these goals would be in opposition. To illustrate why these goals would be in opposition, look at a trivial example of a tagged union (enum) in Rust: enum Foo { Bar(i32), Qux(u32) } At runtime, any value of type Foo will either be in the Bar state, or it will be in the Qux state. Both Bar and Qux hold types that are 32 bits in size, and since only one of those states can be active at a time, we know that Foo only needs 32 bits of storage to satisfy both these states. But additionally, it needs extra storage to store the runtime information telling us which state it's currently in. The smallest "extra" amount of storage that can be added to a type is 8 bits, so we would expect every value of type Foo to be 40 bits in size at runtime. In fact it's larger, due to alignment and padding, so Foo will be 64 bits in size at runtime. Let's bring it back around to Option, which is an enum that looks like this: enum Option<T> { Some(T), None } A null pointer in C will be the same size as a non-null pointer in C: 64 bits, assuming a 64-bit platform. A Rust reference would also be 64 bits, and these cannot be null. If we were to try to use Option on a reference to "opt-in" to nullability, what size would that "nullable reference" be assuming a naive implementation of Option? Well, firstly we'd need storage for the value of the reference itself (64 bits), and then, as per above, we'd need our "extra" storage to tell us at runtime whether our Option is a Some or a None. And again, because of alignment and padding, this would theoretically result in a type that is 128 bits in size in total, which is real shame since in theory distinguishing between two states only takes a single bit of storage. Overall this would be a performance regression from C, where nullability does not impose any space overhead. Fortunately, Rust's implementation is not naive. Remember: Rust references cannot be null. That means that the Rust compiler knows that any type that is a reference will not contain a value that is all zeroes at runtime. Rust leverages this knowledge for optimization: for any Option containing a reference, only a single pointer-sized piece of memory is needed, and the None case will be represented by a value of all zeroes. This means that the Option is now a zero-overhead abstraction for this use case, because Option<&Foo> will be the same size as &Foo. And this smart logic isn't hardcoded for the Option enum. Any enum, written by anyone, can automatically benefit from the ability to "hide" the enum tag in such "uninhabited" values. The OP's example of NonNull<T> is, like references, an example of of a type that has an uninhabited value that permits this optimization. Others include the NonZeroU8 type and its friends, where Option<NonZeroU8> will be the same size as a standard u8, though which give up the ability to represent zero (in the future this may be extended to allow arbitrary user-defined types which can make whatever values they want act as uninhabited for the purposes of enum size optimization, but it will take some work to get there).
shawn 8y ago
Note that you can drop down to unsafe C-style code in Rust. https://doc.rust-lang.org/stable/nomicon/ https://doc.rust-lang.org/stable/nomicon/ Anyone who claims Rust is simple should ensure they thoroughly understand that book. Another way to "turn off" the borrow checker is to write a scripting language that compiles to Rust which automatically annotates all variables with the longest lifetime possible, and spits out mutable references depending on whether you actually mutate anything. There's also RefCell, which lets you defer borrow checking till runtime. It's handy for pretending like your references are immutable.
- ChrisSD 8y ago
  Also note that none of those things turn off the borrow checker. Which is the point of the article.
- kibwen 8y ago
  I don't think that anyone's claiming that Rust is objectively a simple language. Simpler than some other languages, certainly, but it's a medium-sized language at best. Furthermore, unsafe code gives new users a firm boundary of complexity that can be ignored. New to Rust? Don't use the unsafe keyword. Using the unsafe keyword? Read the nomicon first. It's quite useful for onboarding to know that all the C-style UB shenanigans are behind a gate that can be ignored until you're comfortable with the rest of the language.
jononor 8y ago
My TLDR/alternate title: "unsafe Rust retains most safety benefits of Rust (including the borrow checker)"
twarge 8y ago
As a non-coder (physicist) writing Rust, the thing that really stuck me was that the time between successful compile and flawless operation was significantly shorter than the C and Python I write. Furthermore, this difference when writing anything threaded is simply breathtaking. Im my experience there are simply fewer corner cases that the Rust compiler lets through.
- blueprint 8y ago
  I think that makes you a coder :)

152 points by ngaut 8y ago

kbwt 8y ago

I thought this was going to be a response to Jonathan Blow's video about how doing your own memory management is effectively turning off the borrow checker: https://www.youtube.com/watch?v=4t1K66dMhWk https://www.youtube.com/watch?v=4t1K66dMhWk The takeaway being that the borrow checker doesn't magically prevent the use-after-free class of bugs. Although you will never experience a segmentation fault in safe Rust, the bug is still there and your program keeps running in an invalid state. The symptoms are changed, but no less dangerous. To make the problem even more obvious, think of allocating a large array to be used as a heap and handing out indices to implement your own malloc. You have bounds checking to prevent indexing outside the bounds of the heap, but it doesn't really help when the elements have logically different lifetimes and occupy different parts of the array. I don't think this is a contrived example either. A less obvious version of this can easily creep into large or complex systems, as evidenced by the Entity Component System in Rust example.

rectang 8y ago
What I'd like to understand better is what idioms are emerging as best practices thanks to the (good!) pressure that Rust puts on us. For memory management, some of the major algorithms available are global allocation, stack allocation, malloc/free, reference counting, and tracing garbage collection. (The Entity Component System model is doing your own allocation so it's a subtype of either malloc/free or ref counting.) Stack allocation and global allocation with borrow checking seem very attractive in terms of safety, but you have to know more in advance about how the memory is going to be used because growing the allocation while you are deeper in the stack is not possible. Are there any idioms or algorithms which are particularly friendly to this model? For instance, traversing a preliminary object graph to establish max allocation requirements before allocating a large block on the stack?
- steveklabnik 8y ago
  > What I'd like to understand better is what idioms are emerging as best practices thanks to the (good!) pressure that Rust puts on us. We all would. They're still emerging! I think the generational index idea is one of them, though.
sdegutis 8y ago
So it’s the difference between crash early (C) and don’t crash but run wrong (Rust), inherent by design in the main selling point of Rust (borrow checker)?
- Tuna-Fish 8y ago
  No. Most modern C++ ECS take the same approach as the Rust ECS that is being discussed. Also, since I started using generational indexes, I have never had a bug caused by using stale data, so at least for me it hasn't been a real issue.
- andrewflnr 8y ago
  Broadly speaking, C is the king of running wrong without crashing, so no. The borrow checker isn't really making anything worse, it's just failing to save you from this particular problem. In C you could still write the exact same bug, or you could even write it with actual pointers. I gather a lot of browser exploits start that way these days.
- gliptic 8y ago
  Why is everyone assuming C is "crash early" as opposed to "undefined behaviour will crash if you're lucky and cause RCE in the worst case, or any weird thing in between". The selling point of the borrow checker (at least one of them) is that it doesn't allow undefined behaviour unless you explicitly enable unsafe operations. If there was a way to detect UB and predictably crash in a performant way, you could probably implement that in Rust as well. In fact, that's often what is done with generational indexes and similar.
  - shawn 8y ago
    https://doc.rust-lang.org/nomicon/print.html https://doc.rust-lang.org/nomicon/print.html Rust is otherwise quite permissive with respect to other dubious operations. Rust considers it "safe" to: Deadlock Have a race condition Leak memory Fail to call destructors Overflow integers Abort the program Delete the production database However any program that actually manages to do such a thing is probably incorrect. Rust provides lots of tools to make these things rare, but these problems are considered impractical to categorically prevent.
zrm 8y ago
> The symptoms are changed, but no less dangerous. Potentially even more dangerous. Having the program segfault immediately is much preferable to, say, Heartbleed.
- barrkel 8y ago
  If only we could guarantee immediate segfaults.
  - kibwen 8y ago
    That's literally what Rust compiler errors are doing. :P
  - zrm 8y ago
    > If only we could guarantee immediate segfaults. It should be possible to create a malloc implementation that does that by making the minimum allocation size a page and then not reusing virtual addresses for new allocations. Then once an allocation is freed, any access to it is permanently a segfault. That may not be practical on existing architectures with 48-bit virtual addressing though, since you could plausibly exhaust the address space. The full 64 bits might be sufficient for most things at least. You could also get most of the benefit by not reusing virtual addresses until you run out.
    - steveklabnik 8y ago
      It's not, because that relies on actually doing the de-reference. Thanks to UB, that may never actually happen, the code may get removed entirely.
      - zrm 8y ago
        If the code is removed entirely then what memory is being improperly accessed?
        steveklabnik 8y ago
sidlls 8y ago
I think the primary argument for Rust advocates here is that what you describe is only possible by using explicitly "unsafe" code (i.e. code using the "unsafe" block that is required). The weakness of this argument is exposed in your comment also: in libraries "unsafe" is often hidden behind an abstraction. To take a simple example, consider the standard Rust Vec: plenty of "unsafe" code resides in this object, but you'll rarely if ever see "unsafe" wrap typical vector operations ("unsafe { myvec[0] }"). Partial mitigation is that this does reduce the surface area where one might have to look if odd behaviors start to appear in a Rust program. Overall it's still a net benefit.
- steveklabnik 8y ago
  > ("unsafe { myvec[0] }") Part of the point in my post is that this does not remove the bounds check. This unsafe block does nothing. This actually might be a better example than the one I picked...
  - kibwen 8y ago
    And just as in the OP, the compiler makes it clear that this unsafe block does nothing: warning: unnecessary `unsafe` block --> src/main.rs:4:3 | 4 | unsafe { myvec[0]; } | ^^^^^^ unnecessary `unsafe` block |
- saghm 8y ago
  I might be in the minority here (and I don't do game development, which is the topic of this thread), but I tend to start by using `vec.get(i)` (which returns an option) and then only switch to the index notation if I end up checking the length immediately beforehand and want to do an early return or something to avoid extra indentation in the code where I use the accessed element.
  - majewsky 8y ago
    I think modern C++ compilers are smart enough to eliminate the bounds check if they can prove that it never fails.
steveklabnik 8y ago
So, I had been thinking about this post for a while, and Blow's video caused some more discussion that made me post it. But it's not a direct response, I still haven't watched the video, and so I don't know what he actually said. If I wanted it to be a response, I would have linked to it. > The symptoms are changed, but no less dangerous. I would take issue with this sentiment. There's a world of difference between "logic error and/or panic" and "undefined behavior". Yes, Rust doesn't fix all bugs. But it's still an improvement here.
- kbwt 8y ago
  Thanks for the response. I wasn't trying to speculate on your intentions for publishing the blog post. > There's a world of difference between "logic error and/or panic" and "undefined behavior". Is is really so different for the programmer who wrote the bug? If you have undefined behavior, the language implementation can do whatever it wants. It won't actively work against you, but the implementer is given permission to ignore what would happen if you violate their assumptions. With a logic error in custom memory management, the program execution will still be following well-defined rules but the invariants assumed by the programmer will no longer hold. The resulting behavior appears effectively undefined to the programmer, because the point of invariants is to ignore what would happen when they are broken. Defensive coding with panics/asserts will definitely help catch some of these mistakes during development. > Yes, Rust doesn't fix all bugs. But it's still an improvement here. I applaud your efforts with Rust, it's great to see someone actually trying to improve the state of programming languages.
  - steveklabnik 8y ago
    It’s all good, it’s a totally reasonable thing, which was also brought up in all the other threads :) > it won’t actively work against you I guess it depends on what you mean by “active.” Consider the Option<NonNull<T>> case. We can do the null check in safe Rust. We know the check is done. Now consider the case with UB: https://blogs.msdn.microsoft.com/oldnewthing/20140627-00/?p=633 https://blogs.msdn.microsoft.com/oldnewthing/20140627-00/?p=... These kinds of things can cause lots of subtle issues. The rust code won’t.
  - 8y ago
pcwalton 8y ago
> The symptoms are changed, but no less dangerous. The symptoms of a use-after-free-style logic error are less dangerous in Rust, because it's much harder to get RCE.

mlevental 8y ago

not that steve isn't correct but Rc and Arc effectively (in exchange for runtime overhead) "turn off" the borrow checker. i'm sure i'll get yelled but it's just this week i had the borrow checker yelling at me for something and i realized that the appropriate thing to do was use Arc (yes of course i'm not advocating for ref counting instead of being more precise).

masklinn 8y ago
Rc and Arc are about (shared) ownership not borrowing. If they're turning off anything, it's ownership. Borrowing works the same way as ever. In fact that's an advantage of Rust here: because of the borrow checker you can safely get a reference to an Rc's contents and hand it off to something without having to alter the refcount, so you get significantly less refcount traffic than in other languages where such a thing is unsafe (or not under your control). That's especially important for Arc. It also provides for nice safe optimisations like "move out of this Rc if I'm the only owner" (Rc/Arc::try_unwrap).
danieldk 8y ago
I think you are confusing Rc/Arc with RefCell, since they are often used together. In order to provide interior mutability, the borrow_mut method of RefCell takes &self rather than &mut self. So, it is possible to obtain two mutable references. The following compiles: let c = RefCell::new(42usize); let mut mut_ref1 = c.borrow_mut(); let mut mut_ref2 = c.borrow_mut(); *mut_ref1 += 1; *mut_ref2 += 2; RefCell uses UnsafeCell to subvert the borrows checker. However, instead RefCell implements run-time borrow checks. So, the code above will panic while attempting to do the second mutable borrow. It is a trade-off. You lose catching errors during compilation, but you can have interior mutability. This is one of the goals of unsafe, which UnsafeCell requires. Sometimes, you need to do something that safe Rust does not permit, but that you can prove to be safe (such as RefCell). That said (and it depends a bit on your domain), I have used Rust for probably 1.5 years without using Arc, Rc, and RefCell. The first time I needed these types when was writing a Gtk+ application and later in some multi-threading programs. These three types are very easy to overuse when you are coming from other languages. IMO it is better to avoid them and try to work with the borrows checker until you have sufficient experience with Rust to identify when interior mutability and reference counting are really necessary.

kibwen 8y ago

I think there's mental mismatch between groups of people who talk about "turning off the borrow checker". The borrow checker is a tool to validate references. Sometimes people using references in Rust might feel like the borrow checker makes using references too cumbersome in a certain situation, so they switch to using a tool other than references (Rc, indexes into a Vec, etc). But this isn't bypassing the borrow checker; it's bypassing references themselves. The same phenomenon happens in C++; if references start to be a pain, you might switch to using something else (shared_ptr, indexes into a vector, etc.). When this happens in C++, we don't call this "bypassing the borrow checker". You don't need a borrow checker to know that references aren't always the right tool for a given job. It's the same in Rust.

steveklabnik 8y ago
Yup. Part of why I wrote this post is for exactly this reason. This phrase is used colloquially, but I think it misleads a lot of people on how Rust actually works.

orf 8y ago

> This means that we can combine it with Option<T>, and the option will use the null case for None: This sounds interesting, can anyone elaborate on this?

tux3 8y ago
It's a really cool optimization, and it doesn't just apply to Option. Here's a StackOverflow thread: https://stackoverflow.com/questions/46557608/what-is-the-null-pointer-optimization-in-rust https://stackoverflow.com/questions/46557608/what-is-the-nul...
kibwen 8y ago
One of the design goals of Option is to be a library-level replacement for the language-level null value found in languages like C. Furthermore, one of the design goals for Rust itself is to have its abstractions be zero-overhead. With a naive implementation of Option, these goals would be in opposition. To illustrate why these goals would be in opposition, look at a trivial example of a tagged union (enum) in Rust: enum Foo { Bar(i32), Qux(u32) } At runtime, any value of type Foo will either be in the Bar state, or it will be in the Qux state. Both Bar and Qux hold types that are 32 bits in size, and since only one of those states can be active at a time, we know that Foo only needs 32 bits of storage to satisfy both these states. But additionally, it needs extra storage to store the runtime information telling us which state it's currently in. The smallest "extra" amount of storage that can be added to a type is 8 bits, so we would expect every value of type Foo to be 40 bits in size at runtime. In fact it's larger, due to alignment and padding, so Foo will be 64 bits in size at runtime. Let's bring it back around to Option, which is an enum that looks like this: enum Option<T> { Some(T), None } A null pointer in C will be the same size as a non-null pointer in C: 64 bits, assuming a 64-bit platform. A Rust reference would also be 64 bits, and these cannot be null. If we were to try to use Option on a reference to "opt-in" to nullability, what size would that "nullable reference" be assuming a naive implementation of Option? Well, firstly we'd need storage for the value of the reference itself (64 bits), and then, as per above, we'd need our "extra" storage to tell us at runtime whether our Option is a Some or a None. And again, because of alignment and padding, this would theoretically result in a type that is 128 bits in size in total, which is real shame since in theory distinguishing between two states only takes a single bit of storage. Overall this would be a performance regression from C, where nullability does not impose any space overhead. Fortunately, Rust's implementation is not naive. Remember: Rust references cannot be null. That means that the Rust compiler knows that any type that is a reference will not contain a value that is all zeroes at runtime. Rust leverages this knowledge for optimization: for any Option containing a reference, only a single pointer-sized piece of memory is needed, and the None case will be represented by a value of all zeroes. This means that the Option is now a zero-overhead abstraction for this use case, because Option<&Foo> will be the same size as &Foo. And this smart logic isn't hardcoded for the Option enum. Any enum, written by anyone, can automatically benefit from the ability to "hide" the enum tag in such "uninhabited" values. The OP's example of NonNull<T> is, like references, an example of of a type that has an uninhabited value that permits this optimization. Others include the NonZeroU8 type and its friends, where Option<NonZeroU8> will be the same size as a standard u8, though which give up the ability to represent zero (in the future this may be extended to allow arbitrary user-defined types which can make whatever values they want act as uninhabited for the purposes of enum size optimization, but it will take some work to get there).

shawn 8y ago

Note that you can drop down to unsafe C-style code in Rust. https://doc.rust-lang.org/stable/nomicon/ https://doc.rust-lang.org/stable/nomicon/ Anyone who claims Rust is simple should ensure they thoroughly understand that book. Another way to "turn off" the borrow checker is to write a scripting language that compiles to Rust which automatically annotates all variables with the longest lifetime possible, and spits out mutable references depending on whether you actually mutate anything. There's also RefCell, which lets you defer borrow checking till runtime. It's handy for pretending like your references are immutable.

ChrisSD 8y ago
Also note that none of those things turn off the borrow checker. Which is the point of the article.
kibwen 8y ago
I don't think that anyone's claiming that Rust is objectively a simple language. Simpler than some other languages, certainly, but it's a medium-sized language at best. Furthermore, unsafe code gives new users a firm boundary of complexity that can be ignored. New to Rust? Don't use the unsafe keyword. Using the unsafe keyword? Read the nomicon first. It's quite useful for onboarding to know that all the C-style UB shenanigans are behind a gate that can be ignored until you're comfortable with the rest of the language.

jononor 8y ago

My TLDR/alternate title: "unsafe Rust retains most safety benefits of Rust (including the borrow checker)"

twarge 8y ago

As a non-coder (physicist) writing Rust, the thing that really stuck me was that the time between successful compile and flawless operation was significantly shorter than the C and Python I write. Furthermore, this difference when writing anything threaded is simply breathtaking. Im my experience there are simply fewer corner cases that the Rust compiler lets through.

blueprint 8y ago
I think that makes you a coder :)