5 ms·

> There's still too much that has to be done with unsafe code. But the unsafe situations are starting to form patterns. I think as long as those patterns can b

by elihu 10y ago

> There's still too much that has to be done with unsafe code. But the unsafe situations are starting to form patterns. I think as long as those patterns can be abstracted out and moved into thoroughly-vetted libraries with a safe interface, unsafe code isn't really a problem. I expect that getting Rust's standard libraries to a place where regular applications very rarely need to create their own unsafe code blocks is going to be a major long-term effort. (Of course, if there were some simple language feature that would make some common use case of unsafe code blocks unnecessary, by all means we should do that as well.)

steveklabnik 10y ago
Is there something today that you're seeing that causes applications to use a lot of unsafe? In general, it should mostly be in library code, not application code.
- elihu 10y ago
  No, but I'm pretty new to rust and haven't written enough code to know how often situations that call for unsafe code blocks come up. (My current project I'm using to learn Rust is to port a ray-tracer I wrote in Haskell. I wouldn't expect functional-style code to require a lot of unsafe blocks and I haven't needed any yet, but who knows?)
  - steveklabnik 10y ago
    Cool, just wondering. In general, unsafe code should be encapsulated, to isolate its possible effects, and once you've got it isolated, it's easy to break that bit out in a library.
    - vvanders 10y ago
      From poking around in Rust for ~1 year my common ones are uninitialized arrays for scratch buffers(> 1Kb) and FFI(obviously). Like you said all the other cases are nicely wrapped into a library but I'm not quite sure how you'd abstract the first one above. It's small enough to do the unsafe block inline that it's not annoying but definitely shows up(esp when you hit APIs that are copy of C interfaces).
      - steveklabnik 10y ago
        Depends on what you're doing; see https://crates.io/crates/init_with https://crates.io/crates/init_with as one example. It's not the exact same thing, and yeah, I would argue that's one area where it's okay to not totally abstract it. Rules are meant to be broken. (I was thinking about removing bounds checks with get_unchecked as well).
        vvanders 10y ago
        That's a much more involved use case that I usually have. Looks like a solid abstraction though. Like I said, not really an issue, just a common pattern that I see.
        steveklabnik 10y ago
        Absolutely, understood. Ironically, this is also a good example of how unsafe is complicated; "hey make an array (not vector) of copies of this thing" has lots of edge cases!
  - Animats 10y ago
    I listed two things Rust doesn't handle well without unsafe code, doubly linked lists and multidimensional arrays. Here are examples of both from popular repositories with high download numbers: - https://github.com/andelf/rust-adivon/blob/master/src/deque.rs https://github.com/andelf/rust-adivon/blob/master/src/deque.... - https://github.com/andelf/rust-adivon/blob/master/src/queue.rs https://github.com/andelf/rust-adivon/blob/master/src/queue.... These are all the doubly-linked list problem: struct Node<T> { item: T, next: Option<Box<Node<T>>>, prev: Rawlink<Node<T>> } Since this is templated code, it might be possible to break it by instantiating it on a type with unusual semantics. - https://github.com/BurntSushi/aho-corasick/blob/master/src/full.rs https://github.com/BurntSushi/aho-corasick/blob/master/src/f... Looks like unsafe code for "performance reasons". But there are no comments near "unsafe", so it's hard to tell. - https://github.com/SiegeLord/RustAlgebloat/blob/master/algebloat/src/index.rs https://github.com/SiegeLord/RustAlgebloat/blob/master/algeb... Matrix math. "Unsafe" all over the place, and unsafeness is exported, allowing the caller to do unsafe things. This is an example of why I occasionally stress the need for multidimensional array support at the language level. If the compiler knew about multidimensional arrays, it could optimize the subscript checks for them, avoiding code such as this. The unsafe version. The caller can store anywhere in memory. fn unsafe_set_idx(&self, $mat: &T, v: f64) { let $self_ = self; let (r, c) = $rc_expr; unsafe { $mat.raw_set(r, c, v) } } Safe, but inefficient version. The compiler can't hoist those checks out of loops. fn set_idx(&self, $mat: &T, v: f64) { let $self_ = self; let (r, c) = $rc_expr; assert!(r < $mat.nrow()); assert!(c < $mat.ncol()); unsafe { $mat.raw_set(r, c, v) } } This is the sort of thing that leads to exploits in code that reads things like JPEG files. Yet you can't do much better in Rust. That's why doing multidimensional arrays in macros and templates isn't good enough.
    - dbaupp 10y ago
      Why do you still think the compiler can't host those out of loops? LLVM is perfectly capable of doing so in many cases and you've been told as much previously, could you make your claim more specific?
      - Animats 10y ago
paulddraper 10y ago
Agreed. And considering that the most common uses of unsafe are for collections, I think a limited number of vetted libraries completely realistic. Rolling your own collections is typically not that great anyway; how much better is your linked list/hash table/skip list than everyone elses?
- phamilton 10y ago
  As long as there are 40+ collections data structures like in Java.
firebones 10y ago
What I worry about is those "thoroughly-vetted libraries" is that the use of those will likely follow a power law distribution in terms of code dependent upon them. When that happens, a disproportionately-high amount of code will depend upon those libraries. So when a defect (or vulnerability) shows up in one of them, a disproportionately-high amount of code will become vulnerable. Reducing this unsafe surface area should result in outsized gains overall. I'm not familiar with the unsafe patterns, and to what extent they could be addressed/reduced through Rust language/compiler design, but that'd be one area where research focus might produce even more benefit. The other weak point may be the LLVM itself, which I guess would be sort of a background hum of risk to Rust. Outsider view: not sure of the relative levels of risk in each layer, and to what extent they can be addressed. Can (or will) these unsafe patterns be capable of being addressed by future changes in Rust?

by elihu 10y ago

steveklabnik 10y ago

Is there something today that you're seeing that causes applications to use a lot of unsafe? In general, it should mostly be in library code, not application code.

elihu 10y ago
No, but I'm pretty new to rust and haven't written enough code to know how often situations that call for unsafe code blocks come up. (My current project I'm using to learn Rust is to port a ray-tracer I wrote in Haskell. I wouldn't expect functional-style code to require a lot of unsafe blocks and I haven't needed any yet, but who knows?)
- steveklabnik 10y ago
  Cool, just wondering. In general, unsafe code should be encapsulated, to isolate its possible effects, and once you've got it isolated, it's easy to break that bit out in a library.
  - vvanders 10y ago
    From poking around in Rust for ~1 year my common ones are uninitialized arrays for scratch buffers(> 1Kb) and FFI(obviously). Like you said all the other cases are nicely wrapped into a library but I'm not quite sure how you'd abstract the first one above. It's small enough to do the unsafe block inline that it's not annoying but definitely shows up(esp when you hit APIs that are copy of C interfaces).
    - steveklabnik 10y ago
      Depends on what you're doing; see https://crates.io/crates/init_with https://crates.io/crates/init_with as one example. It's not the exact same thing, and yeah, I would argue that's one area where it's okay to not totally abstract it. Rules are meant to be broken. (I was thinking about removing bounds checks with get_unchecked as well).
      - vvanders 10y ago
        That's a much more involved use case that I usually have. Looks like a solid abstraction though. Like I said, not really an issue, just a common pattern that I see.
        steveklabnik 10y ago
        Absolutely, understood. Ironically, this is also a good example of how unsafe is complicated; "hey make an array (not vector) of copies of this thing" has lots of edge cases!
- Animats 10y ago
  I listed two things Rust doesn't handle well without unsafe code, doubly linked lists and multidimensional arrays. Here are examples of both from popular repositories with high download numbers: - https://github.com/andelf/rust-adivon/blob/master/src/deque.rs https://github.com/andelf/rust-adivon/blob/master/src/deque.... - https://github.com/andelf/rust-adivon/blob/master/src/queue.rs https://github.com/andelf/rust-adivon/blob/master/src/queue.... These are all the doubly-linked list problem: struct Node<T> { item: T, next: Option<Box<Node<T>>>, prev: Rawlink<Node<T>> } Since this is templated code, it might be possible to break it by instantiating it on a type with unusual semantics. - https://github.com/BurntSushi/aho-corasick/blob/master/src/full.rs https://github.com/BurntSushi/aho-corasick/blob/master/src/f... Looks like unsafe code for "performance reasons". But there are no comments near "unsafe", so it's hard to tell. - https://github.com/SiegeLord/RustAlgebloat/blob/master/algebloat/src/index.rs https://github.com/SiegeLord/RustAlgebloat/blob/master/algeb... Matrix math. "Unsafe" all over the place, and unsafeness is exported, allowing the caller to do unsafe things. This is an example of why I occasionally stress the need for multidimensional array support at the language level. If the compiler knew about multidimensional arrays, it could optimize the subscript checks for them, avoiding code such as this. The unsafe version. The caller can store anywhere in memory. fn unsafe_set_idx(&self, $mat: &T, v: f64) { let $self_ = self; let (r, c) = $rc_expr; unsafe { $mat.raw_set(r, c, v) } } Safe, but inefficient version. The compiler can't hoist those checks out of loops. fn set_idx(&self, $mat: &T, v: f64) { let $self_ = self; let (r, c) = $rc_expr; assert!(r < $mat.nrow()); assert!(c < $mat.ncol()); unsafe { $mat.raw_set(r, c, v) } } This is the sort of thing that leads to exploits in code that reads things like JPEG files. Yet you can't do much better in Rust. That's why doing multidimensional arrays in macros and templates isn't good enough.
  - dbaupp 10y ago
    Why do you still think the compiler can't host those out of loops? LLVM is perfectly capable of doing so in many cases and you've been told as much previously, could you make your claim more specific?
    - Animats 10y ago

paulddraper 10y ago

Agreed. And considering that the most common uses of unsafe are for collections, I think a limited number of vetted libraries completely realistic. Rolling your own collections is typically not that great anyway; how much better is your linked list/hash table/skip list than everyone elses?

phamilton 10y ago
As long as there are 40+ collections data structures like in Java.

firebones 10y ago

What I worry about is those "thoroughly-vetted libraries" is that the use of those will likely follow a power law distribution in terms of code dependent upon them. When that happens, a disproportionately-high amount of code will depend upon those libraries. So when a defect (or vulnerability) shows up in one of them, a disproportionately-high amount of code will become vulnerable. Reducing this unsafe surface area should result in outsized gains overall. I'm not familiar with the unsafe patterns, and to what extent they could be addressed/reduced through Rust language/compiler design, but that'd be one area where research focus might produce even more benefit. The other weak point may be the LLVM itself, which I guess would be sort of a background hum of risk to Rust. Outsider view: not sure of the relative levels of risk in each layer, and to what extent they can be addressed. Can (or will) these unsafe patterns be capable of being addressed by future changes in Rust?