8 ms·

The Rustonomicon: The Dark Arts of Advanced and Unsafe Rust Programming

191 points by jvns 10y ago

rattray 10y ago
I really hope nobody mistakes the sexy title of this book as a reason to learn unsafe rust. If it can be done without this advanced feature, it probably should.
- vvanders 10y ago
  > However if you intend to write unsafe code -- or just want to dig into the guts of the language -- this book contains invaluable information. Agreed, however I think the latter part of the intro is worthwhile to anyone who wants a deeper understanding of Rust. I've always found that a in depth understanding of what you use pays off in unforeseen ways.
- bluejekyll 10y ago
  It's true. I have a 30k+ LoC project, and it has zero unsafe code blocks. Though I do have another project that exports a C API, and that requires unsafe, though it's literally only for wrapping/unwrapping pointers.
  - sanderjd 10y ago
    Are you able to say what your project is? I'm always curious to hear about bigger projects!
    - bluejekyll 10y ago
      Sure: https://github.com/bluejekyll/trust-dns https://github.com/bluejekyll/trust-dns It's a WIP, but mostly feature complete at this point. I'm working on DNSCrypt at the moment.
      - e12e 10y ago
        Nice project. Seems a bit light on documentation? Is the intention to be a bind work-alike, or have you taken some inspiration from other servers, like eg: DJB's djbdns? [ed: just found src/config/test - looks more like a "better bind" than "djbdns in rust"]
        bluejekyll 10y ago
        Not sure I totally understand what you mean by djbdns in Rust. I have taken some of djb's ideas, and plan on incorporating more, like randomized QID and Port to make cache poisoning more difficult (added recently). I've implemented this from RFC's only. It's not based on any other DNS server per sé. My insiration started with yet another Bind exploit, and then grew from there. Unlike djbdns, I want this really to have as little manual operation as possible. After DNSCrypt I'm going to look at implementing a version of Raft on top of DNS updates/notifies, etc, to make server resiliency even better. DNSCrypt is next, mostly so thy I can use it for private comms between DNS nodes. djb's is amazing, but what I'm looking for is to build something more flexible and more feature rich.
        bluejekyll 10y ago
        Re: docs The code is well documented ;) Operational docs will come, but I haven't really had an opportunity to focus on those yet.
- kibwen 10y ago
  I wouldn't worry too much, the text of the book goes to great and colorful lengths to dissuade people from using these features frivolously. :)
- vertex-four 10y ago
  The book actually contains a bunch of information on implementation details and other things that aren't unsafe, but just don't have a good place in the regular book. Additionally, if nobody writes unsafe Rust, who's going to maintain the abstractions that the rest of us rely on? :)
- pjmlp 10y ago
  Looking at Rust code with the eyes of a Wirth and Xerox Parc language fan, I still get the feeling that Rust requires too much uses of unsafe code versus Ada, Modula-3 and similar. Specially for basic data structures like trees, graphs and double linked lists.
  - Gankro 10y ago
    I would be very interested in how these two languages enable safe data structuring. Do they have significant tradeoffs in terms of ergonomics, control, or perf? Keep in mind basically all of Rust's libcollections uses unsafe code for perf and low-level representation reasons. They can all be made safe using the strategies every other safe language uses (and eating the associated downsides). Also keep in mind Rust's data structures are basically language primitives in a lot of other languages!
    - pjmlp 10y ago
      Modula-3 has GC. You can only make use of untraced pointers in unsafe modules. So unless proven otherwise by the profiler, no need to make use of unsafe code. Ada requires explicit use of unsafe code for memory deallocation, but it is not required when using RAII pools. So unless a custom allocator is required, the standard pools can be used. In Rust even a basic double linked list requires unsafe. Then there are those that make use of unsafe to try to express code that cannot be proven by the type system, instead of plain memory corruption issues. Thankfully the core team is against this trend.
      - tatterdemalion 10y ago
        > In Rust even a basic double linked list requires unsafe. Well this just isn't true. You can write a doubly linked list using Rc and Weak pointers and no unsafe code. This is the same as saying you can write a doubly linked list using GC'd pointers.
      - geofft 10y ago
        I didn't see this comment when replying above, but you can write a doubly-linked list in safe Rust with either of these approaches (GC, or stuffing everything into a single giant reference). The only reason a "basic" doubly-linked list in Rust is unsafe is that those aren't really reasonable options, and Rust wants to do better. I don't think it would improve Rust to give it mandatory GC (Go and Java are both great languages if you want that), or to require that all allocations are unsafe unless you use memory pools.
Gankro 10y ago
Hey there, I wrote pretty much all of this. Sadly I wasn't ever able to finish it, and won't ever be able to due to contractual obligations. It has since languished due to lack of maintainership, even though it's part of the official rust docs. If anyone wants to take up the mantle and clean it up, it would be greatly appreciated!
- binarycrusader 10y ago
  Are you writing a book instead? If so, looking forward to it. If not, I'm completely a loss to understand why contractual obligations would prevent you from writing something.
  - loeg 10y ago
    Perhaps the contractual obligations tie up all of his or her time.
    - coolsunglasses 10y ago
      Could be copyright assignment for off-hours work too.
    - Gankro 10y ago
      These answers basically cover it.
      - coolsunglasses 10y ago
        I, too, fear the wrath of Lord Cthulhu.
    - hinkley 10y ago
      Given the entry paragraph, perhaps Lord Cthulhu doesn't want him revealing any more summoning secrets.
- sitkack 10y ago
  From your perspective, do you have any cleanup items that are top priority ?
Animats 10y ago
"However BTreeMap is implemented using a modest spoonful of Unsafe Rust (most collections are)." Unsafe pure Rust code reflects a lack of expressive power. How raw memory becomes a typed object is a touchy area for arrays. To do collections safely, you have to be able to talk about an array which is partially valid. This is quite possible, but you need formal verification like constructs to do it. You need to be able to express that a slice of an array is valid, even though the whole array isn't. If you can talk about a slice being valid, and an element being valid, you're almost there. It's a theorem that if an element is adjacent to the end of a slice, the element has some property, and all elements in the slice have a property, then a slice containing the original slice and the element has that property. Using that, you can prove by induction that as you grow a collection, the elements in use remain valid. If the access primitives don't let you access outside the valid range, they're safe. With some extra predicates and a modest theorem prover, much unsafe code in the collection area could probably be proven safe. Rather than describing unsafe code as a "dark art", it would be more useful to try to formalize safety in this way. The theory is well understood, and there are many proof of correctness systems around for other languages. It might require a lot of annotation in unsafe code, to help the prover along, but that's not a bad thing.
- ineol1 10y ago
  Derek Dreyer and collaborators are working on specifying what needs to be proven about unsafe code, see http://plv.mpi-sws.org/rustbelt/ http://plv.mpi-sws.org/rustbelt/
  - Animats 10y ago
    That's good. They may have trouble recruiting. They're looking for people who could make upwards of $150K in Silicon Valley and want to pay them at postdoc rates. Glassdoor says €41,595. Their own material has even lower rates.[1] [1] https://www.mpg.de/8746248/Foerderrichtlinien_Postdoc_en.pdf https://www.mpg.de/8746248/Foerderrichtlinien_Postdoc_en.pdf
    - dikaiosune 10y ago
arc0re 10y ago
I'm a Lovecraft fan, never really touched Rust, and a name like "Rustonomicon" could really get me into it, hehe.

191 points by jvns 10y ago

I really hope nobody mistakes the sexy title of this book as a reason to learn unsafe rust. If it can be done without this advanced feature, it probably should.

vvanders 10y ago
> However if you intend to write unsafe code -- or just want to dig into the guts of the language -- this book contains invaluable information. Agreed, however I think the latter part of the intro is worthwhile to anyone who wants a deeper understanding of Rust. I've always found that a in depth understanding of what you use pays off in unforeseen ways.
bluejekyll 10y ago
It's true. I have a 30k+ LoC project, and it has zero unsafe code blocks. Though I do have another project that exports a C API, and that requires unsafe, though it's literally only for wrapping/unwrapping pointers.
- sanderjd 10y ago
  Are you able to say what your project is? I'm always curious to hear about bigger projects!
  - bluejekyll 10y ago
    Sure: https://github.com/bluejekyll/trust-dns https://github.com/bluejekyll/trust-dns It's a WIP, but mostly feature complete at this point. I'm working on DNSCrypt at the moment.
    - e12e 10y ago
      Nice project. Seems a bit light on documentation? Is the intention to be a bind work-alike, or have you taken some inspiration from other servers, like eg: DJB's djbdns? [ed: just found src/config/test - looks more like a "better bind" than "djbdns in rust"]
      - bluejekyll 10y ago
        Not sure I totally understand what you mean by djbdns in Rust. I have taken some of djb's ideas, and plan on incorporating more, like randomized QID and Port to make cache poisoning more difficult (added recently). I've implemented this from RFC's only. It's not based on any other DNS server per sé. My insiration started with yet another Bind exploit, and then grew from there. Unlike djbdns, I want this really to have as little manual operation as possible. After DNSCrypt I'm going to look at implementing a version of Raft on top of DNS updates/notifies, etc, to make server resiliency even better. DNSCrypt is next, mostly so thy I can use it for private comms between DNS nodes. djb's is amazing, but what I'm looking for is to build something more flexible and more feature rich.
      - bluejekyll 10y ago
        Re: docs The code is well documented ;) Operational docs will come, but I haven't really had an opportunity to focus on those yet.
kibwen 10y ago
I wouldn't worry too much, the text of the book goes to great and colorful lengths to dissuade people from using these features frivolously. :)
vertex-four 10y ago
The book actually contains a bunch of information on implementation details and other things that aren't unsafe, but just don't have a good place in the regular book. Additionally, if nobody writes unsafe Rust, who's going to maintain the abstractions that the rest of us rely on? :)
pjmlp 10y ago
Looking at Rust code with the eyes of a Wirth and Xerox Parc language fan, I still get the feeling that Rust requires too much uses of unsafe code versus Ada, Modula-3 and similar. Specially for basic data structures like trees, graphs and double linked lists.
- Gankro 10y ago
  I would be very interested in how these two languages enable safe data structuring. Do they have significant tradeoffs in terms of ergonomics, control, or perf? Keep in mind basically all of Rust's libcollections uses unsafe code for perf and low-level representation reasons. They can all be made safe using the strategies every other safe language uses (and eating the associated downsides). Also keep in mind Rust's data structures are basically language primitives in a lot of other languages!
  - pjmlp 10y ago
    Modula-3 has GC. You can only make use of untraced pointers in unsafe modules. So unless proven otherwise by the profiler, no need to make use of unsafe code. Ada requires explicit use of unsafe code for memory deallocation, but it is not required when using RAII pools. So unless a custom allocator is required, the standard pools can be used. In Rust even a basic double linked list requires unsafe. Then there are those that make use of unsafe to try to express code that cannot be proven by the type system, instead of plain memory corruption issues. Thankfully the core team is against this trend.
    - tatterdemalion 10y ago
      > In Rust even a basic double linked list requires unsafe. Well this just isn't true. You can write a doubly linked list using Rc and Weak pointers and no unsafe code. This is the same as saying you can write a doubly linked list using GC'd pointers.
    - geofft 10y ago
      I didn't see this comment when replying above, but you can write a doubly-linked list in safe Rust with either of these approaches (GC, or stuffing everything into a single giant reference). The only reason a "basic" doubly-linked list in Rust is unsafe is that those aren't really reasonable options, and Rust wants to do better. I don't think it would improve Rust to give it mandatory GC (Go and Java are both great languages if you want that), or to require that all allocations are unsafe unless you use memory pools.

Gankro 10y ago

Hey there, I wrote pretty much all of this. Sadly I wasn't ever able to finish it, and won't ever be able to due to contractual obligations. It has since languished due to lack of maintainership, even though it's part of the official rust docs. If anyone wants to take up the mantle and clean it up, it would be greatly appreciated!

binarycrusader 10y ago
Are you writing a book instead? If so, looking forward to it. If not, I'm completely a loss to understand why contractual obligations would prevent you from writing something.
- loeg 10y ago
  Perhaps the contractual obligations tie up all of his or her time.
  - coolsunglasses 10y ago
    Could be copyright assignment for off-hours work too.
  - Gankro 10y ago
    These answers basically cover it.
    - coolsunglasses 10y ago
      I, too, fear the wrath of Lord Cthulhu.
  - hinkley 10y ago
    Given the entry paragraph, perhaps Lord Cthulhu doesn't want him revealing any more summoning secrets.
sitkack 10y ago
From your perspective, do you have any cleanup items that are top priority ?

Animats 10y ago

"However BTreeMap is implemented using a modest spoonful of Unsafe Rust (most collections are)." Unsafe pure Rust code reflects a lack of expressive power. How raw memory becomes a typed object is a touchy area for arrays. To do collections safely, you have to be able to talk about an array which is partially valid. This is quite possible, but you need formal verification like constructs to do it. You need to be able to express that a slice of an array is valid, even though the whole array isn't. If you can talk about a slice being valid, and an element being valid, you're almost there. It's a theorem that if an element is adjacent to the end of a slice, the element has some property, and all elements in the slice have a property, then a slice containing the original slice and the element has that property. Using that, you can prove by induction that as you grow a collection, the elements in use remain valid. If the access primitives don't let you access outside the valid range, they're safe. With some extra predicates and a modest theorem prover, much unsafe code in the collection area could probably be proven safe. Rather than describing unsafe code as a "dark art", it would be more useful to try to formalize safety in this way. The theory is well understood, and there are many proof of correctness systems around for other languages. It might require a lot of annotation in unsafe code, to help the prover along, but that's not a bad thing.

ineol1 10y ago
Derek Dreyer and collaborators are working on specifying what needs to be proven about unsafe code, see http://plv.mpi-sws.org/rustbelt/ http://plv.mpi-sws.org/rustbelt/
- Animats 10y ago
  That's good. They may have trouble recruiting. They're looking for people who could make upwards of $150K in Silicon Valley and want to pay them at postdoc rates. Glassdoor says €41,595. Their own material has even lower rates.[1] [1] https://www.mpg.de/8746248/Foerderrichtlinien_Postdoc_en.pdf https://www.mpg.de/8746248/Foerderrichtlinien_Postdoc_en.pdf
  - dikaiosune 10y ago

arc0re 10y ago

I'm a Lovecraft fan, never really touched Rust, and a name like "Rustonomicon" could really get me into it, hehe.