11 ms·
Concurrency in Rust
- nindalf 11y agoI think Steve Klabnik could clarify this, but the book at that link is in the process of being rewritten. I think it might be good to wait until it is. I personally found it slightly difficult to follow compared to other options like the soon to be published Programming Rust.
- steveklabnik 11y agoI am in the middle of working on a second draft of the book. This page is one of the oldest bits of docs, overall, and isn't my best work. It's not _wrong_, I just have very high personal standards. It was adapted from older documentation and was written in the time up to 1.0, where I had a LOT on my plate.
- galonk 11y agoIt certainly is _confusing_ if not _wrong_. You silently add "move" to the closure without any mention or explanation (then later say "note that we're copying i" without explaining that you're talking about the "move" keyword). The bit about Mutex also has a "just type this to fix the problem with no explanation of how or why it works" flavour (although I guess if you already grok mutexes then its use here might be obvious to you). Not a criticism of the tutorial, but it does something common in Rust tutorials which is really a problem with the language at this point: Rust tutorials always spend a lot of time interpreting Rust's notoriously poor error messages (e.g. "what this message that doesn't mention Sync is trying to tell you is that you need Sync on this type"). That's great when you're doing the tutorial, but as soon as you're on your own man are those errors frustrating.
- steveklabnik 11y agoYeah, this is what I mean. So this was originally written before "move" or even any of the current closure implementation existed. And with so much to do, the focus was on making what existed accurate more than a holistic approach. Furthermore, almost a year after 1.0, we have a lot better understanding of what people struggle with when trying to learn Rust, so we have a better idea of how to teach it. Please file bugs for notoriously poor errors. We have put a lot of work into many of them, but there's a lot of ways to go. It seems like most people really like them or really hate them.
- Manishearth 11y ago> Rust's notoriously poor error messages I've actually heard the opposite feedback; Rust's error messages try to be super helpful. Note that concepts like ownership and Sync are new to most programmers, and it's impossible to explain them in an error message. This is where the extended error messages (via --explain) and the book come in. The tutorial has this style because Rust espouses catching things at compile time, so the tutorial demonstrates this being done by doing the wrong thing a few times and reiterating why it gave an error. ---- Could you give examples of confusing error messages? I'd love to improve them. The one you mention .... doesn't exist. This (http://is.gd/keukPm http://is.gd/keukPm) is what that error message looks like, and (a) it mentions Sync, (b) it also mentions that `Arc<bool>` cannot be shared between threads safely. If you were referring to "So, we need some type that lets us have more than one reference to a value and that we can share between threads, that is it must implement Sync." from the book, the last part about Sync has nothing to do with that error message. If you follow the book, the error message is clear without the context of threads -- `data` was moved into the first spawn() call and the subsequent ones can't use it. One does not conclude that Sync is necessary from this error message, and that's not what the book is trying to say. This sentence is actually skipping a step, one like http://is.gd/RPNOm4 http://is.gd/RPNOm4, where the compiler asks you for a Send type (or a Sync type, depending on the exact code). Instead of stepping through this example, it just introduces Sync directly by noting that we're dealing with threads anyway and the reader already knows what Sync/Send are. It doesn't conclude that Sync is necessary from the error message. > The bit about Mutex also has a "just type this to fix the problem with no explanation of how or why it works" flavour (although I guess if you already grok mutexes then its use here might be obvious to you). The previous sentence says "for example a type that can ensure only one thread at a time is able to mutate the value inside it at any one time.", which is exactly what a mutex does. Unless you want a lower level explanation which IMO isn't necessary. It could explain locking more though; I'll fix that.
- firebones 11y agoI've found the Rust compiler to provide very useful messages. That said, it does take a certain amount of experience in the language before you grok the terminology and appropriate remedies to overcome some compiler errors. As a beginner, I'd typically open about 5-10 different documentation, blog post, SO and tutorial pages to try to figure out what I was doing wrong. Once I understood the underlying concept, I could go back and understand what the compiler message was telling me, as plain as day. After awhile, I had a strong enough understanding of the borrow checker that I could usually interpret the compiler error messages in a pretty straightforward fashion. But it takes time if you're coming from a non C/C++ language! If you're learning Rust, I would recommend tracing back every compiler error you encounter to this page [1] to see some other examples and resolutions (if the compiler's suggestions don't make it clear as to what to do). It's the most underrated page in the entire Rust documentation set, and could be the launch point for a lot of teaching and learning. Read the book to start, certainly (and the revisions Steve Klabnik has been making are very, very good), but that error index page is very valuable for the day to day issues. Rust rewards gumption [2] and perseverance. [1] https://doc.rust-lang.org/error-index.html https://doc.rust-lang.org/error-index.html [2] https://en.wikipedia.org/wiki/Gumption_trap https://en.wikipedia.org/wiki/Gumption_trap
- Manishearth 11y agoFWIW, I fixed some of these concerns here: https://github.com/rust-lang/rust/pull/32529 https://github.com/rust-lang/rust/pull/32529
- lambda 11y agoRust tutorials always spend a lot of time interpreting Rust's notoriously poor error messages I'm wondering where you got the sense that Rust's error messages are "notoriously poor." Can you expand on that? I have always found Rust's error messages to be quite good. Of course, they can be hard to interpret occasionally if you don't already grasp the concepts they are referring to. But I don't see any way to solve that in error messages; at some point, you have to learn enough about the language for the error messages to make sense. And occasionally, the suggestion they provide for how to fix the issue isn't the right suggestion, but I don't know if there's a way to always do that without strong AI that understands what you mean. The best you could hope to do is to provide suggestions for all of the possible fixes, though that could lead to some messages that are too verbose to the point of being useless. I think it would help to provide some examples of error messages you have had trouble with, to help figure out ways to make them better.
- pcwalton 11y ago> what this message that doesn't mention Sync is trying to tell you is that you need Sync on this type There's no such error message that I'm aware of.
- nindalf 11y agoThanks Steve, you're doing great work. Can't wait to read the book once its done!
- steveklabnik 11y agoThanks! You can follow along with the progress here, if you're interested: https://github.com/rust-lang/book https://github.com/rust-lang/book
- Manishearth 11y agoFor a more in-depth explanation of how Send and Sync work theoretically, see http://manishearth.github.io/blog/2015/05/30/how-rust-achieves-thread-safety/ http://manishearth.github.io/blog/2015/05/30/how-rust-achiev... or http://huonw.github.io/blog/2015/02/some-notes-on-send-and-sync/ http://huonw.github.io/blog/2015/02/some-notes-on-send-and-s...
- djhworld 11y agoI'm having a tough time trying to understand this snippet for i in 0..3 { thread::spawn(move || { data[i] += 1; }); } What is the 'move' thing here before the ||
- domenp 11y agoA quick search turned up this: https://doc.rust-lang.org/book/closures.html#move-closures https://doc.rust-lang.org/book/closures.html#move-closures.
- djhworld 11y agoGreat thanks.
- singlow 11y agoA move closure takes copies of the environment, instead of mutating the parent environment. https://doc.rust-lang.org/book/closures.html#move-closures https://doc.rust-lang.org/book/closures.html#move-closures
- pdpi 11y agoIt's more accurate to say that it takes ownership of the captured environment. Whether that means copying the values or moving ownership depends on whether the values are of a type that implements Copy
- rdtsc 11y agoBut isn't it confusing that it is called "move" instead of "copy" if it takes copies?
- steveklabnik 11y agoMove and Copy are identical at the assembly level. The only difference is what you can do with the older binding. Semantically speaking, both cause a memcpy, though the optimizer may elide them. That said, you're right that saying "copy" is misleading, for this reason. But moves _are_ a kind of copy.
- z1mm32m4n 11y agoDoes Rust have a way to work with SIMD concurrency as opposed to just fork/join concurrency? Something along the lines of how openmp or cilk let you do a parallel for all?
- fndjdh 11y agoYou seem to have concurrency confused with parallelism. Concurrency just means working on another task before the prior one has completed. You're describing parallelism which is doing multiple tasks at the same time.
- m0th87 11y agoParallelism necessarily implies concurrency [1] So, "SIMD concurrency" is not incorrect (although SIMD parallelism is more correct :) 1: http://programmers.stackexchange.com/a/155110 http://programmers.stackexchange.com/a/155110
- naasking 11y agoThis not true. CPUs have instruction parallelism, even on a single CPU, but there is no observable concurrency. SIMD is data-level parallelism, not concurrency.
- z1mm32m4n 11y agoBy SIMD concurrency, I meant that I was curious about a construct that actually translated into simultaneously executing code, not just a construct that denotes work that "could" be done simultaneously.
- Chilinot 11y agoA simple google search would have told you that there is work being done the introduce simd in Rust. There appear to be some basics there right now but not much if i understood my quick search.
- kbenson 11y ago
- askyourmother 11y agoWhat about the assumption (fatally flawed decision?) that malloc never fails when rust asks? Sounds like something that could affect concurrency
- anon4 11y agoMalloc never fails, but you might die if you touch the memory. In general, modern OSes don't have a good story about exhausting available memory beyond "let's kill a bunch of processes to free up memory".
- masklinn 11y ago> Malloc never fails malloc can fail, even on default linux (overcommit enabled), if you go above the process's vmem limit for instance (because 32b or rlimited). And of course not all OS overcommit, Windows famously does not.
- Manishearth 11y agoA failed malloc aborts the process. If this is important to you, don't use the heap abstractions in the stdlib then. This is no different from the situation in C++. (You can also plug in a custom allocator which behaves differently)
- qznc 11y agoOn Linux malloc never fails actually. Instead, the kernel kills processes if it runs out of memory.
- gregwebs 11y agoSend + Sync are great. The downside of concurrency in Rust is: 1) There isn't transparent integration with IO in the runtime as in Go or Haskell. Rust probably won't ever do this because although such a model scales well in general, it does create overhead and a runtime. 2) OS threads are difficult to work with compared to a nice M:N threading abstraction (which again are the default in Go or Haskell). OS threads leads to lowest common denominator APIs (there is no way to kill a thread in Rust) and some difficulty in reasoning about performance implications. I am attempting to solve this aspect by using the mioco library, although due to point #1 IO is going to be a little awkward.
- IshKebab 11y agoThere's no way to kill goroutines either. In fact, are there any systems that allow you to cleanly kill threads?
- steveklabnik 11y agoEvery one I know of has regretted it, and seen it as an antipattern. For example, Java way back in 1.5: http://docs.oracle.com/javase/1.5.0/docs/guide/misc/threadPrimitiveDeprecation.html http://docs.oracle.com/javase/1.5.0/docs/guide/misc/threadPr... I think Erlang might be okay with it, because "this thread can fail at any time" is a core value of Erlang. But it's an exception.
- gregwebs 11y agoHaskell has killThread, which rather than being an anti-pattern is often used as an effective way to accurately enforce a timeout on a thread. This functionality seems like it would be very difficult to achieve with most other runtimes. https://news.ycombinator.com/item?id=11370004 https://news.ycombinator.com/item?id=11370004
- steveklabnik 11y agoYou have a sibling comment elsewhere in the thread which disagrees; I'll leave that argument to that sub-thread.
- pmarreck 11y agoReading all this is making me happy about pursuing Elixir (which is of course a language addressing largely different use-cases)
- armitron 11y agoThis looks terribly overcomplicated/overengineered to me, to the point where I doubt many are going to adopt/switch to this style, esp when used to more convenient approaches [even the standard C++ approach, faulty as it may be]. Also note how much boilerplate one has to write and how the code snippets bypass error handling (do it differently in "real" code but don't show us how). Bleh.
- bluejekyll 11y agoTry it before you mock it. This prevents boiler plate issues, and allows the compiler to help you discover threading issues at compile time rather than runtime. It's easy enough to just mark all you structs send+sync and still shoot your foot off just like in any language. The point is, you need to be explicit that your trying to shoot your foot off, as opposed to other languages which basically pull the trigger for you.
- Manishearth 11y ago> overcomplicated/overengineered to me You don't have to worry about most of this. Doing concurrent things in Rust is pretty clean. Designing new concurrent abstractions from scratch is where you need to worry about Send and Sync and be careful. And it's totally worth it, entire classes of concurrency errors just go away. The error handling can get verbose, though with the new `?` operator and `catch` syntax it's much cleaner now.
- RasmusWL 11y agoCan someone enlighten me as to why the first snippet has a data race? Won't the resulting array become [2,3,4]? let mut data = vec![1, 2, 3]; for i in 0..3 { thread::spawn(move || { data[i] += 1; }); }
- gsjs 11y agoI don't think there's a data race there, but the compiler can't check that. What the compiler sees is more than one thread accessing the variable `data`, which could cause a data race.
- Manishearth 11y agoThat's a mistake, clarified: https://github.com/rust-lang/rust/pull/32538 https://github.com/rust-lang/rust/pull/32538
- jonreem 11y agoAnother thing to know about rust concurrency is that it supports safe "scoped" threads, or threads which have plain references to their parent threads stack. This makes it very easy to write, for instance, a concurrent in-place quicksort (this example uses the scoped-pool crate, which provides a thread pool supporting scoped threads): extern crate scoped_pool; // scoped threads extern crate itertools; // generic in-place partition extern crate rand; // for choosing a random pivot use rand::Rng; use scoped_pool::{Pool, Scope}; pub fn quicksort<T: Send + Sync + Ord>(pool: &Pool, data: &mut [T]) { pool.scoped(move |scoped| do_quicksort(scoped, data)) } fn do_quicksort<'a, T: Send + Sync + Ord>(scope: &Scope<'a>, data: &'a mut [T]) { scope.recurse(move |scope| { if data.len() > 1 { // Choose a random pivot. let mut rng = rand::thread_rng(); let len = data.len(); let pivot_index = rng.gen_range(0, len); // Choose a random pivot // Swap the pivot to the end. data.swap(pivot_index, len - 1); let split = { // Retrieve the pivot. let mut iter = data.into_iter(); let pivot = iter.next_back().unwrap(); // In-place partition the array. itertools::partition(iter, |val| &*val <= &pivot) }; // Swap the pivot back in at the split point by putting // the element currently there are at the end of the slice. data.swap(split, len - 1); // Sort both halves (in-place!). let (left, right) = data.split_at_mut(split); do_quicksort(scope, left); do_quicksort(scope, &mut right[1..]); } }) } In this example, quicksort will block until the array is fully sorted, then return.