12 ms·
As mentioned in the post, given Rust wants to operate in the same space as C, this approach makes sense. However from a higher level, building more complex conc
by rdtsc 10y ago
As mentioned in the post, given Rust wants to operate in the same space as C, this approach makes sense. However from a higher level, building more complex concurrent systems, dealing with futures/deferred-s/promises and/or a central select/epoll/kqueue reactor loop gets daunting and doesn't mix with complex business rules.
Deferred based approach has been a round for many years. I experienced it by using Twisted (Python framework) for 5 or so years. And early on it was great However when we switched to using green threads, the logic and amount of code was greatly simplified.
So wondering if Rust provides any ability to add that kind of an N:M threading approach. Perhaps via an extension, macro or some other mechanism.
Note that in C, it being C such things can be done with some low level trickery. Here is a library that attempt that:
http://libmill.org/ http://libmill.org/
And there were a few others before, but none have taken off enough to become mainstream.
- steveklabnik 10y agoYes, this is the lowest layer of the story. For actually writing an app, writing something higher-level makes sense: consider the relationship between raw futures and Finagle, for example. You have to build out the lower layers before you can add the upper ones, though. Rust doesn't natively supply anything like green threads, as they have too heavy of requirements. But there are librarys that can implement them, since Rust is so low-level: https://crates.io/crates/mioco https://crates.io/crates/mioco as an example.
- davidgrenier 10y agoIn other words futures are not as composable as one would hope. John Reppy's CML seems like a much better toolbox which gets composability without pretension. Vesa Karvonen (whom I understand has worked on the MLTon compiler) has offered an excellent delivery in Hopac for C# and F# complete with a slew of combinators: https://github.com/Hopac/Hopac https://github.com/Hopac/Hopac I'm not aware of anyone offering an alternative superior to an informal CSP yet which seems to be the reason why Go and Clojure have picked it as well for their concurrency model. See David Nolen's excellent blog posts on the matter: http://swannodette.github.io/2013/07/12/communicating-sequential-processes http://swannodette.github.io/2013/07/12/communicating-sequen... http://swannodette.github.io/2013/08/17/comparative http://swannodette.github.io/2013/08/17/comparative
- pcwalton 10y agoI think Rust's approach here makes a lot more sense for the language than Go/CML style M:N would. You can recover most of the M:N ergonomics over time via async/await style syntax. But if your foundation is built on a (comparatively) slow "pthreads in userland" threading model, then you hit a performance ceiling that you can never break through. For a language like Rust, it makes sense to begin in the optimum place and then layer zero-cost abstractions on top to achieve the right ergonomics.
- jholman 10y agoI think there's little or no evidence that "you can recover most of the M:N ergonomics over time via async/await style syntax", despite a decade or so of attempts. I think there's an underlying semantic concern that seems unsugarable. Munificent's http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y... expresses the problem eloquently. None of that means you're wrong about async making "a lot more sense for the language", though.
- pcwalton 10y ago> Munificent's http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y.. http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y.... expresses the problem eloquently. I have several issues with that blog post. 1. The performance concerns are glossed over. The conclusion is "threads are superior", but that ignores the reason why we want async I/O in the first place: performance. 2. "What if the new place you want to call it is blue? You’ll have to turn it red." is false. There's a way to convert blocking code to async-friendly code: use a thread pool. This is in fact what cgo does internally—see the next point. 3. You always have red functions and blue functions if you have an FFI. But this isn't a big deal because synchronous functions are easily convertible to asynchronous functions (via a thread pool) and asynchronous functions are easily convertible into synchronous functions (via just blocking). So the supposedly big semantic problem just boils down into a question of making sure you remember to switch into the other mode when necessary (which you can always easily do without restructuring your program). This is something that the language can do for you automatically—proof is that Go does it!—and I would like to see type system or other static approaches to doing this in Rust.
- pcwalton 10y ago> So wondering if Rust provides any ability to add that kind of an N:M threading approach. Perhaps via an extension, macro or some other mechanism. I don't want M:N threading as Go implements it. It's a big loss of performance for marginal benefit over futures. In particular the libmill approach was tried in Rust and the results were far worse than 1:1. However, assuming this takes off I would like to see async/await style syntactic sugar over futures down the road to make it easier to write code that looks blocking but actually isn't. Crucially, this sugar would maintain the zero-cost nature of futures. With that approach, we'd have the ergonomics of languages like Erlang and Go without the significant performance tax associated with M:N.
- saynsedit 10y agoAre libmill and Go's approach the same? I.e. Holding concurrent state in multiple stack frames? What causes that to have degraded performance? Is this new approach different because instead of stack frames you just have dynamically allocated callback state?
- pcwalton 10y ago> Are libmill and Go's approach the same? I.e. Holding concurrent state in multiple stack frames? What causes that to have degraded performance? In the M:N approach you have to allocate stack space for each goroutine that you spawn. This requires that you either know the size of the stack up front (generally not possible without being conservative and requesting a large allocation) or that you start small and grow (resulting in a lot of memory traffic and pauses in the growth case, and much harder to do in C). By contrast, with the zero-cost futures approach we statically know exactly how much per-goroutine size we will ever need, and we can allocate precisely that amount. Furthermore, we only save the data that's absolutely needed across blocking calls. This results in much smaller per-connection state, and as a result it's quicker to allocate. It's the difference between static and dynamic control flow. Full M:N requires us to give up static knowledge of what a goroutine will do and try to do the best we can at runtime. With futures, we have a lot more static knowledge, and as a result we can optimize more aggressively.
- int_19h 10y agoDealing with futures is fairly straightforward if the language offers some syntactic sugar for continuations (such as C#'s async/await that has been spreading around for some time now). Why doesn't it mix well with complex business rules?
- davidgrenier 10y agoI'm not sure I would go as far as saying they can't. I'm quite comfortable with F#'s async computation expression and like them a lot. You get a strange sense however when playing with Hopac (which btw has an analog sugar to F#'s async which works just fine) and going through the content in Reppy's book that the model is strictly more composable. I'd invite you to try it although I don't have a sense for the C# combinators.