10 ms·

Tokio internals: Understanding Rust's async I/O framework

110 points by bowyakka 9y ago

jbirer 9y ago
I prefer golang's syscalls and os modules. Just direct interfaces to the C functions, no abstractions and no cruft.
- wmf 9y ago
  That's not idiomatic Go, is it? Go also abstracts I/O by converting it to async under the hood.
- virgilp 9y ago
  I'm not familiar enough with golang, but from your description it seems like in the Rust world, you're saying "I prefer the mio crate[1]" [1] https://docs.rs/mio/0.6.10/mio/ https://docs.rs/mio/0.6.10/mio/
  - bryanlarsen 9y ago
    AFAICT, a more go-like approach would be to use std::fs inside a std::thread
    - tatterdemalion 9y ago
      What's missing here is that Go has a userspace scheduler and greenthreads. Despite presenting the user with something that looks like old school blocking IO, because you are only blocking a greenthread it is actually asynchronous IO under the hood. And of course there is a huge amount of abstraction to build this up. It's just all hidden in the language runtime, so people can say with a straight face that there is "no abstraction and no cruft."
- jcrites 9y ago
  My understanding is that Go programs typically tackle these problems with goroutines and channels which are definitely an abstraction over kernel functionality.
- vvanders 9y ago
  Does golang use io completion on Windows? I know Tokio does so if that's not the case then there's probably a pretty large delta in performance.
  - ori_b 9y ago
    It does: https://golang.org/src/net/fd_windows.go https://golang.org/src/net/fd_windows.go
- leshow 9y ago
  Isn't go's greenthread offering part of it's runtime? If you use a go channel or thread, is that not calling some go std lib abstraction? What you're talking about seems like a pretty non-idiomatic use of go.
- JoshTriplett 9y ago
  That's still an abstraction, just one built into the language. And Go's approach (as well as other things, like garbage collection) requires a runtime; while that does support an interesting programming model, it also makes Go unusable for a variety of use cases that need to run without a runtime. This approach in Rust, as well as Rust's approach to memory management and other things, allow it to run without a runtime, which allows it to work for anything C does.
  - tatterdemalion 9y ago
    There's another reason Rust's ecosystem uses futures (as opposed to having some library-based greenthreading system): each future is like the stack of a userspace thread, but perfectly sized (it will be as large as the largest stack space needed at a yield point). This reduces the memory footprint of services using futures.
  - cshenton 9y ago
    What use cases need no runtime (genuine question)? I understand the benefit of being compiled, since you can target a cpu instruction set and run without dependencies.
    - JoshTriplett 9y ago
      Low-level firmware, embedded applications, OS kernels, interrupt service routines, applications that can't afford the latency of being periodically interrupted, platforms where you don't have a scheduler or most OS services, libraries intended to be loaded into other applications written in other languages (e.g. where you don't control the main program entry point), writing language runtimes themselves, etc. Early Rust, pre-1.0, did have a runtime and a green-threads mechanism; they ripped it out because they recognized that they couldn't go everywhere they wanted to go if they kept it. And if they hadn't done that, I believe Rust would have been far less successful than it has been.
      - cshenton 9y ago
        So I'm guessing it's the runtime itself that expects to interact with things like an OS scheduler. Does that mean if you attempted to write an OS in Go, you'd basically have to do it without many of the features of the runtime (that provide the abstractions that are useful for application progamming?). That would be pretty crummy.
- lossolo 9y ago
  For all asking what OP is talking about, this is example of using epoll directly in Go: https://gist.github.com/tevino/3a4f4ec4ea9d0ca66d4f https://gist.github.com/tevino/3a4f4ec4ea9d0ca66d4f This is description (from dotGo2017) of how netpoll works in Go under the hood if anyone is interested: https://www.youtube.com/watch?v=xwlo3xigknI https://www.youtube.com/watch?v=xwlo3xigknI
  - fpoling 9y ago
    I am not sure what exactly that gist tries to show. As it forces the go runtime to run each read call on a separated native thread, it just shows that one can use go to implement rather bad epoll antipattern.
- pcwalton 9y ago
  Go's I/O is anything but a direct interface to the C functions. Its scheduler is a large abstraction.
- deleted 9y ago
  [deleted]
- lkjalskdjflasdf 9y ago
  Go IO is implemented with green threads using a scheduler underneath. Rust had green threading in the initial phases but this was voted out.
kindfellow92 9y ago
> Unfortunately, Tokio is notoriously difficult to learn due to its sophisticated abstractions. Aren’t abstractions supposed to make things easier to learn? Something about the idea of “complex abstractions” seems wrong. (Edit: this is not a criticism of Tokio, it’s a criticism of the OP’s characterization of “sophisticated abstractions” which IMO should reduce complexity)
- curun1r 9y ago
  It depends on whether an abstraction is intended for novices or, for lack of a better term, power users. Tokio is, IMHO, more of the latter. It's a complex set of concepts that is designed to scale up very well as the complexity of the task increases but doesn't scale down very well for simple tasks* . Learning quite often involves those simple tasks, so Tokio gets a reputation for being hard to learn. But when you take an easy-to-learn abstraction and try to scale it up to handle very complex problems, you often find the abstraction breaks down much more easily than something like Tokio does. Tokio doesn't subscribe the the Larry Wall philosophy of making the easy things easy and the hard things possible. It seems more focused on making the hard things as easy as possible without much regard for the easy things. * Before anyone attacks this...yes, you can accomplish simple tasks in Tokio, but it requires learning a lot more concepts than should be necessary to accomplish that simple task.
  - lurr 9y ago
    So what, if you aren't already an expert then go away you aren't wanted?
- tatterdemalion 9y ago
  There are various criticisms of tokio, coming from different directions. Some have to do with the fact that some abstractions in the futures ecosystem are leaky today and that makes them less easy to use than they could be (though they won't always be leaky[]). But others have to do with understanding the internal implementation of these abstractions - people who feel they must understand how their library works internally before using it. Of course, schedulers are just complicated. Most of the time you don't think about how complicated your scheduler is, because its either an OS primitive in the kernel or a language primitive in your language's runtime. But since tokio is a library - and modular - it gets criticism for being complex that in my opinion is unfair. [] To be more concrete: a future is essentially a state machine representing the stack state at any yield point; it can't (currently) contain lightweight references into itself because they'd be invalidated when the future is move around. This means using borrowing in futures programs is often infeasible today. Solutions are in the works.
Const-me 9y ago
> The tokio-core crate provides the central event loop Does it mean it only uses a single thread for IO notifications? If yes, the performance won’t be exceptionally great, especially on servers with many CPU cores and fast network interfaces. The underlying OS APIs (both epoll, kqueue and iocp) do support multithreaded asynchronous IO, so that’s not some platform limitation.
- carllerche 9y ago
  One can spawn as many reactors as you would like. The only thing a reactor does is receive events off of epoll (or other system selector) and notify the associated task. The task could be on the reactor thread or across a different thread. Generally speaking, how to optimize concurrency for a network based application is pretty use case specific. tl;dr, you can fully take advantage of many core systems w/ Tokio.
  - br1 9y ago
    We don't want to have to decide which thread will handle each connection, just pick an idle one.
    - carllerche 9y ago
      Yes you can do this. The actual socket is not pinned to any thread, so you can move it to a thread pool or whatever.
  - Const-me 9y ago
    High I/O systems usually spawn multiple reactors, e.g. one reactor per CPU core, and run these reactors on the same set of file descriptors. Does that library support such use case? Or does it imply 1-to-many relation between reactors and files/sockets? The latter doesn’t scale well.
    - carllerche 9y ago
      Tokio is about being flexible (true today, even more true in the upcoming release). It is more about set of primitives that you can assemble in a way that fits your needs. You can structure the concurrency of your application however you want. So... > High I/O systems usually spawn multiple reactors, e.g. one reactor per CPU core. Depends on what you are calling multiple reactors. If you mean a loop that responds to events and run tasks, then yes. For example, you can plug in [this](http://github.com/carllerche/futures-pool http://github.com/carllerche/futures-pool) as the task executor and get a multi threaded, work stealing, scheduler. Or, maybe you are talking about OS level selectors (epoll), in which case you are going to run up against OS limitations.
carllerche 9y ago
I'm (one of) the author of Tokio, hopefully I can clarify some points. > Unfortunately, Tokio is notoriously difficult to learn due to its sophisticated abstractions. IMO, this is largely due to the current state of the docs (which are going to be rewritten as soon as some API changes land). The docs were written at a point where we were still trying to figure out how to present Tokio, and they ended up focusing on the wrong things. The Tokio docs currently focus on a very high level concept (`Service`) which is an RPC like abstraction similar to finagle. The problem is that, Tokio also includes a novel runtime model and future system and the docs don't spend any time explaining this. The next iteration of Tokio's docs is going to focus entirely at the "tokio-core" level, which is the reactor, runtime model, and TCP streams. tl;dr, I think the main reason people have trouble learning Tokio is because the current state of the docs are terrible. > Aren’t abstractions supposed to make things easier to learn? Tokio's goal is to provide as ergonomic abstractions as possible without adding runtime overhead. Tokio will never be as "easy" as high level runtimes simply because we don't accept the overhead that comes with them. The abstractions are also structured to help you avoid a lot of errors that tend to be introduced in asynchronous applications. For example, Tokio doesn't add any implicit buffering anywhere. A lot of other async libraries hide difficult details by adding unlimited buffering layers.
olix0r 9y ago
I've written a few production-facing applications with Tokio; and I think the author exactly identifies the stumbling blocks I hit while learning the landscape. My takeaway from working with Tokio is that it's a fairly low-level abstraction and doesn't do much to address the challenges of building networked _applications_. And this is OK. We'll need higher-level layers that use Tokio, however, to address more specific use cases. I'll point to the nascent tower-grpc[1] library as something in this direction. I hope to see more things like this fall out of our work on Conduit[2]. [1] https://github.com/tower-rs/tower-grpc https://github.com/tower-rs/tower-grpc [2] https://github.com/runconduit/conduit https://github.com/runconduit/conduit

110 points by bowyakka 9y ago

jbirer 9y ago

I prefer golang's syscalls and os modules. Just direct interfaces to the C functions, no abstractions and no cruft.

wmf 9y ago
That's not idiomatic Go, is it? Go also abstracts I/O by converting it to async under the hood.
virgilp 9y ago
I'm not familiar enough with golang, but from your description it seems like in the Rust world, you're saying "I prefer the mio crate[1]" [1] https://docs.rs/mio/0.6.10/mio/ https://docs.rs/mio/0.6.10/mio/
- bryanlarsen 9y ago
  AFAICT, a more go-like approach would be to use std::fs inside a std::thread
  - tatterdemalion 9y ago
    What's missing here is that Go has a userspace scheduler and greenthreads. Despite presenting the user with something that looks like old school blocking IO, because you are only blocking a greenthread it is actually asynchronous IO under the hood. And of course there is a huge amount of abstraction to build this up. It's just all hidden in the language runtime, so people can say with a straight face that there is "no abstraction and no cruft."
jcrites 9y ago
My understanding is that Go programs typically tackle these problems with goroutines and channels which are definitely an abstraction over kernel functionality.
vvanders 9y ago
Does golang use io completion on Windows? I know Tokio does so if that's not the case then there's probably a pretty large delta in performance.
- ori_b 9y ago
  It does: https://golang.org/src/net/fd_windows.go https://golang.org/src/net/fd_windows.go
leshow 9y ago
Isn't go's greenthread offering part of it's runtime? If you use a go channel or thread, is that not calling some go std lib abstraction? What you're talking about seems like a pretty non-idiomatic use of go.
JoshTriplett 9y ago
That's still an abstraction, just one built into the language. And Go's approach (as well as other things, like garbage collection) requires a runtime; while that does support an interesting programming model, it also makes Go unusable for a variety of use cases that need to run without a runtime. This approach in Rust, as well as Rust's approach to memory management and other things, allow it to run without a runtime, which allows it to work for anything C does.
- tatterdemalion 9y ago
  There's another reason Rust's ecosystem uses futures (as opposed to having some library-based greenthreading system): each future is like the stack of a userspace thread, but perfectly sized (it will be as large as the largest stack space needed at a yield point). This reduces the memory footprint of services using futures.
- cshenton 9y ago
  What use cases need no runtime (genuine question)? I understand the benefit of being compiled, since you can target a cpu instruction set and run without dependencies.
  - JoshTriplett 9y ago
    Low-level firmware, embedded applications, OS kernels, interrupt service routines, applications that can't afford the latency of being periodically interrupted, platforms where you don't have a scheduler or most OS services, libraries intended to be loaded into other applications written in other languages (e.g. where you don't control the main program entry point), writing language runtimes themselves, etc. Early Rust, pre-1.0, did have a runtime and a green-threads mechanism; they ripped it out because they recognized that they couldn't go everywhere they wanted to go if they kept it. And if they hadn't done that, I believe Rust would have been far less successful than it has been.
    - cshenton 9y ago
      So I'm guessing it's the runtime itself that expects to interact with things like an OS scheduler. Does that mean if you attempted to write an OS in Go, you'd basically have to do it without many of the features of the runtime (that provide the abstractions that are useful for application progamming?). That would be pretty crummy.
lossolo 9y ago
For all asking what OP is talking about, this is example of using epoll directly in Go: https://gist.github.com/tevino/3a4f4ec4ea9d0ca66d4f https://gist.github.com/tevino/3a4f4ec4ea9d0ca66d4f This is description (from dotGo2017) of how netpoll works in Go under the hood if anyone is interested: https://www.youtube.com/watch?v=xwlo3xigknI https://www.youtube.com/watch?v=xwlo3xigknI
- fpoling 9y ago
  I am not sure what exactly that gist tries to show. As it forces the go runtime to run each read call on a separated native thread, it just shows that one can use go to implement rather bad epoll antipattern.
pcwalton 9y ago
Go's I/O is anything but a direct interface to the C functions. Its scheduler is a large abstraction.
deleted 9y ago
[deleted]
lkjalskdjflasdf 9y ago
Go IO is implemented with green threads using a scheduler underneath. Rust had green threading in the initial phases but this was voted out.

kindfellow92 9y ago

> Unfortunately, Tokio is notoriously difficult to learn due to its sophisticated abstractions. Aren’t abstractions supposed to make things easier to learn? Something about the idea of “complex abstractions” seems wrong. (Edit: this is not a criticism of Tokio, it’s a criticism of the OP’s characterization of “sophisticated abstractions” which IMO should reduce complexity)

curun1r 9y ago
It depends on whether an abstraction is intended for novices or, for lack of a better term, power users. Tokio is, IMHO, more of the latter. It's a complex set of concepts that is designed to scale up very well as the complexity of the task increases but doesn't scale down very well for simple tasks* . Learning quite often involves those simple tasks, so Tokio gets a reputation for being hard to learn. But when you take an easy-to-learn abstraction and try to scale it up to handle very complex problems, you often find the abstraction breaks down much more easily than something like Tokio does. Tokio doesn't subscribe the the Larry Wall philosophy of making the easy things easy and the hard things possible. It seems more focused on making the hard things as easy as possible without much regard for the easy things. * Before anyone attacks this...yes, you can accomplish simple tasks in Tokio, but it requires learning a lot more concepts than should be necessary to accomplish that simple task.
- lurr 9y ago
  So what, if you aren't already an expert then go away you aren't wanted?
tatterdemalion 9y ago
There are various criticisms of tokio, coming from different directions. Some have to do with the fact that some abstractions in the futures ecosystem are leaky today and that makes them less easy to use than they could be (though they won't always be leaky[]). But others have to do with understanding the internal implementation of these abstractions - people who feel they must understand how their library works internally before using it. Of course, schedulers are just complicated. Most of the time you don't think about how complicated your scheduler is, because its either an OS primitive in the kernel or a language primitive in your language's runtime. But since tokio is a library - and modular - it gets criticism for being complex that in my opinion is unfair. [] To be more concrete: a future is essentially a state machine representing the stack state at any yield point; it can't (currently) contain lightweight references into itself because they'd be invalidated when the future is move around. This means using borrowing in futures programs is often infeasible today. Solutions are in the works.

Const-me 9y ago

> The tokio-core crate provides the central event loop Does it mean it only uses a single thread for IO notifications? If yes, the performance won’t be exceptionally great, especially on servers with many CPU cores and fast network interfaces. The underlying OS APIs (both epoll, kqueue and iocp) do support multithreaded asynchronous IO, so that’s not some platform limitation.

carllerche 9y ago
One can spawn as many reactors as you would like. The only thing a reactor does is receive events off of epoll (or other system selector) and notify the associated task. The task could be on the reactor thread or across a different thread. Generally speaking, how to optimize concurrency for a network based application is pretty use case specific. tl;dr, you can fully take advantage of many core systems w/ Tokio.
- br1 9y ago
  We don't want to have to decide which thread will handle each connection, just pick an idle one.
  - carllerche 9y ago
    Yes you can do this. The actual socket is not pinned to any thread, so you can move it to a thread pool or whatever.
- Const-me 9y ago
  High I/O systems usually spawn multiple reactors, e.g. one reactor per CPU core, and run these reactors on the same set of file descriptors. Does that library support such use case? Or does it imply 1-to-many relation between reactors and files/sockets? The latter doesn’t scale well.
  - carllerche 9y ago
    Tokio is about being flexible (true today, even more true in the upcoming release). It is more about set of primitives that you can assemble in a way that fits your needs. You can structure the concurrency of your application however you want. So... > High I/O systems usually spawn multiple reactors, e.g. one reactor per CPU core. Depends on what you are calling multiple reactors. If you mean a loop that responds to events and run tasks, then yes. For example, you can plug in [this](http://github.com/carllerche/futures-pool http://github.com/carllerche/futures-pool) as the task executor and get a multi threaded, work stealing, scheduler. Or, maybe you are talking about OS level selectors (epoll), in which case you are going to run up against OS limitations.

carllerche 9y ago

I'm (one of) the author of Tokio, hopefully I can clarify some points. > Unfortunately, Tokio is notoriously difficult to learn due to its sophisticated abstractions. IMO, this is largely due to the current state of the docs (which are going to be rewritten as soon as some API changes land). The docs were written at a point where we were still trying to figure out how to present Tokio, and they ended up focusing on the wrong things. The Tokio docs currently focus on a very high level concept (`Service`) which is an RPC like abstraction similar to finagle. The problem is that, Tokio also includes a novel runtime model and future system and the docs don't spend any time explaining this. The next iteration of Tokio's docs is going to focus entirely at the "tokio-core" level, which is the reactor, runtime model, and TCP streams. tl;dr, I think the main reason people have trouble learning Tokio is because the current state of the docs are terrible. > Aren’t abstractions supposed to make things easier to learn? Tokio's goal is to provide as ergonomic abstractions as possible without adding runtime overhead. Tokio will never be as "easy" as high level runtimes simply because we don't accept the overhead that comes with them. The abstractions are also structured to help you avoid a lot of errors that tend to be introduced in asynchronous applications. For example, Tokio doesn't add any implicit buffering anywhere. A lot of other async libraries hide difficult details by adding unlimited buffering layers.

olix0r 9y ago

I've written a few production-facing applications with Tokio; and I think the author exactly identifies the stumbling blocks I hit while learning the landscape. My takeaway from working with Tokio is that it's a fairly low-level abstraction and doesn't do much to address the challenges of building networked _applications_. And this is OK. We'll need higher-level layers that use Tokio, however, to address more specific use cases. I'll point to the nascent tower-grpc[1] library as something in this direction. I hope to see more things like this fall out of our work on Conduit[2]. [1] https://github.com/tower-rs/tower-grpc https://github.com/tower-rs/tower-grpc [2] https://github.com/runconduit/conduit https://github.com/runconduit/conduit