6 ms·

It always struck me as odd that Go has select{}, but no way to select on a file descriptor such a socket. There's literally no way to poll a file descriptor: Yo

It always struck me as odd that Go has select{}, but no way to select on a file descriptor such a socket. There's literally no way to poll a file descriptor: You have to read from it, and deal with the result. That also means reads aren't interruptible (unless you count closing the descriptor as interrupting, which is a blunt hammer indeed). AFAIK the only way to do this is to set SetReadDeadline() on the connection to a smallish value, then let the interrupter block on a channel that's invoked when the deadline is reached. Go's concurrency model is pretty friendly, but it's also unfriendly in many surprising ways. (Don't get me started on the whole nil channel thing, or the problem of channel closure/ownership, or channel performance overall.) It doesn't surprise me the least (while it did surprise the Go team) that Go fell flat among the C/C++ system programmer demographic. Anyone want to chip in about how Rust, Swift and Nim compare here (regarding the article)?

nothrabannosir 10y ago
> You have to read from it, and deal with the result. They try to keep it simple. As with all I/O: it's blocking your current goroutine, so if you want it non-blocking you spawn a new goroutine to do this in, pass it a channel, the spawned routine proxies the result to it and your main routine does select on that. The semantics for reading sockets are different from reading channels (compare the types; one is multi valued, the other single valued---EDIT: no they're not, exactly, see [1]), so it kind of makes sense you can't share the same construct. I'm not a Go apologist by any means, I think it has a tremendous amount of flaws. But I/O, (a)synchronicity and concurrency are not among them. They are simple and consistent, and (most importantly) orthogonal building blocks which you can combine at your leisure to create more complex constructs. (Besides, how do you interrupt a read, anyway? Or any syscall? You can't do that in C, either...?) [1] edit: fair is fair, go channel reads are (of course) also multi valued. however, the types still don't exactly match, since you can't pass arguments (buffer) to a channel read. you'd need some extra hoops to jump through, as other commenters suggest, which starts becoming magic and breaking down the simplicity and orthogonality of the constructs.
- lobster_johnson 10y ago
  Sorry for being unclear. You don't interrupt the read as such. You interrupt the select. The usual trick on POSIX is to use a self-pipe, since select()/epoll() only work on file descriptors and not, say, pthread primitives. Windows is slightly better here: WaitForMultipleObjects supports waiting on any primitive that supports blocking, so you wait on the socket + on an event object, and signaling the event object will unblock the wait. Windows also has completion ports, and the APIs are generally more extensive (not a fan of Windows, but they got this part very right). I do understand why Go is the way it is. But it's important to recognize that it's a compromise in design that inherently reduces your flexibility — such as in this particular case. I don't think there's anything technically preventing Go from supporting non-blocking I/O, though. Channels and FDs/sockets are both conceptually multi-valued; what did you mean by that? One major difference is that reads can fail with an error, and channel reads can't. Edit: Also, I didn't mean that select{} on a file descriptor ought to actually return data. It would be more appropriate for it to yield a true/false value to signal that the FD has data, so you'd do "case <-fd: fd.Read(bufSize)".
  - nothrabannosir 10y ago
    Agree with everything. Just to reply to this: > Channels and FDs/sockets are both conceptually multi-valued; what did you mean by that? I was trying to preempt pedantry, but ended up being the pedant myself :P "You can simulate multi valued responses in a channel by passing a struct."
  - mentat 10y ago
    You can select on the read and an time.After channel which "interrupts" the select for a timeout.
    - lobster_johnson 10y ago
      You can't select on a read, though. select{} only works on channels. You can run the read in a separate goroutine, but that doesn't stop the read from happening.
steveklabnik 10y ago
So, soon I will have some stuff to point you to with regards to how Rust is doing it, but we're not _quite_ there, so as a summary... With Tokio, you write non-blocking IO stuff as "futures", similarly to promises in JS. These then get compiled down to what's effectively a state machine, and they run in an event loop on top of epoll/kqueue. That event loop is single threaded, and fully non-blocking. I am not a super, super expert on these low-level details, but I believe that this should mean that it is closer to the C code than the Go code.
- lobster_johnson 10y ago
  Interesting solution! I like futures, though cancelation has historically been a controversial topic there, too (at least in the JavaScript world; JS promises are not cancelable, which is problematic for HTTP requests [1]). I do like that you guys chose to not make green threads a language-level construct (as Go did with "go" and goroutines), which means you have the option of supporting multiple models, for the programmer to choose according to their particular use case. [1] https://github.com/whatwg/fetch/issues/27 https://github.com/whatwg/fetch/issues/27
  - steveklabnik 10y ago
    Cancellation falls out of the model: you drop the future, and it Just Works. (drop being the regular old Rust destructor stuff, which happens when it goes out of scope, or when you call std::mem::drop.) Yup! The danger there is that the ecosystem may fragment, however, basically everyone is rallying around tokio, including the other people who were working on similar libraries, so in practice, this should be fine.
- dom96 10y ago
  This sounds similar to what we have in Nim right now. There are multiple layers (from highest to lowest): * Async await * The async dispatcher (event loop) * Selectors module Each builds on top of the layers below it. The selectors module is an abstraction for the various system `select()` mechanism such as epoll or kqueue. If you wish you can use it directly, and some people have done so successfully[1]. But you must be aware that on Windows it only supports `select()`. The async dispatcher implements the proactor pattern and also has an implementation of IOCP. It implements the proactor pattern for other OS' via the selectors module. Futures are also implemented here, and indeed all the procedures return a future. The final layer is the async await transformation. This allows you to write asynchronous code in a synchronous style and works similar to C#'s await. The transformation works by converting any async procedures into an iterator. This is also single threaded and fully non-blocking and does give you a lot of control. In theory you should be able to work with any of these layers interchangeably (as long as you only care about non-Windows platforms). 1 - http://goran.krampe.se/2014/10/25/nim-socketserver/ http://goran.krampe.se/2014/10/25/nim-socketserver/
eternalban 10y ago
> there is literally no way to poll a file descriptor https://godoc.org/golang.org/x/sys https://godoc.org/golang.org/x/sys https://godoc.org/golang.org/x/sys/unix#EpollWait https://godoc.org/golang.org/x/sys/unix#EpollWait https://golang.org/pkg/syscall/ https://golang.org/pkg/syscall/ Of course it is a bit of a pain to write OS specific variants if you are building a completely crossplatform solution, but it is supported out of the box: https://dave.cheney.net/2013/10/12/how-to-use-conditional-compilation-with-the-go-build-tool https://dave.cheney.net/2013/10/12/how-to-use-conditional-co... That said, I agree [regarding the] puzzling decision to not to treat 'select' as a general contract for all asynchronous ops as that fits perfectly with the CSP approach.
- lobster_johnson 10y ago
  Indeed, though I was referring to select{}, i.e. Go's first-class language constructs.
sedachv 10y ago
I still think that John Reppy's Concurrent ML design is the best thought out way to do non-blocking IO and event handling. There has been a lot of progress on CML but Reppy's 1992 PhD dissertation remains a very readable introduction: http://people.cs.uchicago.edu/~jhr/papers/1992/phd-thesis.html http://people.cs.uchicago.edu/~jhr/papers/1992/phd-thesis.ht...

nothrabannosir 10y ago

> You have to read from it, and deal with the result. They try to keep it simple. As with all I/O: it's blocking your current goroutine, so if you want it non-blocking you spawn a new goroutine to do this in, pass it a channel, the spawned routine proxies the result to it and your main routine does select on that. The semantics for reading sockets are different from reading channels (compare the types; one is multi valued, the other single valued---EDIT: no they're not, exactly, see [1]), so it kind of makes sense you can't share the same construct. I'm not a Go apologist by any means, I think it has a tremendous amount of flaws. But I/O, (a)synchronicity and concurrency are not among them. They are simple and consistent, and (most importantly) orthogonal building blocks which you can combine at your leisure to create more complex constructs. (Besides, how do you interrupt a read, anyway? Or any syscall? You can't do that in C, either...?) [1] edit: fair is fair, go channel reads are (of course) also multi valued. however, the types still don't exactly match, since you can't pass arguments (buffer) to a channel read. you'd need some extra hoops to jump through, as other commenters suggest, which starts becoming magic and breaking down the simplicity and orthogonality of the constructs.

lobster_johnson 10y ago
Sorry for being unclear. You don't interrupt the read as such. You interrupt the select. The usual trick on POSIX is to use a self-pipe, since select()/epoll() only work on file descriptors and not, say, pthread primitives. Windows is slightly better here: WaitForMultipleObjects supports waiting on any primitive that supports blocking, so you wait on the socket + on an event object, and signaling the event object will unblock the wait. Windows also has completion ports, and the APIs are generally more extensive (not a fan of Windows, but they got this part very right). I do understand why Go is the way it is. But it's important to recognize that it's a compromise in design that inherently reduces your flexibility — such as in this particular case. I don't think there's anything technically preventing Go from supporting non-blocking I/O, though. Channels and FDs/sockets are both conceptually multi-valued; what did you mean by that? One major difference is that reads can fail with an error, and channel reads can't. Edit: Also, I didn't mean that select{} on a file descriptor ought to actually return data. It would be more appropriate for it to yield a true/false value to signal that the FD has data, so you'd do "case <-fd: fd.Read(bufSize)".
- nothrabannosir 10y ago
  Agree with everything. Just to reply to this: > Channels and FDs/sockets are both conceptually multi-valued; what did you mean by that? I was trying to preempt pedantry, but ended up being the pedant myself :P "You can simulate multi valued responses in a channel by passing a struct."
- mentat 10y ago
  You can select on the read and an time.After channel which "interrupts" the select for a timeout.
  - lobster_johnson 10y ago
    You can't select on a read, though. select{} only works on channels. You can run the read in a separate goroutine, but that doesn't stop the read from happening.

steveklabnik 10y ago

So, soon I will have some stuff to point you to with regards to how Rust is doing it, but we're not _quite_ there, so as a summary... With Tokio, you write non-blocking IO stuff as "futures", similarly to promises in JS. These then get compiled down to what's effectively a state machine, and they run in an event loop on top of epoll/kqueue. That event loop is single threaded, and fully non-blocking. I am not a super, super expert on these low-level details, but I believe that this should mean that it is closer to the C code than the Go code.

lobster_johnson 10y ago
Interesting solution! I like futures, though cancelation has historically been a controversial topic there, too (at least in the JavaScript world; JS promises are not cancelable, which is problematic for HTTP requests [1]). I do like that you guys chose to not make green threads a language-level construct (as Go did with "go" and goroutines), which means you have the option of supporting multiple models, for the programmer to choose according to their particular use case. [1] https://github.com/whatwg/fetch/issues/27 https://github.com/whatwg/fetch/issues/27
- steveklabnik 10y ago
  Cancellation falls out of the model: you drop the future, and it Just Works. (drop being the regular old Rust destructor stuff, which happens when it goes out of scope, or when you call std::mem::drop.) Yup! The danger there is that the ecosystem may fragment, however, basically everyone is rallying around tokio, including the other people who were working on similar libraries, so in practice, this should be fine.
dom96 10y ago
This sounds similar to what we have in Nim right now. There are multiple layers (from highest to lowest): * Async await * The async dispatcher (event loop) * Selectors module Each builds on top of the layers below it. The selectors module is an abstraction for the various system `select()` mechanism such as epoll or kqueue. If you wish you can use it directly, and some people have done so successfully[1]. But you must be aware that on Windows it only supports `select()`. The async dispatcher implements the proactor pattern and also has an implementation of IOCP. It implements the proactor pattern for other OS' via the selectors module. Futures are also implemented here, and indeed all the procedures return a future. The final layer is the async await transformation. This allows you to write asynchronous code in a synchronous style and works similar to C#'s await. The transformation works by converting any async procedures into an iterator. This is also single threaded and fully non-blocking and does give you a lot of control. In theory you should be able to work with any of these layers interchangeably (as long as you only care about non-Windows platforms). 1 - http://goran.krampe.se/2014/10/25/nim-socketserver/ http://goran.krampe.se/2014/10/25/nim-socketserver/

eternalban 10y ago

> there is literally no way to poll a file descriptor https://godoc.org/golang.org/x/sys https://godoc.org/golang.org/x/sys https://godoc.org/golang.org/x/sys/unix#EpollWait https://godoc.org/golang.org/x/sys/unix#EpollWait https://golang.org/pkg/syscall/ https://golang.org/pkg/syscall/ Of course it is a bit of a pain to write OS specific variants if you are building a completely crossplatform solution, but it is supported out of the box: https://dave.cheney.net/2013/10/12/how-to-use-conditional-compilation-with-the-go-build-tool https://dave.cheney.net/2013/10/12/how-to-use-conditional-co... That said, I agree [regarding the] puzzling decision to not to treat 'select' as a general contract for all asynchronous ops as that fits perfectly with the CSP approach.

lobster_johnson 10y ago
Indeed, though I was referring to select{}, i.e. Go's first-class language constructs.

sedachv 10y ago

I still think that John Reppy's Concurrent ML design is the best thought out way to do non-blocking IO and event handling. There has been a lot of progress on CML but Reppy's 1992 PhD dissertation remains a very readable introduction: http://people.cs.uchicago.edu/~jhr/papers/1992/phd-thesis.html http://people.cs.uchicago.edu/~jhr/papers/1992/phd-thesis.ht...