3 ms·

The design used one dedicated thread per connection with blocking I/O. On Windows, that type of design is very slow. I took a few weeks out and rewrote the Refl

by blasdel 18y ago

The design used one dedicated thread per connection with blocking I/O. On Windows, that type of design is very slow. I took a few weeks out and rewrote the Reflector to use overlapping I/O with thread pools. How'd you become enlightened as to the terribleness of your original design? From what I can tell "overlapping I/O" appears to be windows-ese for good old poll(2) -- non-blocking asynchronous I/O with level-triggered notification. Does Windows not have a way to do edge-triggered notifications like Linux's epoll(7)? Also, is there some incentive on Windows for using a thread pool instead of a handful of independent processes per box? Realistically you'll have the server running on multiple machines just for sanity's sake, and you'll have a way to distribute clients across them with failover, so why have two levels of indirection when one will do?

gecko 18y ago
Er, I'm not sure what you were reading, but Windows has a poll function. It's called poll. Overlapped I/O is the edge-trigged notifications you want, and correspond to epoll/kqueues. You make a function call, which returns immediately. You then get a callback (either literally, if you're on .NET, or as a Windows event otherwise) when data is available. Overlapped I/O automatically uses the Windows thread pools, so making my own wouldn't make sense.
- barrkel 18y ago
  Native Windows overlapped I/O can give you a callback without needing to use an event: WriteFileEx (or WSASend) accepts a callback routine, etc. Also, the OVERLAPPED structure requires you to provide the event handle, and it's optional, so you don't need to use that kernel resource if you don't want to. The "poll" function you speak of - I presume you mean Socket.Poll - is spelled "select" in native code, and is equivalent to the BSD sockets version.
  - gecko 18y ago
    We're getting a bit knit-picky, but yes, you can get a callback directly--which is in fact what we're doing. Vista and later (which includes Windows 2k8 Server) actually introduces a proper WSAPoll function, so you don't need to use the godawful select() anymore unless being masochistic really is just your thing.
barrkel 18y ago
FWIW, it's not obvious that the one-thread one-connection blocking I/O design is terrible. How terrible it is is a function of address-space starvation from too many threads (a non-issue in 64-bit code), and the overhead of the OS thread scheduler. In a green threading model, I/O can be implemented behind the scenes as non-blocking I/O, but still give user code the ease of use of blocking I/O. Back to Windows: overlapped I/O is not equivalent to poll, it's much better than that. It is edge-triggered, but Windows manages the threads itself, and tries to ensure 100% CPU utilization in user code, rather than in thread switching logic. Windows overlapped I/O is effectively a way to write efficient I/O intensive applications in continuation-passing style. You don't need to manage the threads or dispatch loop yourself. As to why use threads rather than processes, the issue is largely the awkwardness of duplicating the connection handle for a child thread. In Posix land, there's the fork function that makes handing off the handle fairly easy. WSADuplicateHandle can create a new handle that can be used by a child process, but you need to know the child process's ID first, so you need to perform IPC to just to send off the handle. On the other hand, shared memory concurrency has its advantages too.
- blasdel 18y ago
  FWIW, it's not obvious that the one-thread one-connection blocking I/O design is terrible. Any time you're using a 'thread' idiom, even if they are green, something has to be maintaining their individual execution state (stack frames, thread-local address space, etc.). I like to think of it as being analagous to unoptimized tail-recursion, with a purely event-driven callback model being like tail-call optimization. Thanks for explaining Windows' overlapped I/O, it's way better than I thought it was from reading the documentation. I didn't realize that it does all the thread-pooling stuff all by itself -- I guess it makes the most sense not to fight it, and just accept the free utilization (even if it does give a slight latency hit).
  - barrkel 18y ago
    No - not tail call - event driven callback model I/O is continuation-passing style (CPS). The two models are equivalent. Code written in the thread idiom can, in principle, be mechanically transformed into CPS and thereby use the event-driven callback model. In the event callback style, the state that would otherwise be on the stack is held in the callback closure. In the .NET space, F# does this transformation automatically with what it calls asynchronous workflows (http://blogs.msdn.com/dsyme/archive/2007/10/11/introducing-f-asynchronous-workflows.aspx http://blogs.msdn.com/dsyme/archive/2007/10/11/introducing-f...). It uses the 'let!' assignment as a dividing point between the call to the asynchronous method and converting the remainder of the method body into a continuation that gets passed to the asynchronous call.

by blasdel 18y ago

gecko 18y ago

Er, I'm not sure what you were reading, but Windows has a poll function. It's called poll. Overlapped I/O is the edge-trigged notifications you want, and correspond to epoll/kqueues. You make a function call, which returns immediately. You then get a callback (either literally, if you're on .NET, or as a Windows event otherwise) when data is available. Overlapped I/O automatically uses the Windows thread pools, so making my own wouldn't make sense.

barrkel 18y ago
Native Windows overlapped I/O can give you a callback without needing to use an event: WriteFileEx (or WSASend) accepts a callback routine, etc. Also, the OVERLAPPED structure requires you to provide the event handle, and it's optional, so you don't need to use that kernel resource if you don't want to. The "poll" function you speak of - I presume you mean Socket.Poll - is spelled "select" in native code, and is equivalent to the BSD sockets version.
- gecko 18y ago
  We're getting a bit knit-picky, but yes, you can get a callback directly--which is in fact what we're doing. Vista and later (which includes Windows 2k8 Server) actually introduces a proper WSAPoll function, so you don't need to use the godawful select() anymore unless being masochistic really is just your thing.

barrkel 18y ago

FWIW, it's not obvious that the one-thread one-connection blocking I/O design is terrible. How terrible it is is a function of address-space starvation from too many threads (a non-issue in 64-bit code), and the overhead of the OS thread scheduler. In a green threading model, I/O can be implemented behind the scenes as non-blocking I/O, but still give user code the ease of use of blocking I/O. Back to Windows: overlapped I/O is not equivalent to poll, it's much better than that. It is edge-triggered, but Windows manages the threads itself, and tries to ensure 100% CPU utilization in user code, rather than in thread switching logic. Windows overlapped I/O is effectively a way to write efficient I/O intensive applications in continuation-passing style. You don't need to manage the threads or dispatch loop yourself. As to why use threads rather than processes, the issue is largely the awkwardness of duplicating the connection handle for a child thread. In Posix land, there's the fork function that makes handing off the handle fairly easy. WSADuplicateHandle can create a new handle that can be used by a child process, but you need to know the child process's ID first, so you need to perform IPC to just to send off the handle. On the other hand, shared memory concurrency has its advantages too.

blasdel 18y ago
FWIW, it's not obvious that the one-thread one-connection blocking I/O design is terrible. Any time you're using a 'thread' idiom, even if they are green, something has to be maintaining their individual execution state (stack frames, thread-local address space, etc.). I like to think of it as being analagous to unoptimized tail-recursion, with a purely event-driven callback model being like tail-call optimization. Thanks for explaining Windows' overlapped I/O, it's way better than I thought it was from reading the documentation. I didn't realize that it does all the thread-pooling stuff all by itself -- I guess it makes the most sense not to fight it, and just accept the free utilization (even if it does give a slight latency hit).
- barrkel 18y ago
  No - not tail call - event driven callback model I/O is continuation-passing style (CPS). The two models are equivalent. Code written in the thread idiom can, in principle, be mechanically transformed into CPS and thereby use the event-driven callback model. In the event callback style, the state that would otherwise be on the stack is held in the callback closure. In the .NET space, F# does this transformation automatically with what it calls asynchronous workflows (http://blogs.msdn.com/dsyme/archive/2007/10/11/introducing-f-asynchronous-workflows.aspx http://blogs.msdn.com/dsyme/archive/2007/10/11/introducing-f...). It uses the 'let!' assignment as a dividing point between the call to the asynchronous method and converting the remainder of the method body into a continuation that gets passed to the asynchronous call.