5 ms·

It's all about fixed resource allocations. The default stack size for user-land threads tends to be 1MB or 2MB, and there's also (smaller) kernel stacks. In i

It's all about fixed resource allocations. The default stack size for user-land threads tends to be 1MB or 2MB, and there's also (smaller) kernel stacks. In implementations of languages like Scheme or Haskell where no stack may even be allocated, or where stacks might be allocated in chunks, the cost of a co-routine can be very small -- as small as the cost of a closure. If you take that to the limit and allocate every call frame on the heap, and if you make heavy use of closures/continuations, then you end up with more GC pressure because instead of freeing every frame on return you have to free many of them via the GC. In terms of ease of programming to deal with async events, the spectrum runs from threads on the one hand to callback hell on the other. Callback hell can be ameliorated by allowing lambdas and closures, but you still end up with indentation and/or paren/brace hell. Co-routines are somewhere in the middle of the spectrum, but closer to threads than to callback hell. Await is something closer to callback hell with nice syntax to make things easier on the programmer. Ultimately it's all about how to represent state. In threaded programming program state is largely implicit in the call stack (including all the local variables). In callback hell program state is largely explicit in the form of the context argument that gets passed to the callback functions (or which they close over), with a little bit of state implicit in the extant event registrations. Threaded programming is easier because the programmer doesn't have to think about how to compactly represent program state, but the cost is higher resource consumption, especially memory. More memory consumption == more cache pressure == more cache misses == slower performance. Callback hell is much harder on the programmer because it forces the programmer to be much more explicit about program state, but this also allows the programmer to better compress program state, thus using fewer resources, thus allowing the program to deal with many more clients at once and also be faster than threaded programming. Everything in computer science in the async I/O space in the last 30 years has been about striking the right balance between minimizing program state on the one hand and minimizing programmer pain on the other. Continuations, delineated continuations, await, and so on -- all are about finding that sweet spot.

weberc2 8y ago
An important distinction between goroutines and typical coroutine implementations is that the Go compiler puts in the yield points for you, so goroutines are exactly as friendly as threads but with better performance.
- cryptonector 8y ago
  That's not new at all. This is known as cooperative multitasking, and you have the system define places where a co-routine switches to another one waiting for CPU. These points are usually those at which I/O is done. Not at all new.
  - weberc2 8y ago
    I didn’t claim it was new, I claimed it was different from typical coroutine implementations where you need to explicitly yield (all coroutines are cooperative multitasking by definition). I also claimed that this gives the usability properties of threads (you claimed that coroutines are somewhere between threads and callback hell WRT ease of use, which is true for traditional coroutines but not for goroutines. Another important distinction is that traditionally coroutines are not M:N, while goroutines are.
    - cryptonector 8y ago
      Traditional co-routines absolutely are M:N. A co-routine system where every co-routine is backed by a kernel thread is known as threading.
  - TimJYoung 8y ago
    I'm not sure what you mean by "system" here, but if the OS does it then the architecture is probably not cooperative, but rather preemptive. On the other hand, if the runtime (such as the Go runtime) causes a context switch between coroutines because one of the coroutines is waiting (on blocking I/O, etc.), then the architecture could still be considered cooperative because the scheduling is such that none of the coroutines would ever be actually preempted by the runtime while executing instructions. Of course, they could still be pre-empted by the OS because they're ultimately all running as system threads, which is one of the downsides of all this: eventually, it all comes back to threads, how many of them are present, and how the OS handles them.
TimJYoung 8y ago
Very good summary. I would only add that it pays to mention that thread/fiber stacks typically only reserve 1-2MB of VM, and only a small amount is committed initially (1 to 2 pages). With 32-bit address spaces, this is a minor distinction because the size of the address space is what matters the most, but with 64-bit address spaces, it can mean that you can aren't putting as much pressure on the VMM as one might think. If all of the threads in a process only use the default two pages, and the system page size is 4K, then 10,000 threads are only going to result in a commit size of 81MB of VM for the stacks. Of course, that's a lot of "ifs", and it completely ignores the effects of that many threads on the per-process VM mappings/CPU caches.
- cryptonector 8y ago
  I keep getting comments about this, but it's not that simple. First of all, unless you're overcommitting, those stacks consume more resources than one might think at first blush. Second, the apps I am familiar with (on the Java side) get pretty deep stacks (you can tell from the stack traces in exceptions!). Third, if you want millions of stacks on a 64-bit address-space, I'm thinking you'll want larger pages if much more than a handful of 4KB pages per-stack get used (as otherwise the page tables get large), and then you're consuming large amounts of RAM anyways. I may be wrong as to the third item, but I suspect the actual depth of the stacks as-used makes a big difference. The point is that thread-per-client is just never going to scale better than C10K no matter what, and that the work we see in this space is all about finding the right balance between the ease-of-programming of thread-per-client and the efficiency of callback hell (typical C10K). For any size computer (larger than tiny), C10K designs will wring out orders of magnitude better scaling than typical thread-per-client apps no matter how much you optimize the hardware for the latter. Thread-per-client can only possibly compare in the same ballpark as C10K when the actual stack space used is tiny and comparable to the amount of explicit state kept in the C10K alternative, and even then, the additional kernel resources consumed by those per-client threads will dwarf those needed for the C10K case (thread per-CPU), thus adding cache pressure. In practice, thread-per-client is never that efficient because the whole point of it is that it makes it trivial to employ layered libraries of synchronous APIs. Now, I'm not saying that the alternatives to thread-per-client are easy, but we should do a lot better at building new libraries so that async I/O can get easier.

weberc2 8y ago

An important distinction between goroutines and typical coroutine implementations is that the Go compiler puts in the yield points for you, so goroutines are exactly as friendly as threads but with better performance.

cryptonector 8y ago
That's not new at all. This is known as cooperative multitasking, and you have the system define places where a co-routine switches to another one waiting for CPU. These points are usually those at which I/O is done. Not at all new.
- weberc2 8y ago
  I didn’t claim it was new, I claimed it was different from typical coroutine implementations where you need to explicitly yield (all coroutines are cooperative multitasking by definition). I also claimed that this gives the usability properties of threads (you claimed that coroutines are somewhere between threads and callback hell WRT ease of use, which is true for traditional coroutines but not for goroutines. Another important distinction is that traditionally coroutines are not M:N, while goroutines are.
  - cryptonector 8y ago
    Traditional co-routines absolutely are M:N. A co-routine system where every co-routine is backed by a kernel thread is known as threading.
- TimJYoung 8y ago
  I'm not sure what you mean by "system" here, but if the OS does it then the architecture is probably not cooperative, but rather preemptive. On the other hand, if the runtime (such as the Go runtime) causes a context switch between coroutines because one of the coroutines is waiting (on blocking I/O, etc.), then the architecture could still be considered cooperative because the scheduling is such that none of the coroutines would ever be actually preempted by the runtime while executing instructions. Of course, they could still be pre-empted by the OS because they're ultimately all running as system threads, which is one of the downsides of all this: eventually, it all comes back to threads, how many of them are present, and how the OS handles them.

TimJYoung 8y ago

Very good summary. I would only add that it pays to mention that thread/fiber stacks typically only reserve 1-2MB of VM, and only a small amount is committed initially (1 to 2 pages). With 32-bit address spaces, this is a minor distinction because the size of the address space is what matters the most, but with 64-bit address spaces, it can mean that you can aren't putting as much pressure on the VMM as one might think. If all of the threads in a process only use the default two pages, and the system page size is 4K, then 10,000 threads are only going to result in a commit size of 81MB of VM for the stacks. Of course, that's a lot of "ifs", and it completely ignores the effects of that many threads on the per-process VM mappings/CPU caches.

cryptonector 8y ago
I keep getting comments about this, but it's not that simple. First of all, unless you're overcommitting, those stacks consume more resources than one might think at first blush. Second, the apps I am familiar with (on the Java side) get pretty deep stacks (you can tell from the stack traces in exceptions!). Third, if you want millions of stacks on a 64-bit address-space, I'm thinking you'll want larger pages if much more than a handful of 4KB pages per-stack get used (as otherwise the page tables get large), and then you're consuming large amounts of RAM anyways. I may be wrong as to the third item, but I suspect the actual depth of the stacks as-used makes a big difference. The point is that thread-per-client is just never going to scale better than C10K no matter what, and that the work we see in this space is all about finding the right balance between the ease-of-programming of thread-per-client and the efficiency of callback hell (typical C10K). For any size computer (larger than tiny), C10K designs will wring out orders of magnitude better scaling than typical thread-per-client apps no matter how much you optimize the hardware for the latter. Thread-per-client can only possibly compare in the same ballpark as C10K when the actual stack space used is tiny and comparable to the amount of explicit state kept in the C10K alternative, and even then, the additional kernel resources consumed by those per-client threads will dwarf those needed for the C10K case (thread per-CPU), thus adding cache pressure. In practice, thread-per-client is never that efficient because the whole point of it is that it makes it trivial to employ layered libraries of synchronous APIs. Now, I'm not saying that the alternatives to thread-per-client are easy, but we should do a lot better at building new libraries so that async I/O can get easier.