8 ms·

On the Performance of User-Mode Threads and Coroutines

147 points by carimura 6y ago

MaxBarraclough 6y ago
See also this thread from 2 days ago on Sub-10 ms Latency in Java: Concurrent GC with Green Threads. There's some discussion of Project Loom there. https://news.ycombinator.com/item?id=24059335 https://news.ycombinator.com/item?id=24059335
carimura 6y ago
If interested in aggregated Project Loom news: https://inside.java/tag/loom https://inside.java/tag/loom
vii 6y ago
The article explains that the primary benefit of user-mode green-thread fibers is not switching speed, but that they are much cheaper than OS threads, so you can have many more of them. The costs are paid in memory usage and also the operating system also has to do considerable bookkeeping for threads. However, Netty has offered strong support for callback style IO under the JVM for a long time. This effectively allows the same efficiencies. Of course it is also possible to do without Netty. Therefore the real advantage of Loom user-mode threads and co-routines is syntactical programming convenience. That's the real innovation in Loom!
- free_rms 6y ago
  I thought the memory cost of threads was more of a JVM thing (default 1mb stack) than an OS thing (pages that aren't mapped/resident don't cost much). What's the cost to the kernel besides a few structs?
  - pron 6y ago
    Once a page is committed, it cannot be uncommitted until the thread dies, because the OS can't be sure how much of the stack is actually used. It cannot even assume that only addresses above sp are used. Also, the granularity is that of a page, which could be significantly larger than a whole stack of some small, "shallow" thread, and we want lots of small threads.
    - marvy 6y ago
      How many parked "fibers" do you think a 32-bit JVM might be able to handle once loom is ready for prime time? A thousand? A million? Somewhere in between?
      - pron 6y ago
        When somebody shows up to do a 32-bit port we'll think about it :)
        marvy 6y ago
        I don't see why you need to wait for that. You already know the memory usage for 64-bit, and you implied elsewhere in this thread that memory usage is the key metric that makes these things cheaper than kernel threads. So you should be able to give an order of magnitude guesstimate even if no 32-bit port is in the works yet. Or no?
        pron 6y ago
        OK, so let's see. Suppose you have 2GB available for threads, and you want a million threads. This leaves 2KB per thread. Can 2KB suffice for a thread? Sure, for some. Could most threads take up less than 2KB? Probably not. Right now the representation of the stack in RAM is not very efficient, but it could be made much more so. If anyone wants to port to 32 bit and that turns out to be a problem, then I guess improving the stack representation would become a higher priority.
        marvy 6y ago
        Thanks! I think that answers my question. Since kernel threads will likely have 64KB stacks or even more, this is still a huge win, even for 32-bit code I guess.
      - chris_overseas 6y ago
        I doubt 32bit is an issue for this and I'd assume well over a million, if Kotlin's coroutines are anything to go by: https://kotlinlang.org/docs/tutorials/coroutines/coroutines-basic-jvm.html#lets-run-a-lot-of-them https://kotlinlang.org/docs/tutorials/coroutines/coroutines-...
  - asdfasgasdgasdg 6y ago
    Well, there is a cap on the number of PIDs you can have going, I think. Also, it is my understanding that thread switching causes TLB flushes, which can be expensive. You can also fit several stackless thread contexts within a single page, depending on the size of the context, whereas you'll certainly never get more than one stackful thread into a single page.
    - MaxBarraclough 6y ago
      I don't think there should be a TLB flush for a switch between threads, as they share a memory space. It's only necessary for switching between processes. https://stackoverflow.com/a/5440165/ https://stackoverflow.com/a/5440165/
      - asdfasgasdgasdg 6y ago
        Oops, I think you're right and I was wrong.
- willtim 6y ago
  > Therefore the real advantage of Loom user-mode threads and co-routines is syntactical programming convenience. That's the real innovation in Loom! It's an interesting alternative to Async-Await / Haskell Monads / F# workflows for sure; and a better fit for Java. But I think some credit is also due for languages like C#, F# and Haskell for providing some healthy competition and prior art in this area.
  - pron 6y ago
    The relevant prior art is in Scheme (and OCaml, a bit), Erlang and Go, and we do credit them where relevant. Nevertheless, there are a few innovations in Java, both in implementation and design. For example, Java allows you to provide a custom scheduler for virtual threads.
    - willtim 6y ago
      I am very impressed with the recent changes happening to Java, all of which seem very carefully thought out and planned, keep up the good work!
- The_rationalist 6y ago
  Netty approach should be inferior because of the followings: It doesn't have access to the JVM profiling and semantics It doesn't yet use io uring unlike loom (but there is an active gsoc that should fill this gap) It doesn't use restartable sequences unlike loom. I expect Netty to support Loom instead of their current mechanism, when it becomes available. The thing I wait the most is keeping the standard jdbc API but making it seamlessly truly asynchronous/non socket blocking, that would actually allow Java frameworks to win the TechEmpower benchmarck
The_rationalist 6y ago
I wonder if kernel aware green threads will obscolete conventional green threads such as in Loom. https://www.phoronix.com/scan.php?page=news_item&px=Google-User-Thread-Futex-Swap https://www.phoronix.com/scan.php?page=news_item&px=Google-U...
- pron 6y ago
  User-scheduled kernel threads mostly address the context-switch question (and, more generally, the choice of an appropriate scheduling algorithm), but don't help as much with the footprint, and so with the level of concurrency, L, and that's where the biggest win is. To do that requires a deeper knowledge of how the language makes use of the stack than the OS can do. Still, they have their uses, too, and the expectation is that when they arrive, Java will support them, too, as a third implementation of threads.
  - deleted 6y ago
    [deleted]
  - The_rationalist 6y ago
    To do that requires a deeper knowledge of how the language makes use of the stack than the OS can do. I often hear that argument but isn't the reverse argument also true? While the JVM has access to information and language semantics useful for threading, that the kernel doesn't have access to. Doesn't the kernel have access to useful information for threading that the JVM green threads doesn't have access to? The one information that come to my mind is being aware of all threads from all programs and not only the ones from the JVM. So could next generation green threads have some access to some of this information? Could this enable greater performance? If so, how? Could Google futex wait help there? they have their uses, too, and the expectation is that when they arrive, Java will support them, too, as a third implementation of threads. Great to hear that but I hope that their use cases will be well defined and not overlap too much with loom. wild idea : could the JVM loom scheduler at runtime convert a green thread into google thread & vice versa? That could help performance, especially on pathological cases? Finally, I believe that say 1000 green threads actually reduce to a thread pool of kernel threads encapsulated by the JVM, if this belief is correct, then the real kernel threads underlying the green threads could benefit from being futex wait kernel threads?
1f60c 6y ago
This is the first time I've heard of the .java TLD, but sadly only Oracle and its affiliates are allowed to register domains[0]. [0]: https://www.oracle.com/a/ocom/docs/registration-policy-java-2418402.pdf https://www.oracle.com/a/ocom/docs/registration-policy-java-...
barbarbar 6y ago
This site seems to have a lot interesting articles. I have been trying to read about things like this on OpenJDK. But this site seems to be very easy to navigate on (unlike openjdk). It is also highly likely that I am an idiot.
jillesvangurp 6y ago
The word performance is a bit misleading here. A key reason to use co-routines is not necessarily maxing out CPUs but IO. Non-blocking IO and co-routines allow handling many connections on very modest hardware. Blocking IO and languages that don't handle concurrency very well (scripting languages like ruby or python) deal with this by forking multiple processes, each of which typically can only do one thing at the time. Java historically worked around this by using threads. Using processes doesn't scale nearly as well and you typically run out of memory before you run out of CPU this way. Using threads like Java does scales a bit better but there are only so many threads you can juggle. Co-routines or green threads combined with non blocking IO is much better and is also becoming common on the JVM where pretty much most modern frameworks support this. And of course if you use Kotlin, co-routines provide a really solid implementation and programming model for this. This is also the reason for the popularity of go and node.js, which aren't known for their particular ability to get the last bit of performance out of CPUs but do scale rather nicely when it comes to handling non blocking IO asynchronously.
- game_the0ry 6y ago
  > Co-routines or green threads combined with non blocking IO is much better and is also becoming common on the JVM where pretty much most modern frameworks support this. Appreciate the insightful comment. Basically, the ideal utilization of compute resource (cpu and ram) is engineering for both async I/O and and maximizing cpu threads, correct? I agree, mostly. I can't say I am very confident in understanding / implementing concurrent threading in any language. I am working on trying to get it, but sometimes I don't know if it is worth the mental over head or the added code complexity. Single threaded async alone took me a while to grasp, and I am sure I still have gaps in my knowledge. But mentally modeling a single threaded execution is trivial. I guess I try maximize for my own understanding and efficiency versus maximizing for performance.
pcwalton 6y ago
This seems to be a response to Google's switchto work in the Linux kernel, which is motivated by task-switching costs, as per this 2013 LPC presentation: http://pdxplumbers.osuosl.org/2013/ocw//system/presentations/1653/original/LPC%20-%20User%20Threading.pdf http://pdxplumbers.osuosl.org/2013/ocw//system/presentations... Note the parenthetical in the article: "And that is how user-mode threads help: they increase L by orders of magnitude with potentially millions of user-mode threads instead of the meager thousands the OS can support (but don’t expect a 1000x increase in capacity; we’ve neglected computation costs and are bound to hit bottlenecks in the auxiliary services.)" This is something I'd like to see more of a focus on. For the generator use case, I can easily see how the kernel thread spawn operation is the bottleneck. But for the thread-per-connection server use case, I'm not sure how expensive this cost is relative to all the other work that the thread does. My suspicion is that Amdahl's Law is going to quickly rear its head here. Take stack size for example: assuming the kernel stack is 10kB, if your thread itself uses 10kB of stack, you've cut the theoretical memory advantage of M:N down from the cited 1000x to a mere 2x…
- pron 6y ago
  > This seems to be a response to Google's switchto work in the Linux kernel Actually, it's a response to a discussion where somebody asked, "isn't it all about context-switch cost?" But user-scheduled kernel thread is something we had in mind when designing Loom, and we've made sure we can be compatible with them. We've introduced the concept of a pluggable custom scheduler, and that could be used for kernel threads just as it's used for virtual threads. The user can choose a thread implementation -- all kernel, all user-mode, or part kernel/part usermode -- without changing any code. It's all an implementation detail. > My suspicion is that Amdahl's Law is going to quickly rear its head here. That depends. Amdahl's law is about accelerating one job by parallelising it, while here we're more concerned with Little's law, which is about the rate of independent requests you can process. > Take stack size for example: assuming the kernel stack is 10kB, if your thread itself uses 10kB of stack, you've cut the theoretical memory advantage of M:N down from the cited 1000x to a mere 2x… Ah, except that's not so easy to do. It's very hard to have "tight" stacks that are managed by the kernel for the reasons I mentioned here: https://news.ycombinator.com/item?id=24082951 https://news.ycombinator.com/item?id=24082951
kgoutham93 6y ago
Are there any online resources that provide an overview of various advanced concurrency patterns in various programming languages. I would like to understand various trade-offs between multiple approaches.
scott_s 6y ago
I was massively surprised a few years ago how efficient Linux thread context switching could be. I designed and implemented a dynamic scheduler for a streaming dataflow language a few years ago [1]. We wanted a runtime system which could have hundreds of OS-level threads execute thousands of dataflow operators that communicate in a dataflow manner. Threads should not be statically assigned portions of the dataflow graph so that we could elastically add or remove OS-level threads based on observed performance. We compared that scheduler to some other options, including just giving every operator in the graph its own dedicated thread. One test application was a simple 1,000 operator pipeline. We used two machines, one with 176 cores, the other 184 cores. To my surprise, with the pipeline application, the dedicated thread model beat my fancy scheduler in raw performance by up to a factor of 2. Keep in mind that that's 1,000 threads, all doing work, on machines with only 176 and 184 cores. Of course, you would not want to do this in practice, even though the raw performance was so high: the machine was so massively oversubscribed during such experiments that it could barely keep up with a simple interactive shell. But, my intuition had been wrong: I had thought that surely having 10x the number of threads as cores would mean the overall performance would crawl because of context switching time. It did not. See section 5.1 of my paper below for the experiment. [1] Low-Synchronization, Mostly Lock-Free, Elastic Scheduling for Streaming Runtimes, PLDI 2017, https://www.scott-a-s.com/files/pldi2017_lf_elastic_scheduling.pdf https://www.scott-a-s.com/files/pldi2017_lf_elastic_scheduli...
- twoodfin 6y ago
  This matches my experience: Linux is very good at maximizing throughput, even when significantly over-subscribed. Two broad gotchas: - If the "task workflow" of an application involves a pipeline of N threads performing logically connected operations in sequence, the direct and indirect cost of N context switches is going to put a lower bound on latency and resource consumption for each task. This gets worse if the N threads aren't all in the same process, and TLB flushes enter the picture. - Without careful priority tuning, at high levels of over-subscription, latency becomes highly variable, as you saw in an interactive shell. There's no free lunch; if you want predictable latency you need to reserve cycles that would otherwise be available to maximize throughput.
nhoughto 6y ago
In my mind user-mode threads are much more about engineering productivity than raw performance. The comparison is wasteful blocking network calls using threads written in a simple synchronous way vs efficient user mode threads making network calls in a non blocking way also written synchronously. Today to get non blocking efficiencies you need to write async code bc without loom there is no otb user mode threads, even if user mode threads aren’t as efficient in raw numbers it’s still waaay ahead of the predominant way threads are used today in Java (mostly wrt network calls).
ciconia 6y ago
I think the Go scheduler makes a good case for multiplexing a large number of user-mode threads (or goroutines) over a limited number of OS threads: whenever a goroutine needs to make a blocking syscall (e.g. writing to a socket), the work is offloaded to a "network poller" thread, and meanwhile the Go scheduler can continue to service other runnable goroutines. It seems to me that this allows Go programs to maximize the use of their OS-thread time slices without being preempted on each syscall. Does anybody care to comment on how this impacts throughput and latency?

147 points by carimura 6y ago

See also this thread from 2 days ago on Sub-10 ms Latency in Java: Concurrent GC with Green Threads. There's some discussion of Project Loom there. https://news.ycombinator.com/item?id=24059335 https://news.ycombinator.com/item?id=24059335

carimura 6y ago

If interested in aggregated Project Loom news: https://inside.java/tag/loom https://inside.java/tag/loom

vii 6y ago

The article explains that the primary benefit of user-mode green-thread fibers is not switching speed, but that they are much cheaper than OS threads, so you can have many more of them. The costs are paid in memory usage and also the operating system also has to do considerable bookkeeping for threads. However, Netty has offered strong support for callback style IO under the JVM for a long time. This effectively allows the same efficiencies. Of course it is also possible to do without Netty. Therefore the real advantage of Loom user-mode threads and co-routines is syntactical programming convenience. That's the real innovation in Loom!

free_rms 6y ago
I thought the memory cost of threads was more of a JVM thing (default 1mb stack) than an OS thing (pages that aren't mapped/resident don't cost much). What's the cost to the kernel besides a few structs?
- pron 6y ago
  Once a page is committed, it cannot be uncommitted until the thread dies, because the OS can't be sure how much of the stack is actually used. It cannot even assume that only addresses above sp are used. Also, the granularity is that of a page, which could be significantly larger than a whole stack of some small, "shallow" thread, and we want lots of small threads.
  - marvy 6y ago
    How many parked "fibers" do you think a 32-bit JVM might be able to handle once loom is ready for prime time? A thousand? A million? Somewhere in between?
    - pron 6y ago
      When somebody shows up to do a 32-bit port we'll think about it :)
      - marvy 6y ago
        I don't see why you need to wait for that. You already know the memory usage for 64-bit, and you implied elsewhere in this thread that memory usage is the key metric that makes these things cheaper than kernel threads. So you should be able to give an order of magnitude guesstimate even if no 32-bit port is in the works yet. Or no?
        pron 6y ago
        OK, so let's see. Suppose you have 2GB available for threads, and you want a million threads. This leaves 2KB per thread. Can 2KB suffice for a thread? Sure, for some. Could most threads take up less than 2KB? Probably not. Right now the representation of the stack in RAM is not very efficient, but it could be made much more so. If anyone wants to port to 32 bit and that turns out to be a problem, then I guess improving the stack representation would become a higher priority.
        marvy 6y ago
        Thanks! I think that answers my question. Since kernel threads will likely have 64KB stacks or even more, this is still a huge win, even for 32-bit code I guess.
    - chris_overseas 6y ago
      I doubt 32bit is an issue for this and I'd assume well over a million, if Kotlin's coroutines are anything to go by: https://kotlinlang.org/docs/tutorials/coroutines/coroutines-basic-jvm.html#lets-run-a-lot-of-them https://kotlinlang.org/docs/tutorials/coroutines/coroutines-...
- asdfasgasdgasdg 6y ago
  Well, there is a cap on the number of PIDs you can have going, I think. Also, it is my understanding that thread switching causes TLB flushes, which can be expensive. You can also fit several stackless thread contexts within a single page, depending on the size of the context, whereas you'll certainly never get more than one stackful thread into a single page.
  - MaxBarraclough 6y ago
    I don't think there should be a TLB flush for a switch between threads, as they share a memory space. It's only necessary for switching between processes. https://stackoverflow.com/a/5440165/ https://stackoverflow.com/a/5440165/
    - asdfasgasdgasdg 6y ago
      Oops, I think you're right and I was wrong.
willtim 6y ago
> Therefore the real advantage of Loom user-mode threads and co-routines is syntactical programming convenience. That's the real innovation in Loom! It's an interesting alternative to Async-Await / Haskell Monads / F# workflows for sure; and a better fit for Java. But I think some credit is also due for languages like C#, F# and Haskell for providing some healthy competition and prior art in this area.
- pron 6y ago
  The relevant prior art is in Scheme (and OCaml, a bit), Erlang and Go, and we do credit them where relevant. Nevertheless, there are a few innovations in Java, both in implementation and design. For example, Java allows you to provide a custom scheduler for virtual threads.
  - willtim 6y ago
    I am very impressed with the recent changes happening to Java, all of which seem very carefully thought out and planned, keep up the good work!
The_rationalist 6y ago
Netty approach should be inferior because of the followings: It doesn't have access to the JVM profiling and semantics It doesn't yet use io uring unlike loom (but there is an active gsoc that should fill this gap) It doesn't use restartable sequences unlike loom. I expect Netty to support Loom instead of their current mechanism, when it becomes available. The thing I wait the most is keeping the standard jdbc API but making it seamlessly truly asynchronous/non socket blocking, that would actually allow Java frameworks to win the TechEmpower benchmarck

The_rationalist 6y ago

I wonder if kernel aware green threads will obscolete conventional green threads such as in Loom. https://www.phoronix.com/scan.php?page=news_item&px=Google-User-Thread-Futex-Swap https://www.phoronix.com/scan.php?page=news_item&px=Google-U...

pron 6y ago
User-scheduled kernel threads mostly address the context-switch question (and, more generally, the choice of an appropriate scheduling algorithm), but don't help as much with the footprint, and so with the level of concurrency, L, and that's where the biggest win is. To do that requires a deeper knowledge of how the language makes use of the stack than the OS can do. Still, they have their uses, too, and the expectation is that when they arrive, Java will support them, too, as a third implementation of threads.
- deleted 6y ago
  [deleted]
- The_rationalist 6y ago
  To do that requires a deeper knowledge of how the language makes use of the stack than the OS can do. I often hear that argument but isn't the reverse argument also true? While the JVM has access to information and language semantics useful for threading, that the kernel doesn't have access to. Doesn't the kernel have access to useful information for threading that the JVM green threads doesn't have access to? The one information that come to my mind is being aware of all threads from all programs and not only the ones from the JVM. So could next generation green threads have some access to some of this information? Could this enable greater performance? If so, how? Could Google futex wait help there? they have their uses, too, and the expectation is that when they arrive, Java will support them, too, as a third implementation of threads. Great to hear that but I hope that their use cases will be well defined and not overlap too much with loom. wild idea : could the JVM loom scheduler at runtime convert a green thread into google thread & vice versa? That could help performance, especially on pathological cases? Finally, I believe that say 1000 green threads actually reduce to a thread pool of kernel threads encapsulated by the JVM, if this belief is correct, then the real kernel threads underlying the green threads could benefit from being futex wait kernel threads?

1f60c 6y ago

This is the first time I've heard of the .java TLD, but sadly only Oracle and its affiliates are allowed to register domains[0]. [0]: https://www.oracle.com/a/ocom/docs/registration-policy-java-2418402.pdf https://www.oracle.com/a/ocom/docs/registration-policy-java-...

barbarbar 6y ago

This site seems to have a lot interesting articles. I have been trying to read about things like this on OpenJDK. But this site seems to be very easy to navigate on (unlike openjdk). It is also highly likely that I am an idiot.

jillesvangurp 6y ago

The word performance is a bit misleading here. A key reason to use co-routines is not necessarily maxing out CPUs but IO. Non-blocking IO and co-routines allow handling many connections on very modest hardware. Blocking IO and languages that don't handle concurrency very well (scripting languages like ruby or python) deal with this by forking multiple processes, each of which typically can only do one thing at the time. Java historically worked around this by using threads. Using processes doesn't scale nearly as well and you typically run out of memory before you run out of CPU this way. Using threads like Java does scales a bit better but there are only so many threads you can juggle. Co-routines or green threads combined with non blocking IO is much better and is also becoming common on the JVM where pretty much most modern frameworks support this. And of course if you use Kotlin, co-routines provide a really solid implementation and programming model for this. This is also the reason for the popularity of go and node.js, which aren't known for their particular ability to get the last bit of performance out of CPUs but do scale rather nicely when it comes to handling non blocking IO asynchronously.

game_the0ry 6y ago
> Co-routines or green threads combined with non blocking IO is much better and is also becoming common on the JVM where pretty much most modern frameworks support this. Appreciate the insightful comment. Basically, the ideal utilization of compute resource (cpu and ram) is engineering for both async I/O and and maximizing cpu threads, correct? I agree, mostly. I can't say I am very confident in understanding / implementing concurrent threading in any language. I am working on trying to get it, but sometimes I don't know if it is worth the mental over head or the added code complexity. Single threaded async alone took me a while to grasp, and I am sure I still have gaps in my knowledge. But mentally modeling a single threaded execution is trivial. I guess I try maximize for my own understanding and efficiency versus maximizing for performance.

pcwalton 6y ago

This seems to be a response to Google's switchto work in the Linux kernel, which is motivated by task-switching costs, as per this 2013 LPC presentation: http://pdxplumbers.osuosl.org/2013/ocw//system/presentations/1653/original/LPC%20-%20User%20Threading.pdf http://pdxplumbers.osuosl.org/2013/ocw//system/presentations... Note the parenthetical in the article: "And that is how user-mode threads help: they increase L by orders of magnitude with potentially millions of user-mode threads instead of the meager thousands the OS can support (but don’t expect a 1000x increase in capacity; we’ve neglected computation costs and are bound to hit bottlenecks in the auxiliary services.)" This is something I'd like to see more of a focus on. For the generator use case, I can easily see how the kernel thread spawn operation is the bottleneck. But for the thread-per-connection server use case, I'm not sure how expensive this cost is relative to all the other work that the thread does. My suspicion is that Amdahl's Law is going to quickly rear its head here. Take stack size for example: assuming the kernel stack is 10kB, if your thread itself uses 10kB of stack, you've cut the theoretical memory advantage of M:N down from the cited 1000x to a mere 2x…

pron 6y ago
> This seems to be a response to Google's switchto work in the Linux kernel Actually, it's a response to a discussion where somebody asked, "isn't it all about context-switch cost?" But user-scheduled kernel thread is something we had in mind when designing Loom, and we've made sure we can be compatible with them. We've introduced the concept of a pluggable custom scheduler, and that could be used for kernel threads just as it's used for virtual threads. The user can choose a thread implementation -- all kernel, all user-mode, or part kernel/part usermode -- without changing any code. It's all an implementation detail. > My suspicion is that Amdahl's Law is going to quickly rear its head here. That depends. Amdahl's law is about accelerating one job by parallelising it, while here we're more concerned with Little's law, which is about the rate of independent requests you can process. > Take stack size for example: assuming the kernel stack is 10kB, if your thread itself uses 10kB of stack, you've cut the theoretical memory advantage of M:N down from the cited 1000x to a mere 2x… Ah, except that's not so easy to do. It's very hard to have "tight" stacks that are managed by the kernel for the reasons I mentioned here: https://news.ycombinator.com/item?id=24082951 https://news.ycombinator.com/item?id=24082951

kgoutham93 6y ago

Are there any online resources that provide an overview of various advanced concurrency patterns in various programming languages. I would like to understand various trade-offs between multiple approaches.

scott_s 6y ago

I was massively surprised a few years ago how efficient Linux thread context switching could be. I designed and implemented a dynamic scheduler for a streaming dataflow language a few years ago [1]. We wanted a runtime system which could have hundreds of OS-level threads execute thousands of dataflow operators that communicate in a dataflow manner. Threads should not be statically assigned portions of the dataflow graph so that we could elastically add or remove OS-level threads based on observed performance. We compared that scheduler to some other options, including just giving every operator in the graph its own dedicated thread. One test application was a simple 1,000 operator pipeline. We used two machines, one with 176 cores, the other 184 cores. To my surprise, with the pipeline application, the dedicated thread model beat my fancy scheduler in raw performance by up to a factor of 2. Keep in mind that that's 1,000 threads, all doing work, on machines with only 176 and 184 cores. Of course, you would not want to do this in practice, even though the raw performance was so high: the machine was so massively oversubscribed during such experiments that it could barely keep up with a simple interactive shell. But, my intuition had been wrong: I had thought that surely having 10x the number of threads as cores would mean the overall performance would crawl because of context switching time. It did not. See section 5.1 of my paper below for the experiment. [1] Low-Synchronization, Mostly Lock-Free, Elastic Scheduling for Streaming Runtimes, PLDI 2017, https://www.scott-a-s.com/files/pldi2017_lf_elastic_scheduling.pdf https://www.scott-a-s.com/files/pldi2017_lf_elastic_scheduli...

twoodfin 6y ago
This matches my experience: Linux is very good at maximizing throughput, even when significantly over-subscribed. Two broad gotchas: - If the "task workflow" of an application involves a pipeline of N threads performing logically connected operations in sequence, the direct and indirect cost of N context switches is going to put a lower bound on latency and resource consumption for each task. This gets worse if the N threads aren't all in the same process, and TLB flushes enter the picture. - Without careful priority tuning, at high levels of over-subscription, latency becomes highly variable, as you saw in an interactive shell. There's no free lunch; if you want predictable latency you need to reserve cycles that would otherwise be available to maximize throughput.

nhoughto 6y ago

In my mind user-mode threads are much more about engineering productivity than raw performance. The comparison is wasteful blocking network calls using threads written in a simple synchronous way vs efficient user mode threads making network calls in a non blocking way also written synchronously. Today to get non blocking efficiencies you need to write async code bc without loom there is no otb user mode threads, even if user mode threads aren’t as efficient in raw numbers it’s still waaay ahead of the predominant way threads are used today in Java (mostly wrt network calls).

ciconia 6y ago

I think the Go scheduler makes a good case for multiplexing a large number of user-mode threads (or goroutines) over a limited number of OS threads: whenever a goroutine needs to make a blocking syscall (e.g. writing to a socket), the work is offloaded to a "network poller" thread, and meanwhile the Go scheduler can continue to service other runnable goroutines. It seems to me that this allows Go programs to maximize the use of their OS-thread time slices without being preempted on each syscall. Does anybody care to comment on how this impacts throughput and latency?