5 ms·

I often find developers jump to async too early. Clearly a million native threads is too many, but on modern processors a couple thousand mostly idle (due to IO

I often find developers jump to async too early. Clearly a million native threads is too many, but on modern processors a couple thousand mostly idle (due to IO blocking) Java threads are fine (e.g. one per request in a typical middle tier). While Java's stack sizes are large, if a thread's stack isn't deep, most of that memory is never actually allocated. What is that breaking point where developers should switch to more scalable models? From the article: > There is a threshold beyond which the use of synchronous APIs just doesn’t scale What is a good rule of thumb these days?

riobard 5y ago
> on modern processors a couple thousand mostly idle (due to IO blocking) Java threads are fine This “mostly idle” assumption is gonna break the moment you hit a load spike and wake up most of those “idle” threads. The success of async is often (mis)attributed to eliminating unnecessary resources, but in practice it's actually because async model implicitly enforces an invisible job queue, where usually only a single process/thread per core is heavy lifting the actual work, and async I/O will feed processing pressure back to the load source via network (e.g. TCP ACK).
- fulafel 5y ago
  I'm not sure scheduling the work in your own process necessarily solves the load problem better? The described queue behaviour would happen in the OS in the threads scenario, and TCP would work the same too.
  - riobard 5y ago
    It's not about which way is better, but which default is more useful in practice.
DaiPlusPlus 5y ago
Java may be an exception, but on Windows in general, and in .NET, every thread has its stack preallocated and is generally 256KB to 4MB by default, so spawning “a couple thousand threads” as you suggest will immediately consume 250MB-4GB of main-memory which adds considerable memory-pressure. That’s not something to be taken lightly. By using (true) async IO you’ll likely only need 20-50MB total because it uses the threadpool for continuations which is sized according to the actual available level of concurrency. Async NetworkStreams or Sockets in .NET is a well-designed API over traditional BSD sockets’ select() function which is how you scale to tens of thousands of concurrent connections using only as many threads as you have hardware DoP - it’s not that I’m wowed that .NET has modernised it, it’s that Java’s solution is to wallpaper over a gaping hole in their fundamental design instead.
- pjmlp 5y ago
  Tainting the whole call is major problem in .NET code. That is why you will find out lots of code bases full of Task.Run() and direct calls to .Result, instead of using async/await operators. Having worked with both platform since their yearly days, I am not sure which design is actually the best one, considering brown field development.
  - DaiPlusPlus 5y ago
    What you’re describing is just fake-async and that is discouraged in the ecosystem. I’m not aware of any major NuGet packages that use fake-async, and certainly nothing in the BCL.
    - pjmlp 5y ago
      Not at all. What I am describing is what a large majority of .NET consulting projects do, because async/await taints the whole call stack up to Main() or event handlers, and almost no one is doing .NET Core projects from scratch. Just because it is discouraged at conference talks and blog posts, doesn't mean devs aren't doing it on the trenches when dealing with the crossfire of project deadlines and massive codebases.
      - DaiPlusPlus 5y ago
        > Just because it is discouraged at conference talks and blog posts, doesn't mean devs aren't doing it on the trenches when dealing with the crossfire of project deadlines and massive codebases. Oh, I'm sure of that, don't worry. I'm just fortunate that I've never had to experience that myself.
- native_samples 5y ago
  I don't see how this is wallpapering over a hole in the design. Far from it. Java has supported async IO since Java 1.4 using APIs like java.nio.channels.AsynchronousChannel. That's a more or less conventional wrapper around epoll. What's being introduced now is a way to use that API without needing to touch the whole concept of async code. From the developer's perspective blocking calls and threads is pretty nice. It's async that's awkward and hard to use. Making the existing threads facility scale much better is a very clean approach: virtual threads introduces "virtually" no new concepts at all, in fact, if you aren't working with native code via JNI or Panama then you can act as if virtual and physical threads are the same. You literally just upgrade Java and maybe flip a couple of switches and you can now assign one thread per connection yet handle millions of connections (assuming you don't run out of other resources of course).
fulafel 5y ago
A million threads is a lot, but the reasonable limit is more like 100k idle threads than a few thousand, on 64-bit Linux.
- MichaelMoser123 5y ago
  if you get to a very high number of cooperative threads, then each such thread will have a very small stack (otherwise the OOM killer will get you). i am not sure that you can impose such limitations on the stack size in java.
  - fulafel 5y ago
    If you are short on memory, sure. 100k times 10 mb for example is just 1 TB, an order of magnitude or two less than what bigger x86 servers take.
jchw 5y ago
Context switching is pretty cheap, but it’s hard to beat free. Eventually the overhead of OS context switching is probably going to become non-negligible in most cases. I think up to a point the answer to when it matters is “that depends on how deep your pockets are.” You can keep scaling vertically and horizontally forever with more resources, obviously, but I don’t think it would take too long for the memory and CPU efficiency of async to start to win out, at least for runtimes with proper async. Besides that, it’s pretty hard to decide you want to go async later on. In languages where async is not omnipresent, you usually get some form of explicit continuations; async/await sugar if you’re lucky. But you can always accidentally call something not async and hang, or just spend a lot of CPU time with no ability to preempt. I think Erlang, Go and probably a couple of others basically stand alone in this regard; everything is just async by default. It’s a shame that it requires a decent amount of runtime baggage to accomplish this.
- bestinterest 5y ago
  I also think people forget that Robinhood was made with Python, Reddit used Python, Github/Gitlab use Ruby. Stripe AKA payments that need to be processed fast within SLA's use Ruby. Ruby and Python can only ever have one thread running at a time due to global interpreter lock. Though to keep in mind when I/O happens on a thread it can be parked and another thread can be picked up to continue processing. But in essence one thread is running at a time and these languages built billion dollar businesses. Just scale horizontally computers are cheap is the message I get and remember to keep it simple.
  - BenoitP 5y ago
    Python and Ruby are fine for embarrassingly parallel problems, like serving content from well-sharded users. I'd be very surprised if Robinhood's trade matching engine is done in Python. I'd bet they use an off-the-shelf engine in C++/Java or another language with good multi-threading support. > computers are cheap is the message I get and remember to keep it simple. Fully agree! Getting to product-market fit is probably easier with python and ruby.
  - jchw 5y ago
nly 5y ago
Once you have an async API (of any kind), and have paid the cognitive and technical price of having one, you have a lot of flexibility about what you do on the implementation side - including thousands of threads. If you start off with a synchronous API and thousands of threads, migrating to something else later is very expensive.
engie 5y ago
Latency can be a key consideration. Async style code can make it a lot easier to fire off a set of requests in parallel and wait for all the responses to come back (e.g. with https://docs.hhvm.com/hack/asynchronous-operations/concurrent https://docs.hhvm.com/hack/asynchronous-operations/concurren...), rather than the serial request waterfall a basic thread-per-request model will push you towards.
grandinj 5y ago
i run up to about 500 threads on my Java app with ease, with about 1/3 of them pumping significant data (think stereo streaming audio type data). I started writing some fancy asynch code, but on checking with a profiler, the thread overhead was pretty negligable, so I left it alone.
dom96 5y ago
I often find that developers jump for threads too early. Concurrency is much easier to reason about than parallelism.
signa11 5y ago
using thread scheduling for packet scheduling goes only so far.

riobard 5y ago

> on modern processors a couple thousand mostly idle (due to IO blocking) Java threads are fine This “mostly idle” assumption is gonna break the moment you hit a load spike and wake up most of those “idle” threads. The success of async is often (mis)attributed to eliminating unnecessary resources, but in practice it's actually because async model implicitly enforces an invisible job queue, where usually only a single process/thread per core is heavy lifting the actual work, and async I/O will feed processing pressure back to the load source via network (e.g. TCP ACK).

fulafel 5y ago
I'm not sure scheduling the work in your own process necessarily solves the load problem better? The described queue behaviour would happen in the OS in the threads scenario, and TCP would work the same too.
- riobard 5y ago
  It's not about which way is better, but which default is more useful in practice.

DaiPlusPlus 5y ago

Java may be an exception, but on Windows in general, and in .NET, every thread has its stack preallocated and is generally 256KB to 4MB by default, so spawning “a couple thousand threads” as you suggest will immediately consume 250MB-4GB of main-memory which adds considerable memory-pressure. That’s not something to be taken lightly. By using (true) async IO you’ll likely only need 20-50MB total because it uses the threadpool for continuations which is sized according to the actual available level of concurrency. Async NetworkStreams or Sockets in .NET is a well-designed API over traditional BSD sockets’ select() function which is how you scale to tens of thousands of concurrent connections using only as many threads as you have hardware DoP - it’s not that I’m wowed that .NET has modernised it, it’s that Java’s solution is to wallpaper over a gaping hole in their fundamental design instead.

pjmlp 5y ago
Tainting the whole call is major problem in .NET code. That is why you will find out lots of code bases full of Task.Run() and direct calls to .Result, instead of using async/await operators. Having worked with both platform since their yearly days, I am not sure which design is actually the best one, considering brown field development.
- DaiPlusPlus 5y ago
  What you’re describing is just fake-async and that is discouraged in the ecosystem. I’m not aware of any major NuGet packages that use fake-async, and certainly nothing in the BCL.
  - pjmlp 5y ago
    Not at all. What I am describing is what a large majority of .NET consulting projects do, because async/await taints the whole call stack up to Main() or event handlers, and almost no one is doing .NET Core projects from scratch. Just because it is discouraged at conference talks and blog posts, doesn't mean devs aren't doing it on the trenches when dealing with the crossfire of project deadlines and massive codebases.
    - DaiPlusPlus 5y ago
      > Just because it is discouraged at conference talks and blog posts, doesn't mean devs aren't doing it on the trenches when dealing with the crossfire of project deadlines and massive codebases. Oh, I'm sure of that, don't worry. I'm just fortunate that I've never had to experience that myself.
native_samples 5y ago
I don't see how this is wallpapering over a hole in the design. Far from it. Java has supported async IO since Java 1.4 using APIs like java.nio.channels.AsynchronousChannel. That's a more or less conventional wrapper around epoll. What's being introduced now is a way to use that API without needing to touch the whole concept of async code. From the developer's perspective blocking calls and threads is pretty nice. It's async that's awkward and hard to use. Making the existing threads facility scale much better is a very clean approach: virtual threads introduces "virtually" no new concepts at all, in fact, if you aren't working with native code via JNI or Panama then you can act as if virtual and physical threads are the same. You literally just upgrade Java and maybe flip a couple of switches and you can now assign one thread per connection yet handle millions of connections (assuming you don't run out of other resources of course).

fulafel 5y ago

A million threads is a lot, but the reasonable limit is more like 100k idle threads than a few thousand, on 64-bit Linux.

MichaelMoser123 5y ago
if you get to a very high number of cooperative threads, then each such thread will have a very small stack (otherwise the OOM killer will get you). i am not sure that you can impose such limitations on the stack size in java.
- fulafel 5y ago
  If you are short on memory, sure. 100k times 10 mb for example is just 1 TB, an order of magnitude or two less than what bigger x86 servers take.

jchw 5y ago

Context switching is pretty cheap, but it’s hard to beat free. Eventually the overhead of OS context switching is probably going to become non-negligible in most cases. I think up to a point the answer to when it matters is “that depends on how deep your pockets are.” You can keep scaling vertically and horizontally forever with more resources, obviously, but I don’t think it would take too long for the memory and CPU efficiency of async to start to win out, at least for runtimes with proper async. Besides that, it’s pretty hard to decide you want to go async later on. In languages where async is not omnipresent, you usually get some form of explicit continuations; async/await sugar if you’re lucky. But you can always accidentally call something not async and hang, or just spend a lot of CPU time with no ability to preempt. I think Erlang, Go and probably a couple of others basically stand alone in this regard; everything is just async by default. It’s a shame that it requires a decent amount of runtime baggage to accomplish this.

bestinterest 5y ago
I also think people forget that Robinhood was made with Python, Reddit used Python, Github/Gitlab use Ruby. Stripe AKA payments that need to be processed fast within SLA's use Ruby. Ruby and Python can only ever have one thread running at a time due to global interpreter lock. Though to keep in mind when I/O happens on a thread it can be parked and another thread can be picked up to continue processing. But in essence one thread is running at a time and these languages built billion dollar businesses. Just scale horizontally computers are cheap is the message I get and remember to keep it simple.
- BenoitP 5y ago
  Python and Ruby are fine for embarrassingly parallel problems, like serving content from well-sharded users. I'd be very surprised if Robinhood's trade matching engine is done in Python. I'd bet they use an off-the-shelf engine in C++/Java or another language with good multi-threading support. > computers are cheap is the message I get and remember to keep it simple. Fully agree! Getting to product-market fit is probably easier with python and ruby.
- jchw 5y ago

nly 5y ago

Once you have an async API (of any kind), and have paid the cognitive and technical price of having one, you have a lot of flexibility about what you do on the implementation side - including thousands of threads. If you start off with a synchronous API and thousands of threads, migrating to something else later is very expensive.

engie 5y ago

Latency can be a key consideration. Async style code can make it a lot easier to fire off a set of requests in parallel and wait for all the responses to come back (e.g. with https://docs.hhvm.com/hack/asynchronous-operations/concurrent https://docs.hhvm.com/hack/asynchronous-operations/concurren...), rather than the serial request waterfall a basic thread-per-request model will push you towards.

grandinj 5y ago

i run up to about 500 threads on my Java app with ease, with about 1/3 of them pumping significant data (think stereo streaming audio type data). I started writing some fancy asynch code, but on checking with a profiler, the thread overhead was pretty negligable, so I left it alone.

dom96 5y ago

I often find that developers jump for threads too early. Concurrency is much easier to reason about than parallelism.

signa11 5y ago

using thread scheduling for packet scheduling goes only so far.