4 ms·

>I mean why not just use a thread per connection and not bother with anything fancier at all unless you really truly need to hit those C10M scales? Which I susp

>I mean why not just use a thread per connection and not bother with anything fancier at all unless you really truly need to hit those C10M scales? Which I suspect is a very rare need for most things? At every larger company I've worked for, that breaks down right away. It's really not hard to see how processing some larger proto/json/soap/xml msg can slow the entire system down.

fulafel 3y ago
Can you spell out how processing a large message would result in a bigger system slowdown with thread-per-connection than with async?
- deleted 3y ago
  [deleted]
- mcguire 3y ago
  Have a number of threads greater than the number of cores results in either synchronization costs or longer latency.
  - IshKebab 3y ago
    No it doesn't. It results in more context switching costs maybe, but I seriously doubt that would make a difference in all but the most extreme cases.
    - mcguire 3y ago
      Context switching, cache coherency, and the way that, in practice, you end up with #threads >> #cores.
  - bcrl 3y ago
    Latency is not the only cost of an excessive number of threads: context switches are expensive! Every context switch is a waste of CPU cycles that could be better spent doing actual work for your application. Furthermore, the cost of context switches keeps going up with every new generation of CPU, and I don't see that trend reversing any time soon.
kllrnohj 3y ago
The larger the workload per connection the better thread-per-connection performs relative to the alternatives. So it'd do the exact opposite of break down under the workload you've outlined.
laurencerowe 3y ago
> At every larger company I've worked for, that breaks down right away. It's really not hard to see how processing some larger proto/json/soap/xml msg can slow the entire system down. What's the problem with thread-per-connection in this case? It usually works well when application code is moderately heavy since the overhead from threads is very small in comparison. By contrast I've often seen issues in event driven systems without work stealing where heavy requests slow everything else down. I think work stealing is a sensible default for an event driven system in a language like Rust. But I do sometimes find myself wishing for the option to write non-async threaded http servers with zero-copy send.
- Spivak 3y ago
  > without work stealing where heavy requests slow everything else down Right because threads vs async doesn't make any sense. You use both, async is your default, free yourself up during io, and is basically the developer friendly abstraction on top of epoll, and then use threads for stuff you actually need threads for.
  - fiddlerwoaroof 3y ago
    I don’t understand this: context switching takes microseconds, I/O latencies are typically in the millisecond range. I’d think thread overhead would be negligible in an I/O-bound application (especially if you take steps to reduce the amount of memory per thread)
    - jandrewrogers 3y ago
      The important distinction is between operation latency and operation rate. Modern I/O devices are highly concurrent and support massive throughput. A device can have millisecond latency while still executing an operation every microsecond. In these cases, the operation latency doesn't matter, your thread has to handle events at the rate the device executes operations. If it is a million operations per second then from the perspective of the thread you have a microsecond to handle each operation. Context switch throughput is much lower by comparison. In these types of systems, you may issue a thousand concurrent I/O operations before the first I/O operation returns a result. Threads don't wait for the first operation to finish, they keep a deep pipeline of I/O operations in flight concurrently so that they always have something to do.
galdosdi 3y ago
Well, at every mid sized company I've been at, it's worked great. You can service a fuckton or clients off of one machine even with thread per connection. And frankly, if well architected so much of the logic and business is well separated from the logistics, it's not the hardest thing (and by then there plenty of money to fund it) to rearchitect a thread per client model into something more effecient. It's hard to go wrong when you create a system initially and stick with the easy lower scale ways, like a single big MySQL (or oracle or postgres or even SQLite or whatever) database, thread per connection, etc. YAGNI, and if you do need it, get it when you do need it

fulafel 3y ago

Can you spell out how processing a large message would result in a bigger system slowdown with thread-per-connection than with async?

deleted 3y ago
[deleted]
mcguire 3y ago
Have a number of threads greater than the number of cores results in either synchronization costs or longer latency.
- IshKebab 3y ago
  No it doesn't. It results in more context switching costs maybe, but I seriously doubt that would make a difference in all but the most extreme cases.
  - mcguire 3y ago
    Context switching, cache coherency, and the way that, in practice, you end up with #threads >> #cores.
- bcrl 3y ago
  Latency is not the only cost of an excessive number of threads: context switches are expensive! Every context switch is a waste of CPU cycles that could be better spent doing actual work for your application. Furthermore, the cost of context switches keeps going up with every new generation of CPU, and I don't see that trend reversing any time soon.

kllrnohj 3y ago

The larger the workload per connection the better thread-per-connection performs relative to the alternatives. So it'd do the exact opposite of break down under the workload you've outlined.

laurencerowe 3y ago

> At every larger company I've worked for, that breaks down right away. It's really not hard to see how processing some larger proto/json/soap/xml msg can slow the entire system down. What's the problem with thread-per-connection in this case? It usually works well when application code is moderately heavy since the overhead from threads is very small in comparison. By contrast I've often seen issues in event driven systems without work stealing where heavy requests slow everything else down. I think work stealing is a sensible default for an event driven system in a language like Rust. But I do sometimes find myself wishing for the option to write non-async threaded http servers with zero-copy send.

Spivak 3y ago
> without work stealing where heavy requests slow everything else down Right because threads vs async doesn't make any sense. You use both, async is your default, free yourself up during io, and is basically the developer friendly abstraction on top of epoll, and then use threads for stuff you actually need threads for.
- fiddlerwoaroof 3y ago
  I don’t understand this: context switching takes microseconds, I/O latencies are typically in the millisecond range. I’d think thread overhead would be negligible in an I/O-bound application (especially if you take steps to reduce the amount of memory per thread)
  - jandrewrogers 3y ago
    The important distinction is between operation latency and operation rate. Modern I/O devices are highly concurrent and support massive throughput. A device can have millisecond latency while still executing an operation every microsecond. In these cases, the operation latency doesn't matter, your thread has to handle events at the rate the device executes operations. If it is a million operations per second then from the perspective of the thread you have a microsecond to handle each operation. Context switch throughput is much lower by comparison. In these types of systems, you may issue a thousand concurrent I/O operations before the first I/O operation returns a result. Threads don't wait for the first operation to finish, they keep a deep pipeline of I/O operations in flight concurrently so that they always have something to do.

galdosdi 3y ago

Well, at every mid sized company I've been at, it's worked great. You can service a fuckton or clients off of one machine even with thread per connection. And frankly, if well architected so much of the logic and business is well separated from the logistics, it's not the hardest thing (and by then there plenty of money to fund it) to rearchitect a thread per client model into something more effecient. It's hard to go wrong when you create a system initially and stick with the easy lower scale ways, like a single big MySQL (or oracle or postgres or even SQLite or whatever) database, thread per connection, etc. YAGNI, and if you do need it, get it when you do need it