4 ms·

> At every larger company I've worked for, that breaks down right away. It's really not hard to see how processing some larger proto/json/soap/xml msg can slow

> At every larger company I've worked for, that breaks down right away. It's really not hard to see how processing some larger proto/json/soap/xml msg can slow the entire system down. What's the problem with thread-per-connection in this case? It usually works well when application code is moderately heavy since the overhead from threads is very small in comparison. By contrast I've often seen issues in event driven systems without work stealing where heavy requests slow everything else down. I think work stealing is a sensible default for an event driven system in a language like Rust. But I do sometimes find myself wishing for the option to write non-async threaded http servers with zero-copy send.

Spivak 3y ago
> without work stealing where heavy requests slow everything else down Right because threads vs async doesn't make any sense. You use both, async is your default, free yourself up during io, and is basically the developer friendly abstraction on top of epoll, and then use threads for stuff you actually need threads for.
- fiddlerwoaroof 3y ago
  I don’t understand this: context switching takes microseconds, I/O latencies are typically in the millisecond range. I’d think thread overhead would be negligible in an I/O-bound application (especially if you take steps to reduce the amount of memory per thread)
  - jandrewrogers 3y ago
    The important distinction is between operation latency and operation rate. Modern I/O devices are highly concurrent and support massive throughput. A device can have millisecond latency while still executing an operation every microsecond. In these cases, the operation latency doesn't matter, your thread has to handle events at the rate the device executes operations. If it is a million operations per second then from the perspective of the thread you have a microsecond to handle each operation. Context switch throughput is much lower by comparison. In these types of systems, you may issue a thousand concurrent I/O operations before the first I/O operation returns a result. Threads don't wait for the first operation to finish, they keep a deep pipeline of I/O operations in flight concurrently so that they always have something to do.
    - fiddlerwoaroof 3y ago
      > Modern I/O devices are highly concurrent and support massive throughput. A device can have millisecond latency while still executing an operation every microsecond. In these cases, the operation latency doesn't matter, your thread has to handle events at the rate the device executes operations. This is true for some applications, like an OLAP database or similar. It’s not true for the typical user-facing app where you want to finish requests as soon as possible because a user is waiting and every millisecond costs you money.
- laurencerowe 3y ago
  Mixing the two is absolutely the right approach in general. But I still find myself wishing it were easier to build a very simple threaded web server in Rust so I could play around with stuff like MSG_ZEROCOPY [1]. Last time I looked there were no non-async http libraries in cargo other than a few toy servers. [1] https://lwn.net/Articles/726917/ https://lwn.net/Articles/726917/

Spivak 3y ago

> without work stealing where heavy requests slow everything else down Right because threads vs async doesn't make any sense. You use both, async is your default, free yourself up during io, and is basically the developer friendly abstraction on top of epoll, and then use threads for stuff you actually need threads for.

fiddlerwoaroof 3y ago
I don’t understand this: context switching takes microseconds, I/O latencies are typically in the millisecond range. I’d think thread overhead would be negligible in an I/O-bound application (especially if you take steps to reduce the amount of memory per thread)
- jandrewrogers 3y ago
  The important distinction is between operation latency and operation rate. Modern I/O devices are highly concurrent and support massive throughput. A device can have millisecond latency while still executing an operation every microsecond. In these cases, the operation latency doesn't matter, your thread has to handle events at the rate the device executes operations. If it is a million operations per second then from the perspective of the thread you have a microsecond to handle each operation. Context switch throughput is much lower by comparison. In these types of systems, you may issue a thousand concurrent I/O operations before the first I/O operation returns a result. Threads don't wait for the first operation to finish, they keep a deep pipeline of I/O operations in flight concurrently so that they always have something to do.
  - fiddlerwoaroof 3y ago
    > Modern I/O devices are highly concurrent and support massive throughput. A device can have millisecond latency while still executing an operation every microsecond. In these cases, the operation latency doesn't matter, your thread has to handle events at the rate the device executes operations. This is true for some applications, like an OLAP database or similar. It’s not true for the typical user-facing app where you want to finish requests as soon as possible because a user is waiting and every millisecond costs you money.
laurencerowe 3y ago
Mixing the two is absolutely the right approach in general. But I still find myself wishing it were easier to build a very simple threaded web server in Rust so I could play around with stuff like MSG_ZEROCOPY [1]. Last time I looked there were no non-async http libraries in cargo other than a few toy servers. [1] https://lwn.net/Articles/726917/ https://lwn.net/Articles/726917/