5 ms·

#2 seems questionable. I always program in a thread-per-request style and start threads for blocking sub-operations of top level requests as well. I know there

#2 seems questionable. I always program in a thread-per-request style and start threads for blocking sub-operations of top level requests as well. I know there are platforms where this doesn't work well and I also know a lot of people cut their teeth on dreadful systems like sunos where it took ten seconds to start a thread, but on modern Linux threads are dirt cheap and well worth the clarity that synchronous programming allows.

dclusin 11y ago
A context switch takes about 50 microseconds. If you want to handle 10,000 requests in a second 500 milliseconds is lost to context switching alone. Probably more. The nature of IO & user input in general is to hurry up and wait. It's usually simpler to do async requests & polling file descriptors rather than thread-per-request. The former seems on the surface more complicated and more lines of code, but IMO, ends up being less complicated because you won't have to spend time digging through logs trying to figure out that the reason some other unrelated thread stalled for so long is because you were blasted with thousands of requests in the same second.
- thrownaway2424 11y ago
  Where did you get that number? That's the kind of SunOS derangement I'm talking about. On Linux with NPTL (which is over a decade old) and any recent Intel CPU (Westmere or later) a context switch between threads of the same process takes only about 1-2 microseconds.
  - jcoffland 11y ago
    Regardless, context switching, waiting on locks and the added cost of allocating a stack to each thread are all relatively expensive. I've gotten huge performance increases by rewriting threaded code to instead use non-blocking IO. Nginx vs. Apache is a great example of this difference. https://www.nginx.com/blog/inside-nginx-how-we-designed-for-performance-scale/ https://www.nginx.com/blog/inside-nginx-how-we-designed-for-...
    - thrownaway2424 11y ago
      I think these can be and often are combined. You can definitely have an event driven main IO loop accepting connections and reading packets, that nevertheless dispatches whole requests onto their own threads. I work with a lot of services written that way. The fact that you can reuse the frontend for several applications amortizes the cost of developing it as a high performance asynchronous thing. As for stack allocation I definitely see your point there. It is bizarre that Linux defaults to 2MB or whatever it is these days. I usually work with a 64KB thread stack.
- jcoffland 11y ago
  This is exactly what I'm talking about. If you want to reach really high transactions rates then you cannot use one thread per transaction.
  - thrownaway2424 11y ago
    I won't argue that async software thread per hardware thread style will have better maximum performance than a thread-per-request style, but I think that difference will not be material in most applications (the difference between the two styles of C++ server programming being far smaller than the difference between, say, a Go program and a C++ program), and the cost paid in code readability is pretty high. All that async continuation passing makes your code a twisty maze, and that style is fraught with lifetime problems that are the very thing most C++ haters are complaining about. In synchronous style all the allocations are scoped by nature. In async style the programmer has to assign ownership of everything to some master continuation that will delete them all in the final callback. That adds a lot of code.
    - jcoffland 11y ago
      You are right about the added difficulty of resource ownership in the face of asynchronous code. This is a trade off. lambda functions in C++11 can help with this. They make it possible to pass resource allocations (smart pointers) through to callbacks.
    - zzzcpan 11y ago
      > All that async continuation passing makes your code a twisty maze On the contrary, it gives your code clarity, consistency and predictability. > some master continuation that will delete them all in the final callback Even nginx in plain C is not doing that, but simply creates destructors on allocation to avoid tracking stuff manually. Just don't jump into the first thing you read somewhere, remember one simple rule: if it feels harder than threads - you are doing it wrong.
      - SamReidHughes 11y ago
zzzcpan 11y ago
Synchronous looking programming doesn't offer any clarity for anything beyond dumb sequential requests to a single node, that nobody cares about. Try building a distributed k/v store with that. It fails miserably once you have to deal with zero downtime restarts for continuous deployment, rate limiting, requests cancellations, waiting for one or multiple responses from multiple nodes, making more requests to other nodes just after a bit of time to guarantee latency, using a lot of different kinds of shared data, like response queues, log queues, connection caches, data caches, all used by multiple clients, while at the same time doing some synchronization of data between nodes, etc. Anything non-trivial is impossible with threads in practice.
- thrownaway2424 11y ago
  These statements are so cute. I can reasonably assure you that the distributed k/v store I have written in the thread-per-request style is currently out there serving more traffic than you've ever dreamed of. As for your specific points, I disagree totally. Each of those things is easier to me in the synchronous style than in the async style. Certainly something like hedging is way easier.
  - CyberDildonics 11y ago
    Those are some extraordinary claims, but I'm not seeing an extraordinary evidence.
    - SamReidHughes 11y ago
      If you're talking to disk, you'll eventually be doing it synchronously anyway, because on Linux the API's for async disk I/O aren't (weren't?) all there yet. For some time I was working on a product with a command line flag to decide between using a worker pool of threads versus event-based disk I/O, and there wasn't a big enough performance difference (if any, I don't remember) for us to keep the event-based code. If you're getting some fancy PCI SSD's or persistent RAM and really want to minimize latency-to-fsync time, maybe things would be different. Also, if you use an event loop, you have to do scheduling yourself. For distributed key/value stores, good thread scheduling is even more important. It removes a big amount of technical risk to lean on the kernel's, unless your product is sufficiently minimalistic. Async will get you real performance advantages for pure networking, but if you're talking to disk it's, when not buggy, not very useful anyway. Also, as for programming style, even if you want event-based syscalls, cooperative userland-managed threads are a big win over callback hell. The only bad thing about them is wasting address-space on 32-bit systems, and maybe just wasting memory too.
ArkyBeagle 11y ago
Having things be designed as event-driven may provide more opportunity for something incrementally more akin to proof of correctness. Add a large quantum of salt and caveat that; it's simply slightly easier that way. People don't like it in general, but event driven works for me, mostly. Your mileage will almost certainly vary.

dclusin 11y ago

A context switch takes about 50 microseconds. If you want to handle 10,000 requests in a second 500 milliseconds is lost to context switching alone. Probably more. The nature of IO & user input in general is to hurry up and wait. It's usually simpler to do async requests & polling file descriptors rather than thread-per-request. The former seems on the surface more complicated and more lines of code, but IMO, ends up being less complicated because you won't have to spend time digging through logs trying to figure out that the reason some other unrelated thread stalled for so long is because you were blasted with thousands of requests in the same second.

thrownaway2424 11y ago
Where did you get that number? That's the kind of SunOS derangement I'm talking about. On Linux with NPTL (which is over a decade old) and any recent Intel CPU (Westmere or later) a context switch between threads of the same process takes only about 1-2 microseconds.
- jcoffland 11y ago
  Regardless, context switching, waiting on locks and the added cost of allocating a stack to each thread are all relatively expensive. I've gotten huge performance increases by rewriting threaded code to instead use non-blocking IO. Nginx vs. Apache is a great example of this difference. https://www.nginx.com/blog/inside-nginx-how-we-designed-for-performance-scale/ https://www.nginx.com/blog/inside-nginx-how-we-designed-for-...
  - thrownaway2424 11y ago
    I think these can be and often are combined. You can definitely have an event driven main IO loop accepting connections and reading packets, that nevertheless dispatches whole requests onto their own threads. I work with a lot of services written that way. The fact that you can reuse the frontend for several applications amortizes the cost of developing it as a high performance asynchronous thing. As for stack allocation I definitely see your point there. It is bizarre that Linux defaults to 2MB or whatever it is these days. I usually work with a 64KB thread stack.
jcoffland 11y ago
This is exactly what I'm talking about. If you want to reach really high transactions rates then you cannot use one thread per transaction.
- thrownaway2424 11y ago
  I won't argue that async software thread per hardware thread style will have better maximum performance than a thread-per-request style, but I think that difference will not be material in most applications (the difference between the two styles of C++ server programming being far smaller than the difference between, say, a Go program and a C++ program), and the cost paid in code readability is pretty high. All that async continuation passing makes your code a twisty maze, and that style is fraught with lifetime problems that are the very thing most C++ haters are complaining about. In synchronous style all the allocations are scoped by nature. In async style the programmer has to assign ownership of everything to some master continuation that will delete them all in the final callback. That adds a lot of code.
  - jcoffland 11y ago
    You are right about the added difficulty of resource ownership in the face of asynchronous code. This is a trade off. lambda functions in C++11 can help with this. They make it possible to pass resource allocations (smart pointers) through to callbacks.
  - zzzcpan 11y ago
    > All that async continuation passing makes your code a twisty maze On the contrary, it gives your code clarity, consistency and predictability. > some master continuation that will delete them all in the final callback Even nginx in plain C is not doing that, but simply creates destructors on allocation to avoid tracking stuff manually. Just don't jump into the first thing you read somewhere, remember one simple rule: if it feels harder than threads - you are doing it wrong.
    - SamReidHughes 11y ago

zzzcpan 11y ago

Synchronous looking programming doesn't offer any clarity for anything beyond dumb sequential requests to a single node, that nobody cares about. Try building a distributed k/v store with that. It fails miserably once you have to deal with zero downtime restarts for continuous deployment, rate limiting, requests cancellations, waiting for one or multiple responses from multiple nodes, making more requests to other nodes just after a bit of time to guarantee latency, using a lot of different kinds of shared data, like response queues, log queues, connection caches, data caches, all used by multiple clients, while at the same time doing some synchronization of data between nodes, etc. Anything non-trivial is impossible with threads in practice.

thrownaway2424 11y ago
These statements are so cute. I can reasonably assure you that the distributed k/v store I have written in the thread-per-request style is currently out there serving more traffic than you've ever dreamed of. As for your specific points, I disagree totally. Each of those things is easier to me in the synchronous style than in the async style. Certainly something like hedging is way easier.
- CyberDildonics 11y ago
  Those are some extraordinary claims, but I'm not seeing an extraordinary evidence.
  - SamReidHughes 11y ago
    If you're talking to disk, you'll eventually be doing it synchronously anyway, because on Linux the API's for async disk I/O aren't (weren't?) all there yet. For some time I was working on a product with a command line flag to decide between using a worker pool of threads versus event-based disk I/O, and there wasn't a big enough performance difference (if any, I don't remember) for us to keep the event-based code. If you're getting some fancy PCI SSD's or persistent RAM and really want to minimize latency-to-fsync time, maybe things would be different. Also, if you use an event loop, you have to do scheduling yourself. For distributed key/value stores, good thread scheduling is even more important. It removes a big amount of technical risk to lean on the kernel's, unless your product is sufficiently minimalistic. Async will get you real performance advantages for pure networking, but if you're talking to disk it's, when not buggy, not very useful anyway. Also, as for programming style, even if you want event-based syscalls, cooperative userland-managed threads are a big win over callback hell. The only bad thing about them is wasting address-space on 32-bit systems, and maybe just wasting memory too.

ArkyBeagle 11y ago

Having things be designed as event-driven may provide more opportunity for something incrementally more akin to proof of correctness. Add a large quantum of salt and caveat that; it's simply slightly easier that way. People don't like it in general, but event driven works for me, mostly. Your mileage will almost certainly vary.