4 ms·

Author here. Our application can handle concurrent requests just fine. The problem was actually that our application was partly that our application was trying

by thayne 8y ago

Author here. Our application can handle concurrent requests just fine. The problem was actually that our application was partly that our application was trying to handle too many requests in parallel instead of queueing them, and partly that later requests were timing out because our load balancers were configured to expect clients to make a request, wait for the response then send the next request, not send all the requests, then wait for all the responses (which means the last response will take longer to complete from when the request was first sent).

michaelt 8y ago
Most places I've worked, the timeouts have been calibrated for people on 1990s dial-up, and the 99th-percentile response time targets have been 2 orders of magnitude less. Presumably that's not the case for you?
mholt 8y ago
So, your application can handle concurrent requests just fine, as long as clients make one request at a time?
- deleted 8y ago
  [deleted]
- thaumasiotes 8y ago
  I'm guessing that the distinction being drawn has to do with whether the application can handle concurrent requests for the same resource from different sessions on different computers (yes?), vs concurrent requests for every resource at once from a single session (no?, even though this is a much smaller number of requests).
- floatingatoll 8y ago
  This misconstrues the comment it replies to. The post author's comment says that their application was tuned for a certain level of concurrency, and that when the level of concurrency to the load balancers increased due to the HTTP/2 change, their load balancers increased the level of concurrency to the backend, causing issues. This is an extremely common issue with Apache configurations, which often default to accepting hundreds of simultaneous requests without regard for memory capacity. If peak demand causes enough workers to spin up that Apache starts swapping, the entire server effectively goes offline. Depending on the specific characteristics of the application, this could occur when load increases from 50 concurrent requests to 51 concurrent requests, or from 200 to 201, or from any integer A to B where A was fine but B causes the server to become unresponsive. Saying that their A is 1 seems unnecessarily dour, given how common this problem has been over the past couple decades due to Apache's defaults alone.
  - cogman10 8y ago
    I've seen this mistake first hand. "API server not fast enough, how do we fix it?" "More threads! More connections!" The problem, of course, is that HTTP2 is behaves like having infinite connections, so the "more threads" on the server are almost always detrimental to performance. Less is more is the mantra I have unsuccessfully tried to drill. If your API (assuming a basic rest like service) is running at 100% cpu utilization, you've likely over provisioned it.
    - ec109685 8y ago
      If the api server is blocked on IO, more threads is a possibly fine solution.
      - cogman10 8y ago
        Possibly, just depends on what you are doing and where you are going from/to. For example, 200 threads with 200 connections to a single service is insane and likely causing you to be slow already. Increasing that will negatively impact performance. Going from 16 -> 32? That's more reasonable.
zikzak 8y ago
I don't want to get into it but I've dealt with issues like this and I don't think you are terrible or bad at your job. I'm seeing a lot of shade going your way. Sometimes a particular configuration works well until something changes. Just wanted to offer my two cents.
- ummonk 8y ago
  I think people are reacting badly because the title can create the impression that this is HTTP/2’s fault. The actual post seems perfectly reasonable though (essentially “you might think you can just turn on HTTP/2 as a drop in on your load balancer a but if your server code hasn’t been written to rapidly handle the quick bursts of requests that enable HTTP/2 to provide faster overall loads to the client then this can cause issues; you should test first and make sure your server systems are able to handle HTTP/2 request patterns.)
  - ohyeshedid 8y ago
    Somewhat understandable. I didn't get hung up on the title, and if anything the story is an object lesson in the need to familiarize yourself with the intricacies of inbound changes to your stack. I appreciate when people share war stories; I like to think that wisdom is knowledge survived.
jrochkind1 8y ago
I'm curious what platform your app is on, if you are willing to share? A typical non-tuned Rails deployment, for instance, is gonna have queueing built in, with really not as much concurrency as one would want (enough to actually fully utilize the host; the opposite problem). So I'm guessing you aren't on Rails. :) Curious what you are on, if you're willing to share, for how it effects concurrency defaults and options and affordances. (I know full well that properly configuring/tuning this kind of concurrency is not trivial or one-size-fits all. And I am not at all surprised that http/2 changed the parameters disastrously, and appreciate your warning to pay attention to it. I think those who are thinking "it shouldn't matter" are under-experienced or misinformed.)
- thayne 8y ago
  > I'm curious what platform your app is on, if you are willing to share? Sure. We use the Scala Play framework (https://www.playframework.com/ https://www.playframework.com/). And it does have some queuing built into, but we have tweaked it to meet certain application needs.
unscaled 8y ago
I assume your application is using a dedicated thread for each request? Even then you would be handling more requests in parallel than the number of cores you have, but your concurrency would be limited by the cost of context switching and your memory capacity (having to allocate a sizable stack for each thread in most threading implementations). Queueing is usually required for a stable multi-threaded server, but if you were doing async I/O you wouldn't need it. The extra memory overhead for each extra concurrent request (by means of lightweight coroutine stacks, callbacks or state machines) is not much different from the size it would take on the queue, and there is no preemptive context switching. In most cases, you'll get the same behavior as having a queue here. Cooperative task-switching happens only on async I/O boundaries, so if you're processing a request that requires some CPU-heavy work, your application would just hog a core until it completes the request and then move to the next one.
- j16sdiz 8y ago
  > get the same behavior as having a queue here. It is not so easy. The article said it timed out (on the client). When the queue is on client, the client start the timer when the request start.
- ec109685 8y ago
  Sizeable stacks don’t eat into memory unless your thread actually utilizes the full stack allocation. Otherwise, the memory is available for other uses, so you can spin up more threads than you’d think.

by thayne 8y ago

michaelt 8y ago

Most places I've worked, the timeouts have been calibrated for people on 1990s dial-up, and the 99th-percentile response time targets have been 2 orders of magnitude less. Presumably that's not the case for you?

mholt 8y ago

So, your application can handle concurrent requests just fine, as long as clients make one request at a time?

deleted 8y ago
[deleted]
thaumasiotes 8y ago
I'm guessing that the distinction being drawn has to do with whether the application can handle concurrent requests for the same resource from different sessions on different computers (yes?), vs concurrent requests for every resource at once from a single session (no?, even though this is a much smaller number of requests).
floatingatoll 8y ago
This misconstrues the comment it replies to. The post author's comment says that their application was tuned for a certain level of concurrency, and that when the level of concurrency to the load balancers increased due to the HTTP/2 change, their load balancers increased the level of concurrency to the backend, causing issues. This is an extremely common issue with Apache configurations, which often default to accepting hundreds of simultaneous requests without regard for memory capacity. If peak demand causes enough workers to spin up that Apache starts swapping, the entire server effectively goes offline. Depending on the specific characteristics of the application, this could occur when load increases from 50 concurrent requests to 51 concurrent requests, or from 200 to 201, or from any integer A to B where A was fine but B causes the server to become unresponsive. Saying that their A is 1 seems unnecessarily dour, given how common this problem has been over the past couple decades due to Apache's defaults alone.
- cogman10 8y ago
  I've seen this mistake first hand. "API server not fast enough, how do we fix it?" "More threads! More connections!" The problem, of course, is that HTTP2 is behaves like having infinite connections, so the "more threads" on the server are almost always detrimental to performance. Less is more is the mantra I have unsuccessfully tried to drill. If your API (assuming a basic rest like service) is running at 100% cpu utilization, you've likely over provisioned it.
  - ec109685 8y ago
    If the api server is blocked on IO, more threads is a possibly fine solution.
    - cogman10 8y ago
      Possibly, just depends on what you are doing and where you are going from/to. For example, 200 threads with 200 connections to a single service is insane and likely causing you to be slow already. Increasing that will negatively impact performance. Going from 16 -> 32? That's more reasonable.

zikzak 8y ago

I don't want to get into it but I've dealt with issues like this and I don't think you are terrible or bad at your job. I'm seeing a lot of shade going your way. Sometimes a particular configuration works well until something changes. Just wanted to offer my two cents.

ummonk 8y ago
I think people are reacting badly because the title can create the impression that this is HTTP/2’s fault. The actual post seems perfectly reasonable though (essentially “you might think you can just turn on HTTP/2 as a drop in on your load balancer a but if your server code hasn’t been written to rapidly handle the quick bursts of requests that enable HTTP/2 to provide faster overall loads to the client then this can cause issues; you should test first and make sure your server systems are able to handle HTTP/2 request patterns.)
- ohyeshedid 8y ago
  Somewhat understandable. I didn't get hung up on the title, and if anything the story is an object lesson in the need to familiarize yourself with the intricacies of inbound changes to your stack. I appreciate when people share war stories; I like to think that wisdom is knowledge survived.

jrochkind1 8y ago

I'm curious what platform your app is on, if you are willing to share? A typical non-tuned Rails deployment, for instance, is gonna have queueing built in, with really not as much concurrency as one would want (enough to actually fully utilize the host; the opposite problem). So I'm guessing you aren't on Rails. :) Curious what you are on, if you're willing to share, for how it effects concurrency defaults and options and affordances. (I know full well that properly configuring/tuning this kind of concurrency is not trivial or one-size-fits all. And I am not at all surprised that http/2 changed the parameters disastrously, and appreciate your warning to pay attention to it. I think those who are thinking "it shouldn't matter" are under-experienced or misinformed.)

thayne 8y ago
> I'm curious what platform your app is on, if you are willing to share? Sure. We use the Scala Play framework (https://www.playframework.com/ https://www.playframework.com/). And it does have some queuing built into, but we have tweaked it to meet certain application needs.

unscaled 8y ago

I assume your application is using a dedicated thread for each request? Even then you would be handling more requests in parallel than the number of cores you have, but your concurrency would be limited by the cost of context switching and your memory capacity (having to allocate a sizable stack for each thread in most threading implementations). Queueing is usually required for a stable multi-threaded server, but if you were doing async I/O you wouldn't need it. The extra memory overhead for each extra concurrent request (by means of lightweight coroutine stacks, callbacks or state machines) is not much different from the size it would take on the queue, and there is no preemptive context switching. In most cases, you'll get the same behavior as having a queue here. Cooperative task-switching happens only on async I/O boundaries, so if you're processing a request that requires some CPU-heavy work, your application would just hog a core until it completes the request and then move to the next one.

j16sdiz 8y ago
> get the same behavior as having a queue here. It is not so easy. The article said it timed out (on the client). When the queue is on client, the client start the timer when the request start.
ec109685 8y ago
Sizeable stacks don’t eat into memory unless your thread actually utilizes the full stack allocation. Otherwise, the memory is available for other uses, so you can spin up more threads than you’d think.