8 ms·

There's a lot of reasons I expected to see as their justification, but "our application can't handle concurrent requests" wasn't exactly one of them.

macspoofing 7y ago
That's an unfair assessment. HTTP/2 fundamentally changes how requests are handled. With HTTP/1.1 there is a defacto connection pool inside the browser and this throttling has been a feature of front-end development for 15+ years (from when ajax became a thing) so this wasn't something on anybody's mind. HTTP/2 all of sudden removes this constraints and for lucidchart, it led to a number of unintended consequences. This is an important consideration because the mantra has been that HTTP/2 can simply be turned on and everything will simply work as before.
- dstaley 7y ago
  If I'm reading this article correctly, they're claiming their application couldn't handle the load of a single-user loading their webpage. They didn't talk about load spikes during certain times, so it certainly sounds like they just have an inadequate backend.
  - crznp 7y ago
    On HTTP/2, Google says [1]: > all existing applications can be delivered without modification.... > The only observable differences will be improved performance and availability of new capabilities... Lucidcharts may have an inadequate backend, but it wasn't a problem until they moved to HTTP/2, so those statements weren't true for them. For anyone else rolling out HTTP/2, that is worth bearing in mind. [1]: https://developers.google.com/web/fundamentals/performance/http2/ https://developers.google.com/web/fundamentals/performance/h...
    - thayne 7y ago
      precisely
- Aaargh20318 7y ago
  > With HTTP/1.1 there is a defacto connection pool inside the browser and this throttling has been a feature of front-end development for 15+ years This is only true when you look at a single client. If you look at a larger number of clients accessing the service at the same time, you would expect similar numbers of concurrent requests on HTTP/2 as on HTTP/1.1. Clients send larger numbers of requests at the same time, but they are done sending requests earlier so there are requests from fewer clients being processed concurrently. It should average out. If you have, say, a 1000 clients accessing your service in one minute, I doubt the number of requests/second would be very different between both protocol versions. It would only be an issue if the service was built with a small number of concurrent users in mind.
  - macspoofing 7y ago
    >It should average out. But it didn't, unless you're saying that Lucidchart made an incorrect analysis. Is that your argument? >Clients send larger numbers of requests at the same time, but they are done sending requests earlier so there are requests from fewer clients being processed concurrently. It should average out. Again, it didn't average out. And you assume it 'will average out' at your peril. Maybe it will, maybe it won't. Lucidchart engineers thought that too and it turns out that was wrong in a way that wasn't foreseen. >It would only be an issue if the service was built with a small number of concurrent users in mind. I doubt Lucidchart 'was built with a small number of concurrent users in mind'.
    - Aaargh20318 7y ago
      > I doubt Lucidchart 'was built with a small number of concurrent users in mind'. This comment suggests otherwise: https://news.ycombinator.com/item?id=19722637 https://news.ycombinator.com/item?id=19722637
      - macspoofing 7y ago
        No it doesn't. Come on. lucidchart is a large service with many users.
jfoutz 7y ago
I have a lot of sympathy for their situation. I can imagine humming along just fine at 5 concurrent requests over 10 seconds, having busy days where you hit 10/10 But to suddenly hit 50 for 1 second, then nothing for 9 seconds, well, that’s a tough spot to be in. There must be some hard to find sequencing happening there, that they were not really exposed to before.
- deleted 7y ago
  [deleted]
- floatingatoll 7y ago
  The reverse is also quite stellar, because in some scenarios if you can lower peak burst demand per node from N to (0.1 x N) you can often reduce allocated capacity by a factor greater than 0.1. (This is more likely the case when N exceeds SOMAXCONN, for example.)
theandrewbailey 7y ago
There's also no indication on what kinds of requests were timing out, nor if it was possible to send fewer requests, or minimize static assets (if those were the problem).
da_chicken 7y ago
Do you think your local pizza joint could handle every order that they normally get between 4pm and 6pm if they got them all between 5pm and 5:30pm? At the same level of service, with no late orders, and with no additional equipment or labor?
- dstaley 7y ago
  This is more like if a pizza joint that could seat 10 people at once moved into a new location that could seat 100 people but still kept the same wait and kitchen staff and is suddenly surprised that wait times have increased.
- em-bee 7y ago
  they are not getting them at 5 but at 4. they should be able to handle all orders in the next two hours. the problem is not making the pizzas in time but trying to get all pizzas started at once when there is not enough table space to even roll out that much dough, and then trying to squeeze all the pizzas into the oven at once, whereby several of them got messed up.
  - Laforet 7y ago
    At full service resturants, it is the maître d′s duty to control the pace of orders so they arrive in a steady stream at the kitchen, instead of batches of 20 tickets at once that could easily overwhelm the chefs. The logic here is not dissimilar at all: if the backend has no ability to queue and prioritise the requests, then the same function needs to be done elsewhere to safeguard quality of service.
- ummonk 7y ago
  Well, with every pizza joint I’ve ordered from I can order a bunch of pizzas in just one phone call instead of having to make a separate phone call in serial for each pizza. And I certainly don’t have to wait for each pizza to be delivered before ordering another.
nine_k 7y ago
Rather, maybe, "our application can't handle short intense busts, and we haven't built a queuing proxy in front of it"? That speed observed by clients should come from somewhere. In their case, they did not have a large reserve of performance to tap.
coldtea 7y ago
Nice, because that wasn't the justification. The change in traffic patterns http/2 imposes was.
thayne 7y ago
Author here. Our application can handle concurrent requests just fine. The problem was actually that our application was partly that our application was trying to handle too many requests in parallel instead of queueing them, and partly that later requests were timing out because our load balancers were configured to expect clients to make a request, wait for the response then send the next request, not send all the requests, then wait for all the responses (which means the last response will take longer to complete from when the request was first sent).
- michaelt 7y ago
  Most places I've worked, the timeouts have been calibrated for people on 1990s dial-up, and the 99th-percentile response time targets have been 2 orders of magnitude less. Presumably that's not the case for you?
- mholt 7y ago
  So, your application can handle concurrent requests just fine, as long as clients make one request at a time?
  - deleted 7y ago
    [deleted]
  - thaumasiotes 7y ago
    I'm guessing that the distinction being drawn has to do with whether the application can handle concurrent requests for the same resource from different sessions on different computers (yes?), vs concurrent requests for every resource at once from a single session (no?, even though this is a much smaller number of requests).
  - floatingatoll 7y ago
    This misconstrues the comment it replies to. The post author's comment says that their application was tuned for a certain level of concurrency, and that when the level of concurrency to the load balancers increased due to the HTTP/2 change, their load balancers increased the level of concurrency to the backend, causing issues. This is an extremely common issue with Apache configurations, which often default to accepting hundreds of simultaneous requests without regard for memory capacity. If peak demand causes enough workers to spin up that Apache starts swapping, the entire server effectively goes offline. Depending on the specific characteristics of the application, this could occur when load increases from 50 concurrent requests to 51 concurrent requests, or from 200 to 201, or from any integer A to B where A was fine but B causes the server to become unresponsive. Saying that their A is 1 seems unnecessarily dour, given how common this problem has been over the past couple decades due to Apache's defaults alone.
geofft 7y ago
Whether or not that's a fair characterization of their justification, that seems like a perfectly defensible justification to me. Most people out there are dealing with legacy code (as in "code I didn't write") and business logic they can't safely rewrite, and it's more important to deliver business value than to build an emotionally-satisfying architecture. And even so, why design for concurrency when the high-level task (loading a web page in a bounded amount of time) doesn't require it? Personally, having a system that scales very well that doesn't need to scale isn't even an emotionally-satisfying architecture.
- Waterluvian 7y ago
  This is so true and is why I use "pure" functions where possible. Make it super obvious that a function can be replaced or reused safely and your future counterpart will praise your name to the code Gods.
- rmrfrmrf 7y ago
  Designing for concurrent requests is pretty much a given for web servers.
  - geofft 7y ago
    Designing software for concurrent requests, as in using poll/epoll/kqueue/IOCP, implementing worker pools, etc. is given, yes. Designing infrastructure for concurrent requests is definitely not. I've worked on shared hosting systems with high concurrency requirements and it definitely was more complicated than just installing an Apache MPM—we had to think about balancing load across multiple servers, whether virtualizing bare-metal machines into multiple VMs was worthwhile (in our case it was for a very site-specific reason), how many workers to run on each VM, how much memory we should expect to use for OS caching vs. application memory, how to trade off concurrent access to the same page vs. concurrent access to different pages vs. cold start of entirely new pages, whether dependencies like auth backends or SQL databases could handle concurrency and how much we needed to cache those, etc. At the end of the day you have a finite number of CPUs, a finite network pipe, and a finite amount of RAM. You can throw more money at many of these problems (although often not a SQL database) but you generally have a finite amount of money too. I would be surprised if most people had the infrastructure to handle significantly increased concurrency, even at the same throughput, as their current load. It's not a sensible thing to invest infrastructure budget into, most of the time. (You can, of course, solve this by developing software to actively limit concurrency. That's not a given for exactly the reasons that developing for concurrency is a given, and it sounds like Lucidchart didn't have that software and determined that switching back to HTTP/1.1 was as good as writing that software.)
paulddraper 7y ago
> There's a lot of reasons I expected to see but "our application can't handle concurrent requests" wasn't exactly one of them. Hence the blog post. Most would not think about the fact that your spikiness could increase 20x.

macspoofing 7y ago

That's an unfair assessment. HTTP/2 fundamentally changes how requests are handled. With HTTP/1.1 there is a defacto connection pool inside the browser and this throttling has been a feature of front-end development for 15+ years (from when ajax became a thing) so this wasn't something on anybody's mind. HTTP/2 all of sudden removes this constraints and for lucidchart, it led to a number of unintended consequences. This is an important consideration because the mantra has been that HTTP/2 can simply be turned on and everything will simply work as before.

dstaley 7y ago
If I'm reading this article correctly, they're claiming their application couldn't handle the load of a single-user loading their webpage. They didn't talk about load spikes during certain times, so it certainly sounds like they just have an inadequate backend.
- crznp 7y ago
  On HTTP/2, Google says [1]: > all existing applications can be delivered without modification.... > The only observable differences will be improved performance and availability of new capabilities... Lucidcharts may have an inadequate backend, but it wasn't a problem until they moved to HTTP/2, so those statements weren't true for them. For anyone else rolling out HTTP/2, that is worth bearing in mind. [1]: https://developers.google.com/web/fundamentals/performance/http2/ https://developers.google.com/web/fundamentals/performance/h...
  - thayne 7y ago
    precisely
Aaargh20318 7y ago
> With HTTP/1.1 there is a defacto connection pool inside the browser and this throttling has been a feature of front-end development for 15+ years This is only true when you look at a single client. If you look at a larger number of clients accessing the service at the same time, you would expect similar numbers of concurrent requests on HTTP/2 as on HTTP/1.1. Clients send larger numbers of requests at the same time, but they are done sending requests earlier so there are requests from fewer clients being processed concurrently. It should average out. If you have, say, a 1000 clients accessing your service in one minute, I doubt the number of requests/second would be very different between both protocol versions. It would only be an issue if the service was built with a small number of concurrent users in mind.
- macspoofing 7y ago
  >It should average out. But it didn't, unless you're saying that Lucidchart made an incorrect analysis. Is that your argument? >Clients send larger numbers of requests at the same time, but they are done sending requests earlier so there are requests from fewer clients being processed concurrently. It should average out. Again, it didn't average out. And you assume it 'will average out' at your peril. Maybe it will, maybe it won't. Lucidchart engineers thought that too and it turns out that was wrong in a way that wasn't foreseen. >It would only be an issue if the service was built with a small number of concurrent users in mind. I doubt Lucidchart 'was built with a small number of concurrent users in mind'.
  - Aaargh20318 7y ago
    > I doubt Lucidchart 'was built with a small number of concurrent users in mind'. This comment suggests otherwise: https://news.ycombinator.com/item?id=19722637 https://news.ycombinator.com/item?id=19722637
    - macspoofing 7y ago
      No it doesn't. Come on. lucidchart is a large service with many users.

jfoutz 7y ago

I have a lot of sympathy for their situation. I can imagine humming along just fine at 5 concurrent requests over 10 seconds, having busy days where you hit 10/10 But to suddenly hit 50 for 1 second, then nothing for 9 seconds, well, that’s a tough spot to be in. There must be some hard to find sequencing happening there, that they were not really exposed to before.

deleted 7y ago
[deleted]
floatingatoll 7y ago
The reverse is also quite stellar, because in some scenarios if you can lower peak burst demand per node from N to (0.1 x N) you can often reduce allocated capacity by a factor greater than 0.1. (This is more likely the case when N exceeds SOMAXCONN, for example.)

theandrewbailey 7y ago

There's also no indication on what kinds of requests were timing out, nor if it was possible to send fewer requests, or minimize static assets (if those were the problem).

da_chicken 7y ago

Do you think your local pizza joint could handle every order that they normally get between 4pm and 6pm if they got them all between 5pm and 5:30pm? At the same level of service, with no late orders, and with no additional equipment or labor?

dstaley 7y ago
This is more like if a pizza joint that could seat 10 people at once moved into a new location that could seat 100 people but still kept the same wait and kitchen staff and is suddenly surprised that wait times have increased.
em-bee 7y ago
they are not getting them at 5 but at 4. they should be able to handle all orders in the next two hours. the problem is not making the pizzas in time but trying to get all pizzas started at once when there is not enough table space to even roll out that much dough, and then trying to squeeze all the pizzas into the oven at once, whereby several of them got messed up.
- Laforet 7y ago
  At full service resturants, it is the maître d′s duty to control the pace of orders so they arrive in a steady stream at the kitchen, instead of batches of 20 tickets at once that could easily overwhelm the chefs. The logic here is not dissimilar at all: if the backend has no ability to queue and prioritise the requests, then the same function needs to be done elsewhere to safeguard quality of service.
ummonk 7y ago
Well, with every pizza joint I’ve ordered from I can order a bunch of pizzas in just one phone call instead of having to make a separate phone call in serial for each pizza. And I certainly don’t have to wait for each pizza to be delivered before ordering another.

nine_k 7y ago

Rather, maybe, "our application can't handle short intense busts, and we haven't built a queuing proxy in front of it"? That speed observed by clients should come from somewhere. In their case, they did not have a large reserve of performance to tap.

coldtea 7y ago

Nice, because that wasn't the justification. The change in traffic patterns http/2 imposes was.

thayne 7y ago

Author here. Our application can handle concurrent requests just fine. The problem was actually that our application was partly that our application was trying to handle too many requests in parallel instead of queueing them, and partly that later requests were timing out because our load balancers were configured to expect clients to make a request, wait for the response then send the next request, not send all the requests, then wait for all the responses (which means the last response will take longer to complete from when the request was first sent).

michaelt 7y ago
Most places I've worked, the timeouts have been calibrated for people on 1990s dial-up, and the 99th-percentile response time targets have been 2 orders of magnitude less. Presumably that's not the case for you?
mholt 7y ago
So, your application can handle concurrent requests just fine, as long as clients make one request at a time?
- deleted 7y ago
  [deleted]
- thaumasiotes 7y ago
  I'm guessing that the distinction being drawn has to do with whether the application can handle concurrent requests for the same resource from different sessions on different computers (yes?), vs concurrent requests for every resource at once from a single session (no?, even though this is a much smaller number of requests).
- floatingatoll 7y ago
  This misconstrues the comment it replies to. The post author's comment says that their application was tuned for a certain level of concurrency, and that when the level of concurrency to the load balancers increased due to the HTTP/2 change, their load balancers increased the level of concurrency to the backend, causing issues. This is an extremely common issue with Apache configurations, which often default to accepting hundreds of simultaneous requests without regard for memory capacity. If peak demand causes enough workers to spin up that Apache starts swapping, the entire server effectively goes offline. Depending on the specific characteristics of the application, this could occur when load increases from 50 concurrent requests to 51 concurrent requests, or from 200 to 201, or from any integer A to B where A was fine but B causes the server to become unresponsive. Saying that their A is 1 seems unnecessarily dour, given how common this problem has been over the past couple decades due to Apache's defaults alone.

geofft 7y ago

Whether or not that's a fair characterization of their justification, that seems like a perfectly defensible justification to me. Most people out there are dealing with legacy code (as in "code I didn't write") and business logic they can't safely rewrite, and it's more important to deliver business value than to build an emotionally-satisfying architecture. And even so, why design for concurrency when the high-level task (loading a web page in a bounded amount of time) doesn't require it? Personally, having a system that scales very well that doesn't need to scale isn't even an emotionally-satisfying architecture.

Waterluvian 7y ago
This is so true and is why I use "pure" functions where possible. Make it super obvious that a function can be replaced or reused safely and your future counterpart will praise your name to the code Gods.
rmrfrmrf 7y ago
Designing for concurrent requests is pretty much a given for web servers.
- geofft 7y ago
  Designing software for concurrent requests, as in using poll/epoll/kqueue/IOCP, implementing worker pools, etc. is given, yes. Designing infrastructure for concurrent requests is definitely not. I've worked on shared hosting systems with high concurrency requirements and it definitely was more complicated than just installing an Apache MPM—we had to think about balancing load across multiple servers, whether virtualizing bare-metal machines into multiple VMs was worthwhile (in our case it was for a very site-specific reason), how many workers to run on each VM, how much memory we should expect to use for OS caching vs. application memory, how to trade off concurrent access to the same page vs. concurrent access to different pages vs. cold start of entirely new pages, whether dependencies like auth backends or SQL databases could handle concurrency and how much we needed to cache those, etc. At the end of the day you have a finite number of CPUs, a finite network pipe, and a finite amount of RAM. You can throw more money at many of these problems (although often not a SQL database) but you generally have a finite amount of money too. I would be surprised if most people had the infrastructure to handle significantly increased concurrency, even at the same throughput, as their current load. It's not a sensible thing to invest infrastructure budget into, most of the time. (You can, of course, solve this by developing software to actively limit concurrency. That's not a given for exactly the reasons that developing for concurrency is a given, and it sounds like Lucidchart didn't have that software and determined that switching back to HTTP/1.1 was as good as writing that software.)

paulddraper 7y ago

> There's a lot of reasons I expected to see but "our application can't handle concurrent requests" wasn't exactly one of them. Hence the blog post. Most would not think about the fact that your spikiness could increase 20x.