31 ms·

Why Turning on HTTP/2 Was a Mistake

277 points by sciurus 7y ago

manigandham 7y ago
Unless they're making hundreds of requests per visitor I fail to see how the load is shifted so drastically to make such an impact.
- layoutIfNeeded 7y ago
  Spoiler: they are making hundreds of requests per visitor. Welcome to “modern” webdev!
  - macspoofing 7y ago
    What's wrong with that? Lucidchart is a fully-featured application that runs in a web browser with a cloud-backend. I don't understand what you're trying to argue. That you have to optimize for request count? Why?
    - DCoder 7y ago
      There's an argument to be made that this [0] (generated via [1]) is not something to be celebrated. But that's just me, not necessarily what the parent poster had in mind. [0]: https://i.imgur.com/LhEahvi.png https://i.imgur.com/LhEahvi.png [1]: https://www.evidon.com/solutions/trackermap/ https://www.evidon.com/solutions/trackermap/
      - layoutIfNeeded 7y ago
        Exactly.
      - macspoofing 7y ago
        This is what's known as 'moving the goalpost'. OP never mentioned that their issues with lucidchart were due to them using ad trackers and analytics libraries. They talked about how terrible it was that an application made a ton of requests. If they have an issue with the former, I take their point.
    - cmoscoso 7y ago
      Because it is a web browser with a cloud-backend?
andersonrkton 7y ago
Cloudflared <3
iforgotpassword 7y ago
This is a neat little writeup. Although the issues were not fundamental and relatively easy to spot and fix, it's valuable input especially since http/2 advocates seem to insist that you just need to put your webapp behind a http/2 capable proxy and you won't even notice a difference. We didn't enable it yet on our servers and now there's definitely something to test first before we roll out.
dstaley 7y ago
There's a lot of reasons I expected to see as their justification, but "our application can't handle concurrent requests" wasn't exactly one of them.
- macspoofing 7y ago
  That's an unfair assessment. HTTP/2 fundamentally changes how requests are handled. With HTTP/1.1 there is a defacto connection pool inside the browser and this throttling has been a feature of front-end development for 15+ years (from when ajax became a thing) so this wasn't something on anybody's mind. HTTP/2 all of sudden removes this constraints and for lucidchart, it led to a number of unintended consequences. This is an important consideration because the mantra has been that HTTP/2 can simply be turned on and everything will simply work as before.
  - dstaley 7y ago
    If I'm reading this article correctly, they're claiming their application couldn't handle the load of a single-user loading their webpage. They didn't talk about load spikes during certain times, so it certainly sounds like they just have an inadequate backend.
    - crznp 7y ago
      On HTTP/2, Google says [1]: > all existing applications can be delivered without modification.... > The only observable differences will be improved performance and availability of new capabilities... Lucidcharts may have an inadequate backend, but it wasn't a problem until they moved to HTTP/2, so those statements weren't true for them. For anyone else rolling out HTTP/2, that is worth bearing in mind. [1]: https://developers.google.com/web/fundamentals/performance/http2/ https://developers.google.com/web/fundamentals/performance/h...
      - thayne 7y ago
        precisely
stuff4ben 7y ago
We experienced issues when we enabled H2 on our HAProxy 1.8 reverse proxies into our K8s cluster. Didn't anticipate the increased memory consumption and we ran into a few memory-related defects with older versions of HAProxy that were fixed with more recent versions. We'll re-enable it at some point, but we've upgraded our reverse proxies in anticipation of it.
StreamBright 7y ago
Isn’t it happening because the load balancer does not distribute the http requests evenly? We used to use advanced loadbalancers that took the actual http req from a client and used fix number or tcp connections to backend and distributed the http requests through those. Maybe http/2 does not allow this style of load balancing?
- jrockway 7y ago
  HTTP/2 does allow this style of load balancing. Whether or not your load balancer does it, however, is another thing completely.
jrockway 7y ago
This is why Envoy exists. It will take HTTP/2 requests from the user and shard the actual requests out for backends to handle. It appears that what happened to the author is that their web server only balanced TCP connections, which indeed no longer works.
- thayne 7y ago
  We were using AWS ALBs which load balance requests not connections (although it should be noted they do not handle HTTP/2 prioritization). j
  - jrockway 7y ago
    I see. So it sounds like the issue was one of timing, where a bunch of converted HTTP/1.1 requests all arrived at your application at the exact same instant. Did the ALB open a new TCP connection for each request, or does it use a pool of connections?
    - thayne 7y ago
      I think it opens a new connection for each request.
zxcvbn4038 7y ago
The article is short on detail but it sounds to me like they are balancing their traffic by connection instead of by request. Either nginx or haproxy should be able to spread those multiplexed requests across a number of servers and give more the desired backend behavior.
- lxe 7y ago
  Unless the application/request/session state is pinned to a host.
tantalic 7y ago
The underlying lesson that I have learned (the hard way) repeatedly: anything that may change the traffic pattern can result in difficult to predict infrastructure issues. This can be as related as a changing protocol (as shown here) or a seemingly unrelated like a UI change.
deathanatos 7y ago
Really, this is an issue in the library/server: the library/server needs to expose HTTP/2's controls on maximum permitted streams. > And secondly, because with HTTP/2, the requests were all sent together—instead of staggered like they were with HTTP/1.1—so their start times were closer together, which meant they were all likely to time out. No, browsers can pipeline requests (send the requests back-to-back, without first waiting for a response) in HTTP/1.1. The server has to send the responses in order, but it doesn't have to process them in that order if it is willing to buffer the later responses in the case of head-of-line blocking. Honestly, over the long run, this is a feature, not a bug. The server and client can make better use of resources by not having a trivial CSS or JS request waiting on a request that's blocked on a slow DB call. Yes, you shouldn't overload your own server, but that's a matter of not trying to process a large flood all simultaneously. (Or, IDK, maybe do, and just let the OS scheduler deal with it.) Also, if you don't want a ton of requests… don't have a gajillion CSS/JS/webfont for privacy devouring ad networks? It takes 99 requests and 3.1 MB (before decompression) to load lucidchart.com. > If you do queue requests, you should be careful not to process requests after the client has timed out waiting for a response This is a real problem, but I've suffered through that plenty with synchronous HTTP/1.1 servers; a thread blocks, but it's still got other requests buffered, sometimes from that connection, sometimes from others. Good async frameworks can handle these better, but they typically require some form of cancellation, and my understanding is that that's notably absent from JavaScript & Go's async primitives.
- toast0 7y ago
  > No, browsers can pipeline requests (send the requests back-to-back, without first waiting for a response) in HTTP/1.1. The server has to send the responses in order, but it doesn't have to process them in that order if it is willing to buffer the later responses in the case of head-of-line blocking. Browsers can pipeline requests on http/1.1, but I don't think any of them actually do in today's world, at least that's what MDN says. [1] And from my recollection, very few browsers did pipelining prior to http/2 either -- the chances of running into something broken were much too high. [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Connection_management_in_HTTP_1.x#HTTP_pipelining https://developer.mozilla.org/en-US/docs/Web/HTTP/Connection...
bastawhiz 7y ago
How many requests does your page make on initial load (that can't be handled by a CDN)? If you're making more than six XHRs to your application servers concurrently, this sounds like a problem that would have existed anyway had it not been for the browser's (rather arbitrary) connection limit. It's also curious to me that the load balancer doesn't smooth this out. If you have ten application servers and a client makes ten requests over a single HTTP/2 connection, I'd expect each server to respond to one request each. The details are a little fuzzy, but it sounds like the load balancer is only distributing connections, not requests. That seems wrong. High CPU load should be fine, really, if your application servers are processing requests. If the load is unbalanced, then by definition you need a load balancer to balance the load. If you have one and the load is unbalanced, something is misconfigured.
- adventured 7y ago
  > The details are a little fuzzy, but it sounds like the load balancer is only distributing connections, not requests. That seems wrong. They may be using sticky sessions or affinity in some regard, having the load balancer hold each client connection intentionally to a server. It's not necessarily wrong, entirely depends on what you need to accomplish.
  - bastawhiz 7y ago
    If they had been using sticky sessions, this should have been a problem before, but to a lesser extent. The same server would have needed to process all of the requests for a single client. You'd still have spikey metrics. This might not be such a problem with one client artificially limited to a single application server. But in practice, it means that individual servers will be overloaded when they are chosen to handle multiple clients concurrently (while other servers are idle).
- thayne 7y ago
  Author here. > How many requests does your page make on initial load (that can't be handled by a CDN)? If you're making more than six XHRs to your application servers concurrently, this sounds like a problem that would have existed anyway had it not been for the browser's (rather arbitrary) connection limit. I don't know the exact number, but definitely higher than six. And I certainly agree that is a failing in our application. The point is that the browserd _did_ arbitrarily limit connections, and our application (unknowingly) depended on that.
gregoriol 7y ago
What I understand is that their setup/app was broken but mitigated by HTTP/1 which is not that efficient?
schmichael 7y ago
I think a glib attempt at a tl;dr would be: > Sometimes "thundering herd" is a feature, not a bug. To expand: HTTP/1.1 naturally caused the latency of the Internet to pace requests. A sort of implicit intrinsic rate limiting. HTTP/2 intentionally avoids that "problem" by batching/pipelining requests.
- j16sdiz 7y ago
  There is a client-imposed limit. In chrome, it is 8 for http/1.1 and 1000 for http/2
thayne 7y ago
Author here. Let me clarify a couple things I've seen in a lot of the comments. First of all, we do load balance by request, not connections and we do not use sticky sessions. Secondly, we are aware that our application has underlying problems with large numbers of concurrent requests, which was exposed by using HTTP/2 and we are working on fixing them. The point of the post is not "HTTP/2 sucks because it broke our app". It is "moving to HTTP/2 isn't necessarily as easy as just flipping switch".
- otterley 7y ago
  Perhaps the title, then, should have been "Why Turning on HTTP/2 Was a Mistake (For Us)" so as not to imply that doing so is a universally bad idea.
  - draw_down 7y ago
    It doesn't imply that, be honest.
  - geofft 7y ago
    I think it was clear enough - it didn't say "Why Turning On HTTP/2 Is A Mistake" or "Why HTTP/2 Was A Mistake," either which would have implied universal badness. The phrasing had me expecting a specific war story, which is what it ended up being. Besides, I think it's quite reasonable to argue that it's a majority bad idea, even if it's not a universally bad idea. I think many people are probably in the same boat.
    - otterley 7y ago
      Without data, one cannot say either way. What we have before us is a single anecdotal tale. And even that tale, even if you believe it rises to the level of "data," doesn’t provide clear guidance to others, because the details of their stack and capacity are unspecified.
      - titanomachy 7y ago
        Do you mean "anecdotal" rather than "apocryphal"? Apocryphal suggests you have reason to believe this story is untrue.
iamleppert 7y ago
It’s hard to tell because there isn’t anything concrete provided. That said, if this is a traditional web app, this smells to me of a poorly designed application. It sounds like they’re doing on the fly compilation of static assets or something crazy like that, and in any event need to reduce the total number of requests per page or resource and look for opportunities to make things static or cached?
xmichael999 7y ago
Not sure what the authors application is, but we run a dozen servers behind http/2 load balanced and dozens of sites and haven't seen anything similar to what he is describing.
- thayne 7y ago
  Author here. Our application is Lucidchart (www.lucidchart.com). It is a very sophisticated web-app with significant amounts of dynamic data,running on hundreds of servers. I would imagine applications with less dynamic data and requests that require substantial amount of compute wouldn't run into this problem.
thecompilr 7y ago
There is a setting in HTTP/2 called SETTINGS_MAX_CONCURRENT_STREAMS, if set to 1 it works like HTTP/1.1, with no multiplexing. Setting it to 4~8 would make it behave in a similar way a browser actually does with HTTP/1 (creating multiple connections in parallel).
- j16sdiz 7y ago
  This is a client side setting and default to 1000 in chrome. The http/1.1 equivalent was 8 or something like that.
  - floatingatoll 7y ago
    While it is correct to say that it is a client setting, it is also a server setting. The HTTP/2 specification uses the word "peer" since SETTINGS_MAX_CONCURRENT_STREAMS (0x3) is negotiated in both directions as part of the client/server handshakes. $ nghttp -v https://www.google.com | grep -C5 SETTINGS_MAX_CONCURRENT_STREAMS [ 0.076] send SETTINGS frame <length=12, flags=0x00, stream_id=0> [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100] [ 0.091] recv SETTINGS frame <length=18, flags=0x00, stream_id=0> [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
  - ec109685 7y ago
    ALB clips this at 128: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-listeners.html https://docs.aws.amazon.com/elasticloadbalancing/latest/appl...
Izmaki 7y ago
I was hoping to experience this as a lucidchart visualisation of "sweaty spikes" and "work spreaders" because of too many "Internet tubes". Oh well. Another time :D
melan13 7y ago
This is where micro-services shine.
- macspoofing 7y ago
  Why? This has nothing to do with microservices.
  - melan13 7y ago
    It does, think twice about the CPU flow.
    - macspoofing 7y ago
      Uh huh. What is this, exercise for the reader? I suspect you're just saying things to cover for the fact that you don't have anything meaningful to say.
emmelaich 7y ago
It's actually a somewhat difficult thing to predict spikes. Say you have n clients, d timeslices. When does the probability exceed 0.5 that you get more than k requests concurrently? Unfortunately the solution is exponential in k. People might recognise the birthday problem here; for d=365, k=2 (days a year, single share) the well known answer is 23. Wikipedia gives a formula for a rough approximation for n for p=0.5 and k < 20.
nhumrich 7y ago
Do you terminate http/2 at the load balancer and convert it to http1.1? Or do you support http/2 all the way to the end service? I would imagine the former would solve these issues.
- unscaled 7y ago
  If your service can't handle well a bunch of requests coming at once, it doesn't matter if it gets them as individual HTTP/1.1 requests or as multiplexed requests in HTTP/2 coming directly. It only makes a difference if the bottleneck is HTTP/1.1 parsing logic.
stevefan1999 7y ago
First of all, turning on HTTP/2 was not a mistake. Second. turning it on when it is not mature yet is.
- floatingatoll 7y ago
  The maturity of HTTP/2 is not a causative factor here. They removed a previously-unaware limit on the number of concurrent backend requests, which overflowed their backends. They could have experienced this same outage by simply removing that limit without enabling HTTP/2, and then hitting a peak demand period that was sufficient to cause the outage. Yes, HTTP/2 changes traffic patterns, but the issue could easily have occurred with HTTP/1 as well.
StopHammoTime 7y ago
Great article on approaching massive technical change. Honestly, I think a lot of people general think that most things are just a "switch flip". Even something has implementing SSL on internal apps can cause a big change, let alone the underlying protocol for managing your requests. Thanks for this, because honestly I hadn't thought about the implications myself and it'd be good not to accidentally walk into this problem.
tie_ 7y ago
Surely your application/serving process is able to handle the request burst coming from any single user ? (If not, you have a bigger problem to solver first). If so, I don't quite see why queueing is discussed as an option at all. Queueing means extra latency and worse user experience (not to mention DoS potential). What you should be discussing instead is how to (auto-)scale your app and infrastructure to handle your users' requests.
KaiserPro 7y ago
> decreases latency by multiplexing requests on the same TCP connection On a decent connection, kinda. Anything mobile or worse mobile and moving will suffer terribly.
EugeneOZ 7y ago
Pathetic attempt to get some users from HN.

277 points by sciurus 7y ago

Unless they're making hundreds of requests per visitor I fail to see how the load is shifted so drastically to make such an impact.

layoutIfNeeded 7y ago
Spoiler: they are making hundreds of requests per visitor. Welcome to “modern” webdev!
- macspoofing 7y ago
  What's wrong with that? Lucidchart is a fully-featured application that runs in a web browser with a cloud-backend. I don't understand what you're trying to argue. That you have to optimize for request count? Why?
  - DCoder 7y ago
    There's an argument to be made that this [0] (generated via [1]) is not something to be celebrated. But that's just me, not necessarily what the parent poster had in mind. [0]: https://i.imgur.com/LhEahvi.png https://i.imgur.com/LhEahvi.png [1]: https://www.evidon.com/solutions/trackermap/ https://www.evidon.com/solutions/trackermap/
    - layoutIfNeeded 7y ago
      Exactly.
    - macspoofing 7y ago
      This is what's known as 'moving the goalpost'. OP never mentioned that their issues with lucidchart were due to them using ad trackers and analytics libraries. They talked about how terrible it was that an application made a ton of requests. If they have an issue with the former, I take their point.
  - cmoscoso 7y ago
    Because it is a web browser with a cloud-backend?

andersonrkton 7y ago

Cloudflared <3

iforgotpassword 7y ago

This is a neat little writeup. Although the issues were not fundamental and relatively easy to spot and fix, it's valuable input especially since http/2 advocates seem to insist that you just need to put your webapp behind a http/2 capable proxy and you won't even notice a difference. We didn't enable it yet on our servers and now there's definitely something to test first before we roll out.

dstaley 7y ago

There's a lot of reasons I expected to see as their justification, but "our application can't handle concurrent requests" wasn't exactly one of them.

macspoofing 7y ago
That's an unfair assessment. HTTP/2 fundamentally changes how requests are handled. With HTTP/1.1 there is a defacto connection pool inside the browser and this throttling has been a feature of front-end development for 15+ years (from when ajax became a thing) so this wasn't something on anybody's mind. HTTP/2 all of sudden removes this constraints and for lucidchart, it led to a number of unintended consequences. This is an important consideration because the mantra has been that HTTP/2 can simply be turned on and everything will simply work as before.
- dstaley 7y ago
  If I'm reading this article correctly, they're claiming their application couldn't handle the load of a single-user loading their webpage. They didn't talk about load spikes during certain times, so it certainly sounds like they just have an inadequate backend.
  - crznp 7y ago
    On HTTP/2, Google says [1]: > all existing applications can be delivered without modification.... > The only observable differences will be improved performance and availability of new capabilities... Lucidcharts may have an inadequate backend, but it wasn't a problem until they moved to HTTP/2, so those statements weren't true for them. For anyone else rolling out HTTP/2, that is worth bearing in mind. [1]: https://developers.google.com/web/fundamentals/performance/http2/ https://developers.google.com/web/fundamentals/performance/h...
    - thayne 7y ago
      precisely

stuff4ben 7y ago

We experienced issues when we enabled H2 on our HAProxy 1.8 reverse proxies into our K8s cluster. Didn't anticipate the increased memory consumption and we ran into a few memory-related defects with older versions of HAProxy that were fixed with more recent versions. We'll re-enable it at some point, but we've upgraded our reverse proxies in anticipation of it.

StreamBright 7y ago

Isn’t it happening because the load balancer does not distribute the http requests evenly? We used to use advanced loadbalancers that took the actual http req from a client and used fix number or tcp connections to backend and distributed the http requests through those. Maybe http/2 does not allow this style of load balancing?

jrockway 7y ago
HTTP/2 does allow this style of load balancing. Whether or not your load balancer does it, however, is another thing completely.

jrockway 7y ago

This is why Envoy exists. It will take HTTP/2 requests from the user and shard the actual requests out for backends to handle. It appears that what happened to the author is that their web server only balanced TCP connections, which indeed no longer works.

thayne 7y ago
We were using AWS ALBs which load balance requests not connections (although it should be noted they do not handle HTTP/2 prioritization). j
- jrockway 7y ago
  I see. So it sounds like the issue was one of timing, where a bunch of converted HTTP/1.1 requests all arrived at your application at the exact same instant. Did the ALB open a new TCP connection for each request, or does it use a pool of connections?
  - thayne 7y ago
    I think it opens a new connection for each request.

zxcvbn4038 7y ago

The article is short on detail but it sounds to me like they are balancing their traffic by connection instead of by request. Either nginx or haproxy should be able to spread those multiplexed requests across a number of servers and give more the desired backend behavior.

lxe 7y ago
Unless the application/request/session state is pinned to a host.

tantalic 7y ago

The underlying lesson that I have learned (the hard way) repeatedly: anything that may change the traffic pattern can result in difficult to predict infrastructure issues. This can be as related as a changing protocol (as shown here) or a seemingly unrelated like a UI change.

deathanatos 7y ago

Really, this is an issue in the library/server: the library/server needs to expose HTTP/2's controls on maximum permitted streams. > And secondly, because with HTTP/2, the requests were all sent together—instead of staggered like they were with HTTP/1.1—so their start times were closer together, which meant they were all likely to time out. No, browsers can pipeline requests (send the requests back-to-back, without first waiting for a response) in HTTP/1.1. The server has to send the responses in order, but it doesn't have to process them in that order if it is willing to buffer the later responses in the case of head-of-line blocking. Honestly, over the long run, this is a feature, not a bug. The server and client can make better use of resources by not having a trivial CSS or JS request waiting on a request that's blocked on a slow DB call. Yes, you shouldn't overload your own server, but that's a matter of not trying to process a large flood all simultaneously. (Or, IDK, maybe do, and just let the OS scheduler deal with it.) Also, if you don't want a ton of requests… don't have a gajillion CSS/JS/webfont for privacy devouring ad networks? It takes 99 requests and 3.1 MB (before decompression) to load lucidchart.com. > If you do queue requests, you should be careful not to process requests after the client has timed out waiting for a response This is a real problem, but I've suffered through that plenty with synchronous HTTP/1.1 servers; a thread blocks, but it's still got other requests buffered, sometimes from that connection, sometimes from others. Good async frameworks can handle these better, but they typically require some form of cancellation, and my understanding is that that's notably absent from JavaScript & Go's async primitives.

toast0 7y ago
> No, browsers can pipeline requests (send the requests back-to-back, without first waiting for a response) in HTTP/1.1. The server has to send the responses in order, but it doesn't have to process them in that order if it is willing to buffer the later responses in the case of head-of-line blocking. Browsers can pipeline requests on http/1.1, but I don't think any of them actually do in today's world, at least that's what MDN says. [1] And from my recollection, very few browsers did pipelining prior to http/2 either -- the chances of running into something broken were much too high. [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Connection_management_in_HTTP_1.x#HTTP_pipelining https://developer.mozilla.org/en-US/docs/Web/HTTP/Connection...

bastawhiz 7y ago

How many requests does your page make on initial load (that can't be handled by a CDN)? If you're making more than six XHRs to your application servers concurrently, this sounds like a problem that would have existed anyway had it not been for the browser's (rather arbitrary) connection limit. It's also curious to me that the load balancer doesn't smooth this out. If you have ten application servers and a client makes ten requests over a single HTTP/2 connection, I'd expect each server to respond to one request each. The details are a little fuzzy, but it sounds like the load balancer is only distributing connections, not requests. That seems wrong. High CPU load should be fine, really, if your application servers are processing requests. If the load is unbalanced, then by definition you need a load balancer to balance the load. If you have one and the load is unbalanced, something is misconfigured.

adventured 7y ago
> The details are a little fuzzy, but it sounds like the load balancer is only distributing connections, not requests. That seems wrong. They may be using sticky sessions or affinity in some regard, having the load balancer hold each client connection intentionally to a server. It's not necessarily wrong, entirely depends on what you need to accomplish.
- bastawhiz 7y ago
  If they had been using sticky sessions, this should have been a problem before, but to a lesser extent. The same server would have needed to process all of the requests for a single client. You'd still have spikey metrics. This might not be such a problem with one client artificially limited to a single application server. But in practice, it means that individual servers will be overloaded when they are chosen to handle multiple clients concurrently (while other servers are idle).
thayne 7y ago
Author here. > How many requests does your page make on initial load (that can't be handled by a CDN)? If you're making more than six XHRs to your application servers concurrently, this sounds like a problem that would have existed anyway had it not been for the browser's (rather arbitrary) connection limit. I don't know the exact number, but definitely higher than six. And I certainly agree that is a failing in our application. The point is that the browserd _did_ arbitrarily limit connections, and our application (unknowingly) depended on that.

gregoriol 7y ago

What I understand is that their setup/app was broken but mitigated by HTTP/1 which is not that efficient?

schmichael 7y ago

I think a glib attempt at a tl;dr would be: > Sometimes "thundering herd" is a feature, not a bug. To expand: HTTP/1.1 naturally caused the latency of the Internet to pace requests. A sort of implicit intrinsic rate limiting. HTTP/2 intentionally avoids that "problem" by batching/pipelining requests.

j16sdiz 7y ago
There is a client-imposed limit. In chrome, it is 8 for http/1.1 and 1000 for http/2

thayne 7y ago

Author here. Let me clarify a couple things I've seen in a lot of the comments. First of all, we do load balance by request, not connections and we do not use sticky sessions. Secondly, we are aware that our application has underlying problems with large numbers of concurrent requests, which was exposed by using HTTP/2 and we are working on fixing them. The point of the post is not "HTTP/2 sucks because it broke our app". It is "moving to HTTP/2 isn't necessarily as easy as just flipping switch".

otterley 7y ago
Perhaps the title, then, should have been "Why Turning on HTTP/2 Was a Mistake (For Us)" so as not to imply that doing so is a universally bad idea.
- draw_down 7y ago
  It doesn't imply that, be honest.
- geofft 7y ago
  I think it was clear enough - it didn't say "Why Turning On HTTP/2 Is A Mistake" or "Why HTTP/2 Was A Mistake," either which would have implied universal badness. The phrasing had me expecting a specific war story, which is what it ended up being. Besides, I think it's quite reasonable to argue that it's a majority bad idea, even if it's not a universally bad idea. I think many people are probably in the same boat.
  - otterley 7y ago
    Without data, one cannot say either way. What we have before us is a single anecdotal tale. And even that tale, even if you believe it rises to the level of "data," doesn’t provide clear guidance to others, because the details of their stack and capacity are unspecified.
    - titanomachy 7y ago
      Do you mean "anecdotal" rather than "apocryphal"? Apocryphal suggests you have reason to believe this story is untrue.

iamleppert 7y ago

It’s hard to tell because there isn’t anything concrete provided. That said, if this is a traditional web app, this smells to me of a poorly designed application. It sounds like they’re doing on the fly compilation of static assets or something crazy like that, and in any event need to reduce the total number of requests per page or resource and look for opportunities to make things static or cached?

xmichael999 7y ago

Not sure what the authors application is, but we run a dozen servers behind http/2 load balanced and dozens of sites and haven't seen anything similar to what he is describing.

thayne 7y ago
Author here. Our application is Lucidchart (www.lucidchart.com). It is a very sophisticated web-app with significant amounts of dynamic data,running on hundreds of servers. I would imagine applications with less dynamic data and requests that require substantial amount of compute wouldn't run into this problem.

thecompilr 7y ago

There is a setting in HTTP/2 called SETTINGS_MAX_CONCURRENT_STREAMS, if set to 1 it works like HTTP/1.1, with no multiplexing. Setting it to 4~8 would make it behave in a similar way a browser actually does with HTTP/1 (creating multiple connections in parallel).

j16sdiz 7y ago
This is a client side setting and default to 1000 in chrome. The http/1.1 equivalent was 8 or something like that.
- floatingatoll 7y ago
  While it is correct to say that it is a client setting, it is also a server setting. The HTTP/2 specification uses the word "peer" since SETTINGS_MAX_CONCURRENT_STREAMS (0x3) is negotiated in both directions as part of the client/server handshakes. $ nghttp -v https://www.google.com | grep -C5 SETTINGS_MAX_CONCURRENT_STREAMS [ 0.076] send SETTINGS frame <length=12, flags=0x00, stream_id=0> [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100] [ 0.091] recv SETTINGS frame <length=18, flags=0x00, stream_id=0> [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
- ec109685 7y ago
  ALB clips this at 128: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-listeners.html https://docs.aws.amazon.com/elasticloadbalancing/latest/appl...

Izmaki 7y ago

I was hoping to experience this as a lucidchart visualisation of "sweaty spikes" and "work spreaders" because of too many "Internet tubes". Oh well. Another time :D

melan13 7y ago

This is where micro-services shine.

macspoofing 7y ago
Why? This has nothing to do with microservices.
- melan13 7y ago
  It does, think twice about the CPU flow.
  - macspoofing 7y ago
    Uh huh. What is this, exercise for the reader? I suspect you're just saying things to cover for the fact that you don't have anything meaningful to say.

emmelaich 7y ago

It's actually a somewhat difficult thing to predict spikes. Say you have n clients, d timeslices. When does the probability exceed 0.5 that you get more than k requests concurrently? Unfortunately the solution is exponential in k. People might recognise the birthday problem here; for d=365, k=2 (days a year, single share) the well known answer is 23. Wikipedia gives a formula for a rough approximation for n for p=0.5 and k < 20.

nhumrich 7y ago

Do you terminate http/2 at the load balancer and convert it to http1.1? Or do you support http/2 all the way to the end service? I would imagine the former would solve these issues.

unscaled 7y ago
If your service can't handle well a bunch of requests coming at once, it doesn't matter if it gets them as individual HTTP/1.1 requests or as multiplexed requests in HTTP/2 coming directly. It only makes a difference if the bottleneck is HTTP/1.1 parsing logic.

stevefan1999 7y ago

First of all, turning on HTTP/2 was not a mistake. Second. turning it on when it is not mature yet is.

floatingatoll 7y ago
The maturity of HTTP/2 is not a causative factor here. They removed a previously-unaware limit on the number of concurrent backend requests, which overflowed their backends. They could have experienced this same outage by simply removing that limit without enabling HTTP/2, and then hitting a peak demand period that was sufficient to cause the outage. Yes, HTTP/2 changes traffic patterns, but the issue could easily have occurred with HTTP/1 as well.

StopHammoTime 7y ago

Great article on approaching massive technical change. Honestly, I think a lot of people general think that most things are just a "switch flip". Even something has implementing SSL on internal apps can cause a big change, let alone the underlying protocol for managing your requests. Thanks for this, because honestly I hadn't thought about the implications myself and it'd be good not to accidentally walk into this problem.

tie_ 7y ago

Surely your application/serving process is able to handle the request burst coming from any single user ? (If not, you have a bigger problem to solver first). If so, I don't quite see why queueing is discussed as an option at all. Queueing means extra latency and worse user experience (not to mention DoS potential). What you should be discussing instead is how to (auto-)scale your app and infrastructure to handle your users' requests.

KaiserPro 7y ago

> decreases latency by multiplexing requests on the same TCP connection On a decent connection, kinda. Anything mobile or worse mobile and moving will suffer terribly.

EugeneOZ 7y ago

Pathetic attempt to get some users from HN.