4 ms·

Author here. > How many requests does your page make on initial load (that can't be handled by a CDN)? If you're making more than six XHRs to your application

by thayne 8y ago

Author here. > How many requests does your page make on initial load (that can't be handled by a CDN)? If you're making more than six XHRs to your application servers concurrently, this sounds like a problem that would have existed anyway had it not been for the browser's (rather arbitrary) connection limit. I don't know the exact number, but definitely higher than six. And I certainly agree that is a failing in our application. The point is that the browserd _did_ arbitrarily limit connections, and our application (unknowingly) depended on that.

bastawhiz 8y ago
It seems odd to pin the blame on HTTP/2, no? Nobody made the intentional decision to lean on the browser's artificial connection limit. It sounds like nobody also made the intentional decision to make lots of XHRs on page load, or configure the load balancer to work the way that it's working, etc. Had you been using HTTP/2 all along and made a change to your load balancer (that resulted in its current behavior), your blog post would likely not have blamed HTTP/2 but rather haproxy/ELB/etc.
- thayne 8y ago
  > Had you been using HTTP/2 all along and made a change to your load balancer (that resulted in its current behavior), your blog post would likely not have blamed HTTP/2 but rather haproxy/ELB/etc. I'm not exactly blaming HTTP/2, just saying the claim that switching to HTTP/2 is easy, safe, and only brings benefits is false.
  - Dylan16807 8y ago
    > I'm not exactly blaming HTTP/2, just saying the claim that switching to HTTP/2 is easy, safe, and only brings benefits is false. Eh. Technically anything could cause problems. I don't think you'll find much in the way of claims that swapping out subsystems could only bring benefits.
    - pwinnski 8y ago
      Google certainly leans pretty hard in that direction.[0] Any and all changes may have unintended side effects, but this particular combination didn't seem obvious to me except in retrospect. [0] https://developers.google.com/web/fundamentals/performance/http2/ https://developers.google.com/web/fundamentals/performance/h...
      - j16sdiz 8y ago
        Just like what they did with AMP. Google had turned evil.
twblalock 8y ago
I've seen the same kind of thing happen when people switched from one load balancer to another: people were unaware they were dependent on a particular kind of queueing or rate limiting to protect their backend services, and they got hit hard when their new load balancer did not protect them in the same way.
- jrochkind1 8y ago
  Concurrency/rate limiting/queuing for HTTP apps, is, I agree, certainly not a trivial thing. You want to be maximizing utilization of your available host resources, while minimizing latency even under unexpected loads (for both median and upper percentiles). Dealing mostly with CPU resource limits, but other issues can be IO contention or contention for limited shared resources like an rdbms, while also not maxing out your RAM. As with anything involving concurrency and hard-to-predict-exactly usage patterns, it can easily get complicated. Who else remembers the [Heroku routing debacle of 2013](https://blog.heroku.com/routing_performance_update https://blog.heroku.com/routing_performance_update)? This stuff ain't easy. Anyone who thinks this would only happen to an unusually "wrong" app, I think, hasn't looked into it seriously. This post was good information, I think it's unfortunate that so much of the discussion seems to be trying to shame the author (making it less likely people in the future will generously share such post-incident information!). It can also be affected a lot by what platform you are on, the language/framework and server architecture(s). They each have different request-handling concurrency patterns, built-in features, and affordances and standard practices. Node is gonna be really different from Rails. I am curious what platforms were involved here.
  - twblalock 8y ago
    I think the reactions are somewhat justified because HTTP/2 was presented as the problem -- or at least it seemed obvious to read the OP in that way. If the OP had been framed as a cautionary tale about how the devs did not realize their traffic patterns were throttling their requests, the reactions probably would have been more positive.
    - 8y ago
Shorn 8y ago
> our application (unknowingly) depended on that Ah, the good old "unrealised infrastructure dependency" - nice to see you my old friend. People that have never been bitten by one of these never built anything worth talking about :) It's worth observing that Gatling (load testing tool) supports HTTP/2. Once you've got the hang of it, you can fairly easily build load profiles to simulate situations like these. Probably wouldn't have helped you prevent the situation - unknown-unknowns being what they are; but you might find it helpful during remediation.

by thayne 8y ago

bastawhiz 8y ago

It seems odd to pin the blame on HTTP/2, no? Nobody made the intentional decision to lean on the browser's artificial connection limit. It sounds like nobody also made the intentional decision to make lots of XHRs on page load, or configure the load balancer to work the way that it's working, etc. Had you been using HTTP/2 all along and made a change to your load balancer (that resulted in its current behavior), your blog post would likely not have blamed HTTP/2 but rather haproxy/ELB/etc.

thayne 8y ago
> Had you been using HTTP/2 all along and made a change to your load balancer (that resulted in its current behavior), your blog post would likely not have blamed HTTP/2 but rather haproxy/ELB/etc. I'm not exactly blaming HTTP/2, just saying the claim that switching to HTTP/2 is easy, safe, and only brings benefits is false.
- Dylan16807 8y ago
  > I'm not exactly blaming HTTP/2, just saying the claim that switching to HTTP/2 is easy, safe, and only brings benefits is false. Eh. Technically anything could cause problems. I don't think you'll find much in the way of claims that swapping out subsystems could only bring benefits.
  - pwinnski 8y ago
    Google certainly leans pretty hard in that direction.[0] Any and all changes may have unintended side effects, but this particular combination didn't seem obvious to me except in retrospect. [0] https://developers.google.com/web/fundamentals/performance/http2/ https://developers.google.com/web/fundamentals/performance/h...
    - j16sdiz 8y ago
      Just like what they did with AMP. Google had turned evil.

twblalock 8y ago

I've seen the same kind of thing happen when people switched from one load balancer to another: people were unaware they were dependent on a particular kind of queueing or rate limiting to protect their backend services, and they got hit hard when their new load balancer did not protect them in the same way.

jrochkind1 8y ago
Concurrency/rate limiting/queuing for HTTP apps, is, I agree, certainly not a trivial thing. You want to be maximizing utilization of your available host resources, while minimizing latency even under unexpected loads (for both median and upper percentiles). Dealing mostly with CPU resource limits, but other issues can be IO contention or contention for limited shared resources like an rdbms, while also not maxing out your RAM. As with anything involving concurrency and hard-to-predict-exactly usage patterns, it can easily get complicated. Who else remembers the [Heroku routing debacle of 2013](https://blog.heroku.com/routing_performance_update https://blog.heroku.com/routing_performance_update)? This stuff ain't easy. Anyone who thinks this would only happen to an unusually "wrong" app, I think, hasn't looked into it seriously. This post was good information, I think it's unfortunate that so much of the discussion seems to be trying to shame the author (making it less likely people in the future will generously share such post-incident information!). It can also be affected a lot by what platform you are on, the language/framework and server architecture(s). They each have different request-handling concurrency patterns, built-in features, and affordances and standard practices. Node is gonna be really different from Rails. I am curious what platforms were involved here.
- twblalock 8y ago
  I think the reactions are somewhat justified because HTTP/2 was presented as the problem -- or at least it seemed obvious to read the OP in that way. If the OP had been framed as a cautionary tale about how the devs did not realize their traffic patterns were throttling their requests, the reactions probably would have been more positive.
  - 8y ago

Shorn 8y ago

> our application (unknowingly) depended on that Ah, the good old "unrealised infrastructure dependency" - nice to see you my old friend. People that have never been bitten by one of these never built anything worth talking about :) It's worth observing that Gatling (load testing tool) supports HTTP/2. Once you've got the hang of it, you can fairly easily build load profiles to simulate situations like these. Probably wouldn't have helped you prevent the situation - unknown-unknowns being what they are; but you might find it helpful during remediation.