3 ms·
When we first tried to enable HTTP/2 on our load balancers a few years ago, we ended up breaking several applications built on (iirc) gunicorn. We eventually de
by floatingatoll 8y ago
When we first tried to enable HTTP/2 on our load balancers a few years ago, we ended up breaking several applications built on (iirc) gunicorn. We eventually determined the root cause to be:
1) The browser was sending a "streaming data follows" header flag followed by a 0-byte DATA packet in the HTTP/2 stream to work around an ancient SPDY/3 bug.
2) The load balancer was responding to the HTTP/2 "streaming data follows" header packet by activating pipelining to the HTTP/1.1 backend.
3) The backend was terminating the HTTP/1.1 connection from the load balancer with a pipelining-unsupported error.
The browser removed the workaround, the load balancer vendor removed the HTTP/2 frontend's ability to activate HTTP/1.1 pipelining, and after a few months we were able to proceed.
Diagnosing this took weeks of wireshark, source code browsing, and experimental testing. We were lucky that it broke so obviously that the proximity to enabling HTTP/2 was obvious.
- toast0 8y agoIf you can recollect more details, I would love to know what happeend, but I'm not sure about 3) I'm not aware of a pipelining-unsupported error in http (it is a thing in SMTP). It would take a very special HTTP server to look for another request in the socketbuffer after the current one and respond with failure. On the other hand, a quick search found evidence of some very special HTTP servers doing bizarre things with HTTP: https://github.com/elastic/elasticsearch/issues/2665 https://github.com/elastic/elasticsearch/issues/2665
- floatingatoll 8y agoI looked it up and I remembered incorrectly: the bug was due to the load balancer activating chunked transfer encoding to the backend nodes due to receiving the described HTTP2 request. It did not involve pipelining.
- toast0 8y agoThank you, that makes more sense. Chunked transfer encoding is also a hidden danger!