3 ms·
I call BS on the benchmarks AND the theoretical analysis. Every time I read those HTTP/X benchmarks, people don't mention TCP's congestion control and seem to j
by littlecranky67 7y ago
I call BS on the benchmarks AND the theoretical analysis. Every time I read those HTTP/X benchmarks, people don't mention TCP's congestion control and seem to just ignore it in their analysis. Well, you can't. At least if you call "realistic conditions". Congestion control will introduce additional round trips, depending on your link parameters (bandwidth, latency, OS configuration like initcwnd etc.) and limit your bandwidth on the transport layer. And based on your link parameter, 6 parallel TCP connections might achieve a higher bandwidth on cold start because the window scaling in tcp slow start is superior than in a single tcp connection used by HTTP/2.
Additionally, the most common error people do while benchmarking (and i assume the author did too) is to ignore the congestion control's caching of the cwnd in the tcp stack of your OS. That is, once the cwnd is raised from usually 10 mss (~14,6kb for most OSes and setups), your OS will cache the value and re-use the larger cwnd as initcwnd when you open a new tcp socket to the same host. So if you do 100 benchmark runs, you will have one "real" run, and 99 will re-use the cwnd cache and produce non-realistic results. Given the author didn't mention tcp congestion control at all and didn't mention any tweaks to the tcp metrics cache (you can disable or reset it through changing /proc/sys/net/ipv4/tcp_no_metrics_save on linux) I assume all the measured numbers are BS.
- jpambrun 7y agoI have use http2 and http1 with domain shardind extensively and the advantage of http2 multiplexing cannot be denied. TCP connection overhead is negligible (other than the tsl negotiation maybe) and is not even really a factor.
- littlecranky67 7y agoTCP congestion window scaling is not negligible, especially not in high-latency environments. The initcwnd is usually 14,6kb(10x MSS of 1460 byte in common DSL setups), meaning a server can only send those 14,6 kb until the connection is stalled and an ACK is received (=1x roundtrip). Google realized this and tried to get a kernel patch into Linux where the browser (or any userspace program) can manipulate the initcwnd. The patch got rejected, and thats basically why they came up with QUIC: To be able to build on UDP and implement congestion control and other stuff TCP usually does (ordering, checksumming etc) in userspace.
- treve 7y agoHi littlecranky67, Would you be interested in helping me do another run of this test, but with more realistic settings? I care about the results and getting this right but you clearly have a better understanding of how to set this up
- littlecranky67 7y agoSorry to disappoint you, but my own opinion is: it can't really be done. There is dozens of variables in such a benchmark (latency, bandwidth, tcp settings on each node such as tcp fast open, selective ack present etc., http & tcp connection caches etc.) and unclear optimization targets. It's not just a matter of sending data to the client and measuring when it arrived, there is a ton of other stuff to consider regarding how the HTML and its linked resources (CSS, Scripts etc.) are structured. There are even random effects at play here (what if you have a packet loss in important packets that carry ACKs etc.). Some 6 years ago I did this for a study thesis (ungraded undergraduate essay) at university (back then HTML/2 was still SPDY) and it was an eye-opener for me - that by carefully picking your parameters you can basically conclude and simulate whatever you want the outcome to be. If you are interested, the study thesis is here: https://www.dropbox.com/s/6vu91yvfzxbzdue/studythesis.pdf?dl=0 https://www.dropbox.com/s/6vu91yvfzxbzdue/studythesis.pdf?dl... The theoretical analysis is still valid today, but the resulting benchmarks should be taken with a grain of salt as they are very test-setup and implementation specific.