7 ms·

Author here. I am seeing a lot of comments about how the graphs are not anchored at 0. The intent with the graphs was not to "lie" or "mislead" but to fit the

Author here. I am seeing a lot of comments about how the graphs are not anchored at 0. The intent with the graphs was not to "lie" or "mislead" but to fit the data in a way that was mostly readable side by side. The goal was to show the high level change, in a glanceable way, not to get in to individual millisecond comparisons. However, in the future I would pick a different visualization I think :) The benchmarking has also come under fire. My goal was to just to put the same site/assets on three different continents and retrieve them a bunch of times. No more, no less. I think the results are still interesting, personally. Clean room benchmarks are cool, but so are real world tests, imo. Finally, there was no agenda with this post to push HTTP/3 over HTTP/2. I was actually skeptical that HTTP/3 made any kind of difference based on my experience with 1.1 to 2. I expected to write a post about "HTTP/3 is not any better than HTTP/2" and was frankly surprised that it was so much faster in my tests.

aquadrop 5y ago
Thanks for the article. But the goal of the graphs should be to show level of change and let graph speak for itself, if it's high or low. If they were anchored at 0 it would actually allow to see visual difference and for me personally it would be "way that was mostly readable side by side".
kzrdude 5y ago
The missing part of the axis, down to 0, is just 1/9th of the current length, so I think it's absolutely the wrong trade-off to cut the y-axis.
posix_me_less 5y ago
Could it be simply because UDP packets may be treated with higher priority by all the middleman machines? UDP is used by IP phones, video conferences, etc.
- digikata 5y ago
  Because of the use of UDP, I wonder the variation could end up being wider. There are some large outliers on the chart (though that could be an implementation maturity issue). Also I wonder if the routing environment will continue to be the same in terms of prioritization if http/3 becomes more common. There might be some motivation to slow down http/3 to prioritize traditional udp real-time uses.
brnt 5y ago
> I am seeing a lot of comments about how the graphs are not anchored at 0. Personal preference: for large offsets that makes sense. For small ones (~10% of max here) it seems unnecessary, or, to a suspicious mind, meant to hide something ;)
- ReactiveJelly 5y ago
  Some of the India offsets are huge. And the numbers are too small to read, and reading numbers on a graph is mixing System 1 and System 2 thinking, anyway. I agree that the graphs would be better and still impressive even anchored to 0
- eric_trackjs 5y ago
  Yep, I didn't even think about the suspicious angle when I did it. Mostly I was fiddling with how to draw box plots in D3 and that's what came out. Next time I will ensure a 0 axis!
maxmcd 5y ago
I wouldn't interpret those comments as accusation. It's in all our best interests to critique possibly misleading graphs, even when done so unintentionally.
vlmutolo 5y ago
> However, in the future I would pick a different visualization I think I think the box plots were a good choice here. I quickly understood what I was looking at, which is a high compliment for any visualization. When it's done right it seems easy and obvious. But the y-axis really needs to start at 0. It's the only way the reader will perceive the correct relative difference between the various measurements. As an extreme example, if I have measurements [A: 100, B: 101, C: 105], and then scale the axes to "fit around" the data (maybe from 100 to 106 on thy y axis), it will seem like C is 5x larger than B. In reality, it's only 1.05x larger. Leave the whitespace at the bottom of the graph if the relative size of the measurements matters (it usually does).
- eric_trackjs 5y ago
  Agreed. Next time I'll make the text and other things a little larger too (the real graphs are actually quite large, I had to shrink them to fit the article formatting.) I'd already spent so much time on the article I didn't want to go back and redo the graphs (I didn't really think too many people would read it - it was a big surprise to see it on HN)
- remus 5y ago
  I think these choices are more context specific than is often appreciated. For example > if I have measurements [A: 100, B: 101, C: 105], and then scale the axes to "fit around" the data (maybe from 100 to 106 on thy y axis), it will seem like C is 5x larger than B. In reality, it's only 1.05x larger. If you were interested in the absolute difference between the values then starting your axis at 0 is going to make it hard to read.
  - amenod 5y ago
    It is however very rare that absolute differences matter; and even when they do, the scale should (often) be fixed. For example the temperatures: [A: 27.0, B: 29.0, C: 28.0] versus: [A: 27.0, B: 27.2, C: 26.9] If scale is fit to the min and max values, the charts will look the same. Still, as a rule of thumb, when Y axis doesn't start at 0, the chart is probably misleading. It is very rare that the absolute size of the measured quantity doesn't matter.
morrbo 5y ago
I found the article very useful regardless so thank you
rhplus 5y ago
Adding an 'axis break' is a great way to focus in on the range of interest while also highlighting the fact that it's not zero-based.
remram 5y ago
You might also want to actually put HTTP 1/2/3 side-by-side in each graph, and separate graphs by use case. Rather than the current visualization, putting use cases side by side, and HTTP 1/2/3 in different graphs. edit: like this: https://imgur.com/a/7Gvq59j https://imgur.com/a/7Gvq59j
marcos100 5y ago
The charts are ok for the purpose of visualizing the performance of the protocols. At least for me, they are side-by-side with the same min and max values and is easy to compare. The purpose is clear and starting from zero adds nothing of value. People who think the bottom line is zero don't know how to read a chart. What maybe is missing is a table with the statistics to compare numbers with maybe a latency number.
Railsify 5y ago
But they all start at the same anchor value, I was immediately able to interpret them and did not feel misled.

aquadrop 5y ago

Thanks for the article. But the goal of the graphs should be to show level of change and let graph speak for itself, if it's high or low. If they were anchored at 0 it would actually allow to see visual difference and for me personally it would be "way that was mostly readable side by side".

kzrdude 5y ago

The missing part of the axis, down to 0, is just 1/9th of the current length, so I think it's absolutely the wrong trade-off to cut the y-axis.

posix_me_less 5y ago

Could it be simply because UDP packets may be treated with higher priority by all the middleman machines? UDP is used by IP phones, video conferences, etc.

digikata 5y ago
Because of the use of UDP, I wonder the variation could end up being wider. There are some large outliers on the chart (though that could be an implementation maturity issue). Also I wonder if the routing environment will continue to be the same in terms of prioritization if http/3 becomes more common. There might be some motivation to slow down http/3 to prioritize traditional udp real-time uses.

brnt 5y ago

> I am seeing a lot of comments about how the graphs are not anchored at 0. Personal preference: for large offsets that makes sense. For small ones (~10% of max here) it seems unnecessary, or, to a suspicious mind, meant to hide something ;)

ReactiveJelly 5y ago
Some of the India offsets are huge. And the numbers are too small to read, and reading numbers on a graph is mixing System 1 and System 2 thinking, anyway. I agree that the graphs would be better and still impressive even anchored to 0
eric_trackjs 5y ago
Yep, I didn't even think about the suspicious angle when I did it. Mostly I was fiddling with how to draw box plots in D3 and that's what came out. Next time I will ensure a 0 axis!

maxmcd 5y ago

I wouldn't interpret those comments as accusation. It's in all our best interests to critique possibly misleading graphs, even when done so unintentionally.

vlmutolo 5y ago

> However, in the future I would pick a different visualization I think I think the box plots were a good choice here. I quickly understood what I was looking at, which is a high compliment for any visualization. When it's done right it seems easy and obvious. But the y-axis really needs to start at 0. It's the only way the reader will perceive the correct relative difference between the various measurements. As an extreme example, if I have measurements [A: 100, B: 101, C: 105], and then scale the axes to "fit around" the data (maybe from 100 to 106 on thy y axis), it will seem like C is 5x larger than B. In reality, it's only 1.05x larger. Leave the whitespace at the bottom of the graph if the relative size of the measurements matters (it usually does).

eric_trackjs 5y ago
Agreed. Next time I'll make the text and other things a little larger too (the real graphs are actually quite large, I had to shrink them to fit the article formatting.) I'd already spent so much time on the article I didn't want to go back and redo the graphs (I didn't really think too many people would read it - it was a big surprise to see it on HN)
remus 5y ago
I think these choices are more context specific than is often appreciated. For example > if I have measurements [A: 100, B: 101, C: 105], and then scale the axes to "fit around" the data (maybe from 100 to 106 on thy y axis), it will seem like C is 5x larger than B. In reality, it's only 1.05x larger. If you were interested in the absolute difference between the values then starting your axis at 0 is going to make it hard to read.
- amenod 5y ago
  It is however very rare that absolute differences matter; and even when they do, the scale should (often) be fixed. For example the temperatures: [A: 27.0, B: 29.0, C: 28.0] versus: [A: 27.0, B: 27.2, C: 26.9] If scale is fit to the min and max values, the charts will look the same. Still, as a rule of thumb, when Y axis doesn't start at 0, the chart is probably misleading. It is very rare that the absolute size of the measured quantity doesn't matter.

morrbo 5y ago

I found the article very useful regardless so thank you

rhplus 5y ago

Adding an 'axis break' is a great way to focus in on the range of interest while also highlighting the fact that it's not zero-based.

remram 5y ago

You might also want to actually put HTTP 1/2/3 side-by-side in each graph, and separate graphs by use case. Rather than the current visualization, putting use cases side by side, and HTTP 1/2/3 in different graphs. edit: like this: https://imgur.com/a/7Gvq59j https://imgur.com/a/7Gvq59j

marcos100 5y ago

The charts are ok for the purpose of visualizing the performance of the protocols. At least for me, they are side-by-side with the same min and max values and is easy to compare. The purpose is clear and starting from zero adds nothing of value. People who think the bottom line is zero don't know how to read a chart. What maybe is missing is a table with the statistics to compare numbers with maybe a latency number.

Railsify 5y ago

But they all start at the same anchor value, I was immediately able to interpret them and did not feel misled.