5 ms·

Truncating Bar Graphs Persistently Misleads Viewers

112 points by sebg 6y ago

blakesterz 6y ago
You can read a preprint of this article here: https://psyarxiv.com/7aq4h/ https://psyarxiv.com/7aq4h/
- throwawayfb69 6y ago
  I'm not super convinced by this study. The examples show Fox News-levels of graph manipulation, i.e. maximum manipulation. This is clearly done in an attempt to communicate a particular message. If you read the questions as "What message was the author trying to communicate?", which is reasonable, then the study could show that you were willing to repeat back the author's intended message. In this sense, you weren't fooled by the graph - you understood exactly the message it was intended to convey. A truncated graph is, in my opinion, completely appropriate for data where the relative differences or trends in data points is more relevant to the message than the absolute values - a temperature chart, say (where the absolute values are pretty much meaningless). If you took data of a clearly relative type rather than a clearly absolute type, would you be able to say that viewers were misled in the exact opposite direction? That would be interesting.
  - gus_massa 6y ago
    A cropped line graph may be fine. In particular, in a temperature graph, where should we put the horizontal axis, at 0°C, 0°F or 0K? The problem are truncated bar graph. Using a truncated bar graph is never a good idea.
  - manicdee 6y ago
    Regardless of the type of graph, it's important to understand whether a difference of 5 units between one measure and the next is meaningful. For example if you are measuring the length of optical fibre and the difference between the lengths of fibres is 5cm over 100km, is that important in the context? You'll already lose several metres of length in various loops and coils that are important for reliable function of that 100km length of fibre. On the other hand when cutting frames for doors, the difference of a couple of millimetres over the two metre length of the side of the frame can make the difference between passing or failing the building envelope pressurisation test. On the gripping hand when I'm shooting an arrow 50m to a target, what's of interest to me is not the distance from me to the arrow but the tiny variances in distance from the arrow to the bullseye. So measuring the vector from me to the arrow is pointless and the centimetre or two difference in length of those vectors is meaningless.
boringg 6y ago
It's similar to the lack of context. For example budget costs without context so people freak out on total dollar spent instead of how it fits in the context of historical spend. I would argue that time series plots should be given for most large number items so people have contextual history instead of going straight to outrage. Not that there is an incentive for most media organizations to present information in a reasonable way, would lower traffic and engagement.
- kevincox 6y ago
  This is also similar to providing massive numbers without any context. For example $x produces 2.3M tonnes more $waste than $y every year!!! Of course $y produces 8.2T tonnes a year so it isn't /that much/ of a delta. Also vaguely related to my pet peeve of using large denominators to make numbers large and confusing. This factory produces 63M items a year! How about 2 per second, I can visualize 2.
  - leetcrew 6y ago
    > Also vaguely related to my pet peeve of using large denominators to make numbers large and confusing. This factory produces 63M items a year! How about 2 per second, I can visualize 2. smaller denominators can also be misleading though. maybe this is just me, but I usually read the denominator as implying a time period over which the figure is stable. for example, before the pandemic, I drove about 40 miles/day. I really did drive 20 miles each way to work pretty much every day. another way of saying this is that I drove 1.6 miles/hour every work day. true on average but kinda misleading when I spent 23 hours out of the day not driving at all. it might not matter much to your example, but the factory probably didn't produce two items every second. there were probably shift changes, or maybe it stopped production for a few hours every night. depending on market conditions, it might have produced more on some days than others, and so forth.
    - kevincox 6y ago
      That is true. There are cases where the timeframe is important, but often it isn't. I definitely should have put "on average" in there. You can also be more clear and say "Produces 5 per second when running at full capacity, averaging 2 per second throughout the year" if that is appropriate for the info you are sharing.
__s 6y ago
The number of pixels representing quantities in a graph should be proportional to each other as what they represent. This also addresses perspective bias in slanted 3d pi charts Curious what biases one might also find based on color intensity of bars & varying widths Not sure how to balance out this idea with log scales, which can be useful, but require a bit more care in visually digesting
smartmic 6y ago
This is neither new nor surprising. I recommend the little book "How to lie with statistics" by Darrell Huff, first published 1954. Chapter 5, called "The Gee-Whiz Graph" dives into the matter of truncated bar graphs (among others).
- hanche 6y ago
  Absolutely! I am happy to note that the paper does in fact reference How to lie with statistics in the introduction: > While discussions of misleading graphs are not new (e.g., Huff, 1954), empirical research on their assumed consequences is scattered across fields. So on the surface, at least, this looks like just another paper belabouring the obvious.
OliverJones 6y ago
Sure enough. Edward Tufte calls this stuff "chartjunk." His books are really good sources for learning to do this data-presentation well.
7402 6y ago
This could be considered a subset of the more general problem of trying to figure out when two points on a graph are meaningfully different, as opposed to just visually different. In my early physics training the importance of putting error bars on data points was stressed, allowing for a visual comparison of the difference between two points in standard deviations.
snshn 6y ago
We have to use the technology: animate bars on the X/Y scale on the graph, speed of the animation should be tied to the current max value represented on the graph. This way, humans will naturally recognize the value the current bars represent.
pessimizer 6y ago
As an effective technique, it should be in everyone's toolbox.
btilly 6y ago
Wonderful. We now have research to prove what has been well-known and obvious for many decades. After all this exact form of misleading graph was one of the topics in the famous book How to Lie With Statistics written in 1954. And I don't think that the observation was new then.
birdyrooster 6y ago
Funny, I just added y-min: 0 to a lot of my grafana graphs today for this exact reason. It's really good to give yourself sufficient context when creating dashboards or you miss the bigger, more important picture.

112 points by sebg 6y ago

blakesterz 6y ago

You can read a preprint of this article here: https://psyarxiv.com/7aq4h/ https://psyarxiv.com/7aq4h/

throwawayfb69 6y ago
I'm not super convinced by this study. The examples show Fox News-levels of graph manipulation, i.e. maximum manipulation. This is clearly done in an attempt to communicate a particular message. If you read the questions as "What message was the author trying to communicate?", which is reasonable, then the study could show that you were willing to repeat back the author's intended message. In this sense, you weren't fooled by the graph - you understood exactly the message it was intended to convey. A truncated graph is, in my opinion, completely appropriate for data where the relative differences or trends in data points is more relevant to the message than the absolute values - a temperature chart, say (where the absolute values are pretty much meaningless). If you took data of a clearly relative type rather than a clearly absolute type, would you be able to say that viewers were misled in the exact opposite direction? That would be interesting.
- gus_massa 6y ago
  A cropped line graph may be fine. In particular, in a temperature graph, where should we put the horizontal axis, at 0°C, 0°F or 0K? The problem are truncated bar graph. Using a truncated bar graph is never a good idea.
- manicdee 6y ago
  Regardless of the type of graph, it's important to understand whether a difference of 5 units between one measure and the next is meaningful. For example if you are measuring the length of optical fibre and the difference between the lengths of fibres is 5cm over 100km, is that important in the context? You'll already lose several metres of length in various loops and coils that are important for reliable function of that 100km length of fibre. On the other hand when cutting frames for doors, the difference of a couple of millimetres over the two metre length of the side of the frame can make the difference between passing or failing the building envelope pressurisation test. On the gripping hand when I'm shooting an arrow 50m to a target, what's of interest to me is not the distance from me to the arrow but the tiny variances in distance from the arrow to the bullseye. So measuring the vector from me to the arrow is pointless and the centimetre or two difference in length of those vectors is meaningless.

boringg 6y ago

It's similar to the lack of context. For example budget costs without context so people freak out on total dollar spent instead of how it fits in the context of historical spend. I would argue that time series plots should be given for most large number items so people have contextual history instead of going straight to outrage. Not that there is an incentive for most media organizations to present information in a reasonable way, would lower traffic and engagement.

kevincox 6y ago
This is also similar to providing massive numbers without any context. For example $x produces 2.3M tonnes more $waste than $y every year!!! Of course $y produces 8.2T tonnes a year so it isn't /that much/ of a delta. Also vaguely related to my pet peeve of using large denominators to make numbers large and confusing. This factory produces 63M items a year! How about 2 per second, I can visualize 2.
- leetcrew 6y ago
  > Also vaguely related to my pet peeve of using large denominators to make numbers large and confusing. This factory produces 63M items a year! How about 2 per second, I can visualize 2. smaller denominators can also be misleading though. maybe this is just me, but I usually read the denominator as implying a time period over which the figure is stable. for example, before the pandemic, I drove about 40 miles/day. I really did drive 20 miles each way to work pretty much every day. another way of saying this is that I drove 1.6 miles/hour every work day. true on average but kinda misleading when I spent 23 hours out of the day not driving at all. it might not matter much to your example, but the factory probably didn't produce two items every second. there were probably shift changes, or maybe it stopped production for a few hours every night. depending on market conditions, it might have produced more on some days than others, and so forth.
  - kevincox 6y ago
    That is true. There are cases where the timeframe is important, but often it isn't. I definitely should have put "on average" in there. You can also be more clear and say "Produces 5 per second when running at full capacity, averaging 2 per second throughout the year" if that is appropriate for the info you are sharing.

__s 6y ago

The number of pixels representing quantities in a graph should be proportional to each other as what they represent. This also addresses perspective bias in slanted 3d pi charts Curious what biases one might also find based on color intensity of bars & varying widths Not sure how to balance out this idea with log scales, which can be useful, but require a bit more care in visually digesting

smartmic 6y ago

This is neither new nor surprising. I recommend the little book "How to lie with statistics" by Darrell Huff, first published 1954. Chapter 5, called "The Gee-Whiz Graph" dives into the matter of truncated bar graphs (among others).

hanche 6y ago
Absolutely! I am happy to note that the paper does in fact reference How to lie with statistics in the introduction: > While discussions of misleading graphs are not new (e.g., Huff, 1954), empirical research on their assumed consequences is scattered across fields. So on the surface, at least, this looks like just another paper belabouring the obvious.

OliverJones 6y ago

Sure enough. Edward Tufte calls this stuff "chartjunk." His books are really good sources for learning to do this data-presentation well.

7402 6y ago

This could be considered a subset of the more general problem of trying to figure out when two points on a graph are meaningfully different, as opposed to just visually different. In my early physics training the importance of putting error bars on data points was stressed, allowing for a visual comparison of the difference between two points in standard deviations.

snshn 6y ago

We have to use the technology: animate bars on the X/Y scale on the graph, speed of the animation should be tied to the current max value represented on the graph. This way, humans will naturally recognize the value the current bars represent.

pessimizer 6y ago

As an effective technique, it should be in everyone's toolbox.

btilly 6y ago

Wonderful. We now have research to prove what has been well-known and obvious for many decades. After all this exact form of misleading graph was one of the topics in the famous book How to Lie With Statistics written in 1954. And I don't think that the observation was new then.

birdyrooster 6y ago

Funny, I just added y-min: 0 to a lot of my grafana graphs today for this exact reason. It's really good to give yourself sufficient context when creating dashboards or you miss the bigger, more important picture.