4 ms·

Or use scatterplots with points that have opacity < 100%. C'mon there is still a bunch of cases where a scatterplot is much much clearer than that kind of 2d h

by carlob 14y ago

Or use scatterplots with points that have opacity < 100%. C'mon there is still a bunch of cases where a scatterplot is much much clearer than that kind of 2d histogram, for example when there is a non linear relation with very high correlation.

mjw 14y ago
If you have enough point density for this to be a problem there's surely something better you can do though, whether it be binning the data or something a little more sophisticated like plotting a kernel density estimate.
yummyfajitas 14y ago
Here is a hexbin of a highly correlated nonlinear relationship (y=x^2 + gaussian(0,0.1)), together with a scatterplot with opacity: opacity=35%: http://i.imgur.com/VB1E0.png http://i.imgur.com/VB1E0.png opacity=20%, and 2x bigger bins for the hexbin: http://i.imgur.com/yDR3j.png http://i.imgur.com/yDR3j.png Which image do you think better displays the image? Just to make it interesting, I set it up so that the x values are not uniformly distributed. That's very easy to see in the hexbin, but still hard in the scatterplot. The hexbin will always have a higher visual resolution than the scatterplot because the hexbin uses multiple colors to differentiate different densities. The scatterplot uses only one color.
- twstws 14y ago
  The superiority of your hexbin follows from setting the density too high for the scatter plot. With points this dense, opacity of 5 or lower is necessary to see the uneven distribution along the x axis. With appropriate opacity, the two plots are pretty similar visually. What's more, the hexbin by definition has lower resolution, since you lump data into discrete bins. This is an example of bad plotting practice, not a bad plotting method. That said, eyeballing a plot is a weak way to analyze data this dense. That's what statistics are for.
  - yummyfajitas 14y ago
    Ok, redid it with 5% opacity. You are correct, at that level, the density distribution in x is qualitatively visible. http://i.imgur.com/geKbT.png http://i.imgur.com/geKbT.png I disagree that the hexbin has lower resolution. The color dimension allows the human eye to easily differentiate regions having similar densities (e.g., 70 vs 50). The difference between deep red and orange is a lot bigger than the difference between dark blue and slightly less dark blue. The hexbin has lower spatial resolution, it's true, but I'd argue that the spatial resolution you get in a scatterplot is illusory. It doesn't reflect the underlying probability distribution, only the particular sample.
    - twstws 14y ago
      > The difference between deep red and orange is a lot bigger than the difference between dark blue and slightly less dark blue. That may be true, but is it better? Does the scatterplot underemphasize differen densities, or does the density plot overemphasize them? I think the scatterplot is more intuitive. There are 21 equal steps between 0 and 100% black. Two points are twice as dark as one, four points are twice as dark as two. Darker means more, lighter means less. Compare that to shifting from blue to red. Does the shift from orange to red indicate the same density difference as the shift from blue to orange? To decide you need to consult the color scale. The scatterplot is intuitive, and requires no scale. > The hexbin has lower spatial resolution, it's true, but I'd argue that the spatial resolution you get in a scatterplot is illusory. It doesn't reflect the underlying probability distribution, only the particular sample. The spatial resolution of a scatterplot represents empirical reality. Each point corresponds to a single observation, with no probability distribution implied or imposed. The density plot, in contrast, imposes a probability distribution, which may or may not reflect the true distribution of the population. The larger the bins, the more likely the displayed pattern is 'illusory'.
      - yummyfajitas 14y ago
        The spatial resolution of a scatterplot represents empirical reality. Each point corresponds to a single observation, with no probability distribution implied or imposed. tl;dr; I'm a Bayesian, you are a Frequentist (at least with regards to plotting).
- cscheid 14y ago
  Just FYI, the colormap you picked is terrible. There's many experiments to back this claim: http://gvi.seas.harvard.edu/paper/evaluation-artery-visualizations-heart-disease-diagnosis http://gvi.seas.harvard.edu/paper/evaluation-artery-visualiz... http://www.jwave.vt.edu/~rkriz/Projects/create_color_table/color_07.pdf http://www.jwave.vt.edu/~rkriz/Projects/create_color_table/c... Colormap design is arguably as hard as visualization design. My favorite go-to place for them is http://colorbrewer2.org http://colorbrewer2.org, but if you need to know only one thing about them, it's that varying hue continuously does not work nearly as well as you might think it does. In addition, the fundamental reason scatterplots are bad, even with opacity, is essentially that opacity gives rise to an exponential relationship between overplotting and transparency. There exists an alternative solution, which is to use additive blending and an adaptive linear colorscale, from zero to maximum-overdraw. Unfortunately, at present there exists no data visualization toolkits which support this.
  - yummyfajitas 14y ago
    They suggest the rainbow color map is a poor choice because it hides small details. I'm advocating regularization which ensures that you have no small details. But after reading that paper, I do agree that rainbow has some significant issues. One thing that might be worth trying: make a rainbow color map, but map values to colors in such a way that |x-y|=C x cielab_dist(color(x), color(y)).
    - gammarator 14y ago
      I believe the 'Spectral' colormap (with a capital S) in matplotlib does exactly that. Based on the names, it seems [1, 2] (along with all the other capitalized matplotlib colormaps) to be a ColorBrewer [3] colormap, which are all designed with these perceptual considerations in mind [4]. [1] https://github.com/gka/chroma.js/wiki/Predefined-Colors https://github.com/gka/chroma.js/wiki/Predefined-Colors [2] http://matplotlib.sourceforge.net/examples/pylab_examples/show_colormaps.html http://matplotlib.sourceforge.net/examples/pylab_examples/sh... [3] http://colorbrewer2.com/ http://colorbrewer2.com/ [4] http://vis4.net/blog/posts/avoid-equidistant-hsv-colors/ http://vis4.net/blog/posts/avoid-equidistant-hsv-colors/

by carlob 14y ago

mjw 14y ago

If you have enough point density for this to be a problem there's surely something better you can do though, whether it be binning the data or something a little more sophisticated like plotting a kernel density estimate.

yummyfajitas 14y ago

Here is a hexbin of a highly correlated nonlinear relationship (y=x^2 + gaussian(0,0.1)), together with a scatterplot with opacity: opacity=35%: http://i.imgur.com/VB1E0.png http://i.imgur.com/VB1E0.png opacity=20%, and 2x bigger bins for the hexbin: http://i.imgur.com/yDR3j.png http://i.imgur.com/yDR3j.png Which image do you think better displays the image? Just to make it interesting, I set it up so that the x values are not uniformly distributed. That's very easy to see in the hexbin, but still hard in the scatterplot. The hexbin will always have a higher visual resolution than the scatterplot because the hexbin uses multiple colors to differentiate different densities. The scatterplot uses only one color.

twstws 14y ago
The superiority of your hexbin follows from setting the density too high for the scatter plot. With points this dense, opacity of 5 or lower is necessary to see the uneven distribution along the x axis. With appropriate opacity, the two plots are pretty similar visually. What's more, the hexbin by definition has lower resolution, since you lump data into discrete bins. This is an example of bad plotting practice, not a bad plotting method. That said, eyeballing a plot is a weak way to analyze data this dense. That's what statistics are for.
- yummyfajitas 14y ago
  Ok, redid it with 5% opacity. You are correct, at that level, the density distribution in x is qualitatively visible. http://i.imgur.com/geKbT.png http://i.imgur.com/geKbT.png I disagree that the hexbin has lower resolution. The color dimension allows the human eye to easily differentiate regions having similar densities (e.g., 70 vs 50). The difference between deep red and orange is a lot bigger than the difference between dark blue and slightly less dark blue. The hexbin has lower spatial resolution, it's true, but I'd argue that the spatial resolution you get in a scatterplot is illusory. It doesn't reflect the underlying probability distribution, only the particular sample.
  - twstws 14y ago
    > The difference between deep red and orange is a lot bigger than the difference between dark blue and slightly less dark blue. That may be true, but is it better? Does the scatterplot underemphasize differen densities, or does the density plot overemphasize them? I think the scatterplot is more intuitive. There are 21 equal steps between 0 and 100% black. Two points are twice as dark as one, four points are twice as dark as two. Darker means more, lighter means less. Compare that to shifting from blue to red. Does the shift from orange to red indicate the same density difference as the shift from blue to orange? To decide you need to consult the color scale. The scatterplot is intuitive, and requires no scale. > The hexbin has lower spatial resolution, it's true, but I'd argue that the spatial resolution you get in a scatterplot is illusory. It doesn't reflect the underlying probability distribution, only the particular sample. The spatial resolution of a scatterplot represents empirical reality. Each point corresponds to a single observation, with no probability distribution implied or imposed. The density plot, in contrast, imposes a probability distribution, which may or may not reflect the true distribution of the population. The larger the bins, the more likely the displayed pattern is 'illusory'.
    - yummyfajitas 14y ago
      The spatial resolution of a scatterplot represents empirical reality. Each point corresponds to a single observation, with no probability distribution implied or imposed. tl;dr; I'm a Bayesian, you are a Frequentist (at least with regards to plotting).
cscheid 14y ago
Just FYI, the colormap you picked is terrible. There's many experiments to back this claim: http://gvi.seas.harvard.edu/paper/evaluation-artery-visualizations-heart-disease-diagnosis http://gvi.seas.harvard.edu/paper/evaluation-artery-visualiz... http://www.jwave.vt.edu/~rkriz/Projects/create_color_table/color_07.pdf http://www.jwave.vt.edu/~rkriz/Projects/create_color_table/c... Colormap design is arguably as hard as visualization design. My favorite go-to place for them is http://colorbrewer2.org http://colorbrewer2.org, but if you need to know only one thing about them, it's that varying hue continuously does not work nearly as well as you might think it does. In addition, the fundamental reason scatterplots are bad, even with opacity, is essentially that opacity gives rise to an exponential relationship between overplotting and transparency. There exists an alternative solution, which is to use additive blending and an adaptive linear colorscale, from zero to maximum-overdraw. Unfortunately, at present there exists no data visualization toolkits which support this.
- yummyfajitas 14y ago
  They suggest the rainbow color map is a poor choice because it hides small details. I'm advocating regularization which ensures that you have no small details. But after reading that paper, I do agree that rainbow has some significant issues. One thing that might be worth trying: make a rainbow color map, but map values to colors in such a way that |x-y|=C x cielab_dist(color(x), color(y)).
  - gammarator 14y ago
    I believe the 'Spectral' colormap (with a capital S) in matplotlib does exactly that. Based on the names, it seems [1, 2] (along with all the other capitalized matplotlib colormaps) to be a ColorBrewer [3] colormap, which are all designed with these perceptual considerations in mind [4]. [1] https://github.com/gka/chroma.js/wiki/Predefined-Colors https://github.com/gka/chroma.js/wiki/Predefined-Colors [2] http://matplotlib.sourceforge.net/examples/pylab_examples/show_colormaps.html http://matplotlib.sourceforge.net/examples/pylab_examples/sh... [3] http://colorbrewer2.com/ http://colorbrewer2.com/ [4] http://vis4.net/blog/posts/avoid-equidistant-hsv-colors/ http://vis4.net/blog/posts/avoid-equidistant-hsv-colors/