3 ms·

This graph is hilarious: https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9

by epups 4y ago

This graph is hilarious: https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F7809bc5c-5e3b-42fb-b3aa-0afd476cc29d_1628x1212.png https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_... The putative effect is not statistically distinguishable, as observed by the fat overlapping confidence intervals. And the word "old" growing in prominence in movie titles is a very weird, unreliable proxy for the effect at hand. The rest of the article is similar drivel.

Closi 4y ago
This looked odd to me too - and suspiciously like a p-hack. Besides - if the grey shaded area is where we are supposed to believe the 'actual' number should fall, we actually know the 'actual' numbers (the dots) and 60% of the datapoints fall outside of the shaded area, so this seems like total nonsense to me (presumably 95% of the points should sit inside this area?). On top of that - if you added the actual numbers to this graph (which I expect are 1-4 films a year) only a handful of releases can completely skew the data. And then even if you ignore the above, the whole premise titles such as 'old school', 'no country for old men' and 'the 40 year old virgin' are picked because of a trend towards 'old stuff' isn't in any way backed up by either data or common sense when thinking about the movie titles. When you get a graph like this which has such basic failures in stats and common sense, it does make you question the data and rigor behind the rest of the article.
IshKebab 4y ago
Ha yeah you've got to love a graph that looks like random noise until you throw a fancy looking R trend line on it. Most of the data points are outside the confidence interval!
- gwern 4y ago
  Irrelevant. The confidence interval is for the uncertainty of a parameter, the overall mean/trend: if most of your points fell inside it, that would not be good but extremely bad (you have that much imprecision that your estimate of the overall mean could be anywhere inside the range of observations?). Ideally, your CI would be so narrow that ~0 data points fell within it. (An example: you take a population census of adult heights and find the mean height is 5'7 ± 0.001 inches, say. Very few people will fall within that CI. But that's because it's such a good precise CI due to so much data, not because it's a bad CI.) You seem to be thinking of a predictive interval (https://en.wikipedia.org/wiki/Prediction_interval https://en.wikipedia.org/wiki/Prediction_interval), which, however, is not what is being plotted here.
matt123456789 4y ago
Notice also that we are looking at a “minimum 1k” observations, and the y-axis is labeled in increments of one-one thousandth starting at zero, so really it’s “1”, “2”, “3”, etc. The observed fluctuation may be attributable to removing the components of the top-1k which were not movies (reducing the size of the denominator). I don’t know if I would go so far as to call the rest of the article drivel, though. There are some pretty interesting points made elsewhere, especially the average age of actor plot.
jandrese 4y ago
The chart is measuring an effect way down in the noise. 0.3% is the top of the chart!

by epups 4y ago

Closi 4y ago

This looked odd to me too - and suspiciously like a p-hack. Besides - if the grey shaded area is where we are supposed to believe the 'actual' number should fall, we actually know the 'actual' numbers (the dots) and 60% of the datapoints fall outside of the shaded area, so this seems like total nonsense to me (presumably 95% of the points should sit inside this area?). On top of that - if you added the actual numbers to this graph (which I expect are 1-4 films a year) only a handful of releases can completely skew the data. And then even if you ignore the above, the whole premise titles such as 'old school', 'no country for old men' and 'the 40 year old virgin' are picked because of a trend towards 'old stuff' isn't in any way backed up by either data or common sense when thinking about the movie titles. When you get a graph like this which has such basic failures in stats and common sense, it does make you question the data and rigor behind the rest of the article.

IshKebab 4y ago

Ha yeah you've got to love a graph that looks like random noise until you throw a fancy looking R trend line on it. Most of the data points are outside the confidence interval!

gwern 4y ago
Irrelevant. The confidence interval is for the uncertainty of a parameter, the overall mean/trend: if most of your points fell inside it, that would not be good but extremely bad (you have that much imprecision that your estimate of the overall mean could be anywhere inside the range of observations?). Ideally, your CI would be so narrow that ~0 data points fell within it. (An example: you take a population census of adult heights and find the mean height is 5'7 ± 0.001 inches, say. Very few people will fall within that CI. But that's because it's such a good precise CI due to so much data, not because it's a bad CI.) You seem to be thinking of a predictive interval (https://en.wikipedia.org/wiki/Prediction_interval https://en.wikipedia.org/wiki/Prediction_interval), which, however, is not what is being plotted here.

matt123456789 4y ago

Notice also that we are looking at a “minimum 1k” observations, and the y-axis is labeled in increments of one-one thousandth starting at zero, so really it’s “1”, “2”, “3”, etc. The observed fluctuation may be attributable to removing the components of the top-1k which were not movies (reducing the size of the denominator). I don’t know if I would go so far as to call the rest of the article drivel, though. There are some pretty interesting points made elsewhere, especially the average age of actor plot.

jandrese 4y ago

The chart is measuring an effect way down in the noise. 0.3% is the top of the chart!