12 ms·

Unlearning descriptive statistics

343 points by stdbrouw 10y ago

minimaxir 10y ago
It's worth noting that most statistical packages include the ability to generate a five-number summary (https://en.m.wikipedia.org/wiki/Five-number_summary https://en.m.wikipedia.org/wiki/Five-number_summary), which generates many of the described skew-independent descriptive statistics automatically. (R, for example, will generate a five-number summary for each feature of a data frame with a simple summary(df) ) The TL;DR is to just plot everything if possible to visualize skew explicitly (or tricks like the Quartet), and there are a number of tools which can do that without much LOC overhead. (e.g ggplot2)
- jfaucett 10y ago
  you can also use the aptly named fivenum() func in R :)
  - apathy 10y ago
    summary(df) will do that for every column of a dataframe with less code... which is why few people use fivenum() The majority of defaults in R exist because statisticians do these things all the time and don't like typing any more than programmers. Time spent typing could be spent thinking about what assumptions are being violated and whether the results offer any useful insight.
- knowtheory 10y ago
  Yeah, this is a narrative challenge in reporting. Often skewedness of a data set is a(n important) piece of background context, and figuring out how to get that detail into a story without being like AND HERE IS A BIG CHART WHICH YOU WILL NEED HELP INTERPRETING is non-trivial.
- eanzenberg 10y ago
  Pandas has a .describe() function on dataframes too.
jfaucett 10y ago
This is a great read. I've been thinking alot about these basic concepts myself lately, especially the idea of central tendency. Essentially, we can make up any method we want to summarize data and give us a single value that represents the central location of the data i.e. mean vs. least absolute distance vs. distance squared, etc. I haven't thought much about the difference between "typical case" or "expected value" so that's a very useful distinction to be made, especially when deciding which method you want to pick.
- FabHK 10y ago
  One good way to think about it is to ask what norm ("distance") between all the datapoints and your "central tendency" statistic you want to minimize, as the article alluded to. For L2 (squared distance), you get the mean: For fixed x_i, sum of i=1..N of (x_i - M)^2 is smallest for M = mean. For L1 (absolute distance), you get the median. For L0 ("identical or not"), you get the mode. You can come up with other and alternative concepts, but it's kind of neat that the 3 most common descriptions of central tendency pop out of this one unified approach.
clircle 10y ago
For anyone who is interesting in correlation measures, have a look at distance correlation: https://en.wikipedia.org/wiki/Distance_correlation https://en.wikipedia.org/wiki/Distance_correlation http://projecteuclid.org/euclid.aoas/1267453933 http://projecteuclid.org/euclid.aoas/1267453933 It's implemented in the R package "energy", and provides a more comprehensive measure of variable dependence than Spearman's rank correlation. Distance correlation is able to detect any kind of dependence, not just monotonic relationships.
pizza 10y ago
Surely there has to be an analysis somewhere that someone has done which links Kolmogorov complexity and one-statistic-describes-all approaches of interpreting numbers?
- stdbrouw 10y ago
  Something like what you describe is more or less how statisticians do in fact reason about statistics: https://en.wikipedia.org/wiki/Sufficient_statistic#Minimal_sufficiency https://en.wikipedia.org/wiki/Sufficient_statistic#Minimal_s..., though only for i.i.d. data from a parametric distribution.
andy_wrote 10y ago
For readers who are OK with some math, I recommend John Myles White's eye-opening post about means, medians, and modes: http://www.johnmyleswhite.com/notebook/2013/03/22/modes-medians-and-means-an-unifying-perspective/ http://www.johnmyleswhite.com/notebook/2013/03/22/modes-medi... He describes these summary descriptive stats in terms of what penalty function they minimize: mean minimizes L2, median minimizes L1, mode minimizes L0. A single-number statistic is _going_ to leave things out, so if you must boil things down into one number, or even a few numbers, you're going to lose something that you had in the raw data. This is why I find claims along the lines of "statistics don't tell the whole story" a little bemusing - of course they don't, the very definition of a statistic is a summarization of data that is easier to work with. The question is what data is kept or lost, or more generally what importance we place on different aspects of the raw data such that it's reflected in our descriptive statistics. The lessons for non-technical people who want to communicate with descriptive statistics are to recognize that summarization is inherent in the nature of any descriptive statistic, that they are thereby opinionated in some way in terms of what they've preserved and what they've left out, and to recognize whether those opinions are appropriate for your purpose.
- johnmyleswhite 10y ago
  Glad you enjoyed that post so much. It really is a shame that we do such a bad job of teaching students about the inherent subjectivity of descriptive statistics and let students leave their courses with dangerous ideas about the existence of a Holy Grail statistic that will solve all of their problems.
  - nkkar 10y ago
    Your followup post (http://www.johnmyleswhite.com/notebook/2013/03/22/using-norms-to-understand-linear-regression/ http://www.johnmyleswhite.com/notebook/2013/03/22/using-norm...) is excellent. Thank you!
    - johnmyleswhite 10y ago
      Thanks! I really should have finished and written the post about the SVD as well. One of these days...
shmageggy 10y ago
> But why do you want a number at all? Because lots of data analysis questions hinge upon the association between two data sets, and it's nice (crucial) to be able to quantify this value. Especially because > While statisticians are generally quite good at estimating a correlation from a picture and vice versa, most people are not. The author says > Still not happy and absolutely want a number? You would do well to shun correlations even so. OK, so what else do you suggest?
- stdbrouw 10y ago
  > OK, so what else do you suggest? Slopes, a.k.a. the parameters in a regression analysis, ideally as a confidence interval or prediction interval to account for uncertainty. Interpreting and communicating regression analyses is a pretty big topic on its own so I chose to only hint at it, though I understand that might not be very satisfying for some readers.
- apathy 10y ago
  1) it is pretty amazing that normit transformations (map the quantiles of a non-normal distribution onto a Gaussian and use that) don't seem to be on this guy's radar. We use distributions with linearly additive and affine invariant properties (normal plus normal is normal, bernoulli plus bernoulli is bitwise bernoulli) because we find linear algebra very useful. Nonparametric tests and procedures erode your power; normit transformations usually increase it. I did part of my dissertation on this; it's partly to do with asymptotics, but also partly due to the robustness of Gaussian error assumptions thanks to the CLT. I realized recently that a lot of the trouble people have with training neural networks stems from their lack of training in basic model evaluation. If you stack a bunch of shitty penalized regressions (which is what linear/logistic + relU hinge loss represents) you now have one gigantic shitty regression which is harder to debug. If your early steps are thrown out of whack by outliers, your later steps will be too. Dropout is an attempt to remedy this, but you tend to lose power when you shrink your dataset or model, so (per usual) there really is no such thing as a free lunch. But most of the tradeoffs make more sense when you are able to evaluate each layer as a predictor/filter. Scaling this up to deep models is hard, therefore debugging deep models is hard. Not exactly a big leap. There is a reason people say "an expert is a master of the fundamentals". Building a castle on a swamp gives poor results. If you can't design an experiment to test your model and its assumptions, your model will suck. This is not rocket surgery. A GPU allows you to make more mistakes, faster, if that's what you want. If you have the fundamentals nailed down, and enough data to avoid overfitting, then nonlinear approaches can be incredibly powerful. Most of the time a simple logistic regression will offer 80-90% of the power of a DNN, a kernel regression will offer 80-90% of the power of a CNN, and an HMM or Kalman filter will offer 80-90% of the power of an RNN. It's when you need that 10-20% "extra" to compete, and have the data to do it, that deeper or trickier architectures help. If you can transform a bunch of correlated data so that it is 1) decorrelated and 2) close enough to Gaussian for government work, you suddenly get a tremendous amount of power from linear algebra and differential geometry "for free". This is one reason why Bayesian and graphical hierarchical mixed models work well -- you borrow information when you don't have enough to feed the model, and if you have some domain expertise, this allows you to keep the model from making stupid or impossible predictions. Anyways. I have had fun lately playing with various deep, recurrent, and adversarial architectures. I don't mean to imply they aren't tremendously powerful in the right hands. But so is a Hole Hawg. Don't use a Hole Hawg when a paper punch is all you really need. 2) What (good) statisticians excel at is catching faulty assumptions. (I'll leave it to the reader to decide whether this data-scientist-for-hire has done a good job of that in his piece) So we plot our data, marginally or via projections, all the damned time. If you don't, sooner or later it will bite you in the ass, and then you too can join the ranks of the always-plotting. However, choosing which margins or conditional distributions to plot in a high-dimensional or sparse dataset is important to avoid wasting a lot of time. So whether via transformation or penalization (e.g. graphical lasso) or both, we usually try to prune things down and home on on "the good stuff". Prioritizing what to do first is most easily done if you have a number and can rank the putative significance by that number. Use Spearman, use energy statistics (distance correlation), use marginal score tests -- IDGAF, just use these as guidelines and plot the damned data. Corollary: if someone shows you fancy plots and never simple ones containing clouds of individual data points, they're probably full of shit. Boxplots should be beeswarms, loess plots should have scatterplots (smoothed or otherwise) behind them. And for god's sake plot your residuals, either implicitly or explicitly. 3) see above. The author is good at fussing, and brings up some classical points. But they're not really his points. Median and MAD are more robust to outliers than mean and standard deviation, but that makes them less sensitive, too. Check your assumptions, plot everything, use the numbers as advisory quantities rather than final results.
OliverJones 10y ago
This is terrific. In the field of web ops, I've had great success tracking the 95th percentile of request time rather than the mean, median, mode, or any other descriptive metric. The systems I worked with were, like many systems, ordinarily very efficient. That meant the mean and median metrics washed away the occasional troublesome request and hid it from the metrics dashboard. But knowing the 95th percentile was out of bounds allowed my team to investigate and try to remediate the trouble spots. This was especially useful in troubleshooting fax delivery (yeah, I know, I know: it was healthcare stuff; fax is considered secure and email is not) through unbelievably flakey private branch exchanges (big city hospitals)
- keithpeter 10y ago
  Sounds fine to me: fits in with 'management by exception' - look for the extreme cases. Standard in the industrial quality control world (2*sd above or below).
- telotortium 10y ago
  Gil Tene has an entire talk about this point called "How NOT to measure latency" (https://www.youtube.com/watch?v=lJ8ydIuPFeU https://www.youtube.com/watch?v=lJ8ydIuPFeU). In many circumstances, a web service (either server or client) has large numbers of dependencies, so a user only has to hit 95th-percentile latency for one dependency to have their overall latency significantly hurt. In other words, users may encounter the "rare" bad case on a majority of requests. Thus, even higher percentiles should be tightly controlled in order to truly keep a web service's overall latency under control.
somestag 10y ago
I think the message of the article is great: move beyond the "standard" descriptions and pay more attention to what you're trying to show and who your audience is. That said, it's a slight pet peeve of mine when people recommend the median over the mean to describe center. The median, on its own, does not describe what is "typical" any more than the mean does; it just has a small advantage in that it will always map a real observation, so for discrete data you don't end up with things like "1.9 legs." (That said, a mean of 1.9 legs actually seems much more informative to me than a median of 2 legs, so even in that case I prefer the mean.) It's easy to envision many situations where the median's representation is wildly inaccurate, just as you can imagine ways in which the mean can be misleading. The median is also insensitive to skew, which is often cited as a good thing, but really that's something you should determine on a case-by-case basis. In many (most?) applications, there's no tangible benefit in having a measure of center that ignores skew. The median's insensitivity also creates strange situations where subgroups in the population end up unrepresented (e.g. if the poorest 20% become even poorer because of changes to the tax code, the median doesn't budge). In general, the median conveys remarkably little information. It's great at showing the center, but it gives zero indication of anything else. The mean certainly has its issues, but it's far from inferior. Use the median when the situation calls for it, certainly, but realize that it's just as limited as any other measure, if not more so.
- closed 10y ago
  Great points. Discrete data with few categories are a good case where the median might not cut it. As a contrived example.. data: 0 0 0 1 1 mean: 2 / 5 median: 0
  - maxerickson 10y ago
    With a small number of categories a full summary of the data easily fits into text. It might still be desirable to report a summary statistic, but omitting the totals for 2 or 3 categories and reporting a summary statistic is pretty sloppy.
    - closed 10y ago
TorKlingberg 10y ago
I think "skew" is on of those things that are taught not because it is useful, but because it is easy to teach, easy to memorize and easy to test for.
heinrichhartman 10y ago
I wrote an article for the ACM Queue (also published in CACM) about this very issue called "Statistics for Engineers" http://queue.acm.org/detail.cfm?id=2903468 http://queue.acm.org/detail.cfm?id=2903468
- bigger_cheese 10y ago
  I took a "Statistics for Engineers" course while at university. It was mostly focused around ANOVA and hypothesis testing. By the end of the course I was sick of hearing the phrase "Null hypothesis". I wish the course had introduced some more practical applications instead it was very high level and theoretical. Working as an engineer now (Materials Engineering) here is a short list from the top of my head of what I feel are useful statistics applications that were not covered in uni. - Alternative probability distributions (Rosin-Ramler) - Harmonic and Geometric means - Multivariate regression - Statistical Sampling techniques (how to obtain a representative sample and avoid sampling bias) - Statistical process control - Time Series analysis
  - heinrichhartman 10y ago
    Thanks for your list. In the IT Operations / Monitoring domain time series models play a large role. At the same time the level of sophistication is rather low. You definetly have to start with the basics when teaching such a course.
FabHK 10y ago
To be fair the article should not be called "Unlearning descriptive statistics", but "Learning just a bit more (and often better) descriptive statistics", but granted, it's not that snappy. Certainly better than "I took a statistics course, and you won't believe what happened next!!!1!!"
- sevensor 10y ago
  Agreed, but I've got to say from experience that quantile statistics like the author proposes are often way more informative. The only downside is that they can be considerably more expensive to compute for bigger data sets, whereas mean and standard deviation are cheap and can be updated without taking another pass over the data.
pyromine 10y ago
I don't feel particularly convinced by this article, I feel like it misses the bullet by going beyond a reasonable standard. The article is phrased in such a manner that it acts as if typical descriptive statistics are not useful, but the more important point is more reasonable and understand when to use what statistics. In fact, many of his examples of what to use are just other descriptive statistics, median is no less a descriptive statistic than the mean is. I think the problem is not so much that descriptive statistics are bad, so much as they are not particularly useful when they lack context. Unfortunately, statistics is a much more complex field than your average product designer doing an A/B test thinks. The majority of statistics and models the average person utilizes has assumptions and qualifications that they don't fully understand, and frankly that's not a slight to them but rather just the additional depth of the field. Just as an example, ask the average person to derive the degrees of freedom for their t-test without just utilizing the formula they were taught. As in most fields this is an issue of nuance, not that beginner techniques are bad, but beginner techniques alone are not nuanced enough to capture useful or rigorous insights.
- ykler 10y ago
  "In fact, many of his examples of what to use are just other descriptive statistics, median is no less a descriptive statistic than the mean is." I think the author would fully agree with this. He is saying we should use more easily understandable descriptive statistics. "I think the problem is not so much that descriptive statistics are bad, so much as they are not particularly useful when they lack context. Unfortunately, statistics is a much more complex field than your average product designer doing an A/B test thinks." Whatever you say publicly is likely to be stripped of its context and to be received by an audience that does not understand the complexities of your field. The answer to this is to be change what you say, not to change the world, which verges on impossible.
eanzenberg 10y ago
This is actually a great post. I think it's a wonderful idea to move away from worshipping summary statistics (ave, median) and start studying the underlying distributions.
yomritoyj 10y ago
I think that the article is unfair to statisticians in saying that they ignore descriptive statistics. The problem is that one cannot choose systematically between different descriptive statistics without clearly specifying our objectives and what we believe about the data-generating process. Otherwise on what grounds can you say that the median is 'better' than the mean or the other way round? But once you specify these things you are doing statistical estimation, which is a major branch of mathematical statistics.
TheGorramBatman 10y ago
Who the hell would consider Tukey a fringe statistician?
- stdbrouw 10y ago
  Lauded, sure, but just about all of his books are out of print and his work is rarely a part of the curriculum.
wodenokoto 10y ago
Great article overall, but I wish it went a little deeper in explaining how we should interpret the statistics it recommend. It basically says "don't use the normal numbers, use these instead, because they are closer to what you would expect them to mean" However I don't find all of these statistics super obvious.
em500 10y ago
To answer the author's postscript: "why did nobody tell me this?" Because you didn't pay attention at school? Means, medians, modes and percentiles were standard fare in my high school, and again in freshman university courses. And I didn't go to particularly expensive schools or elite universities. Then again, I also taught intro stat courses to BA students for a few years, and it was easy to tell that the vast majority of them cared very little, since understanding their uses wasn't exam material.
- rankam 10y ago
  I think the author is trying to point out that mean, median, and mode are things that people think they understand because they know how to calculate them - any one of them is trivial for the majority of the high school educated population (14-18 years old). However, just because you can calculate something doesn't mean you know how to properly apply it. When you taught your intro stats class, did you try to give the students guidance (and convey understanding) as to when and why they should apply one of the statistics? If so, good on you, but from my, obviously anecdotal experience, that has not been the case.
deleted 10y ago
[deleted]
jordigh 10y ago
I would like to advertise again my favourite measurement of skewness, the medcouple: https://en.wikipedia.org/wiki/Medcouple https://en.wikipedia.org/wiki/Medcouple It's robust (25% breakdown point) and is kind of the "optimal" measurement of skewness, as it's computed as the median of all possible interquartile range-like statistics. I hope someone implements the faster algorithm for Python's statsmodels.
wnoise 10y ago
Variance and standard deviation gets a lot of hate here, that I don't think it deserves. It really does have nice properties, generally nicer than the MAD. Squaring feels to me much less of a hack than the absolute value. The problem that most people don't quite understand it is certainly true, but that's as much a problem with the people as the method.
krick 10y ago
Can somebody recommend a good book (or any other resource) on statistics? I mean kind of stuff this post talks about: descriptives, tests, all the basic stuff. Despite being somewhat familiar with that, I sometimes feel I really lack the solid understanding of the subject and am longing for something explanatory, with real life examples and exercises.
- stdbrouw 10y ago
  Think Stats (http://greenteapress.com/wp/think-stats-2e/ http://greenteapress.com/wp/think-stats-2e/) and the follow-up Think Bayes are awesome if you're a coder. https://bblais.github.io/statistical-inference-for-everyone-sie.html https://bblais.github.io/statistical-inference-for-everyone-... is a complete rethink of how basic statistics should be taught and is excellent as well.
  - krick 10y ago
    Thanks, looks awesome.
OscarCunningham 10y ago
What's the "ordinal center"?
- stdbrouw 10y ago
  If you've got a bunch of ordered categories, like "bad", "okay" and "good", the category in the middle. (So if humans can have 0, 1 or 2 legs, then 1 would be at the center.)
kriro 10y ago
On a somewhat related note, I think it's interesting that there is data on how people perceive information but that isn't really taught in descriptive statistics (at least it wasn't at my university). I feel like data representation and charts/graphs should be more of a focus if the title of the class is descriptive statistics. I hadn't even heard about "How to Lie with Statistics" or "The Visual Display of Quantitative Information" before randomly finding them mentioned somewhere. I simply learned about some standard graphs without much explanation on why you use them or how they impact the understanding of the audience.

343 points by stdbrouw 10y ago

It's worth noting that most statistical packages include the ability to generate a five-number summary (https://en.m.wikipedia.org/wiki/Five-number_summary https://en.m.wikipedia.org/wiki/Five-number_summary), which generates many of the described skew-independent descriptive statistics automatically. (R, for example, will generate a five-number summary for each feature of a data frame with a simple summary(df) ) The TL;DR is to just plot everything if possible to visualize skew explicitly (or tricks like the Quartet), and there are a number of tools which can do that without much LOC overhead. (e.g ggplot2)

jfaucett 10y ago
you can also use the aptly named fivenum() func in R :)
- apathy 10y ago
  summary(df) will do that for every column of a dataframe with less code... which is why few people use fivenum() The majority of defaults in R exist because statisticians do these things all the time and don't like typing any more than programmers. Time spent typing could be spent thinking about what assumptions are being violated and whether the results offer any useful insight.
knowtheory 10y ago
Yeah, this is a narrative challenge in reporting. Often skewedness of a data set is a(n important) piece of background context, and figuring out how to get that detail into a story without being like AND HERE IS A BIG CHART WHICH YOU WILL NEED HELP INTERPRETING is non-trivial.
eanzenberg 10y ago
Pandas has a .describe() function on dataframes too.

jfaucett 10y ago

This is a great read. I've been thinking alot about these basic concepts myself lately, especially the idea of central tendency. Essentially, we can make up any method we want to summarize data and give us a single value that represents the central location of the data i.e. mean vs. least absolute distance vs. distance squared, etc. I haven't thought much about the difference between "typical case" or "expected value" so that's a very useful distinction to be made, especially when deciding which method you want to pick.

FabHK 10y ago
One good way to think about it is to ask what norm ("distance") between all the datapoints and your "central tendency" statistic you want to minimize, as the article alluded to. For L2 (squared distance), you get the mean: For fixed x_i, sum of i=1..N of (x_i - M)^2 is smallest for M = mean. For L1 (absolute distance), you get the median. For L0 ("identical or not"), you get the mode. You can come up with other and alternative concepts, but it's kind of neat that the 3 most common descriptions of central tendency pop out of this one unified approach.

clircle 10y ago

For anyone who is interesting in correlation measures, have a look at distance correlation: https://en.wikipedia.org/wiki/Distance_correlation https://en.wikipedia.org/wiki/Distance_correlation http://projecteuclid.org/euclid.aoas/1267453933 http://projecteuclid.org/euclid.aoas/1267453933 It's implemented in the R package "energy", and provides a more comprehensive measure of variable dependence than Spearman's rank correlation. Distance correlation is able to detect any kind of dependence, not just monotonic relationships.

pizza 10y ago

Surely there has to be an analysis somewhere that someone has done which links Kolmogorov complexity and one-statistic-describes-all approaches of interpreting numbers?

stdbrouw 10y ago
Something like what you describe is more or less how statisticians do in fact reason about statistics: https://en.wikipedia.org/wiki/Sufficient_statistic#Minimal_sufficiency https://en.wikipedia.org/wiki/Sufficient_statistic#Minimal_s..., though only for i.i.d. data from a parametric distribution.

andy_wrote 10y ago

For readers who are OK with some math, I recommend John Myles White's eye-opening post about means, medians, and modes: http://www.johnmyleswhite.com/notebook/2013/03/22/modes-medians-and-means-an-unifying-perspective/ http://www.johnmyleswhite.com/notebook/2013/03/22/modes-medi... He describes these summary descriptive stats in terms of what penalty function they minimize: mean minimizes L2, median minimizes L1, mode minimizes L0. A single-number statistic is _going_ to leave things out, so if you must boil things down into one number, or even a few numbers, you're going to lose something that you had in the raw data. This is why I find claims along the lines of "statistics don't tell the whole story" a little bemusing - of course they don't, the very definition of a statistic is a summarization of data that is easier to work with. The question is what data is kept or lost, or more generally what importance we place on different aspects of the raw data such that it's reflected in our descriptive statistics. The lessons for non-technical people who want to communicate with descriptive statistics are to recognize that summarization is inherent in the nature of any descriptive statistic, that they are thereby opinionated in some way in terms of what they've preserved and what they've left out, and to recognize whether those opinions are appropriate for your purpose.

johnmyleswhite 10y ago
Glad you enjoyed that post so much. It really is a shame that we do such a bad job of teaching students about the inherent subjectivity of descriptive statistics and let students leave their courses with dangerous ideas about the existence of a Holy Grail statistic that will solve all of their problems.
- nkkar 10y ago
  Your followup post (http://www.johnmyleswhite.com/notebook/2013/03/22/using-norms-to-understand-linear-regression/ http://www.johnmyleswhite.com/notebook/2013/03/22/using-norm...) is excellent. Thank you!
  - johnmyleswhite 10y ago
    Thanks! I really should have finished and written the post about the SVD as well. One of these days...

shmageggy 10y ago

> But why do you want a number at all? Because lots of data analysis questions hinge upon the association between two data sets, and it's nice (crucial) to be able to quantify this value. Especially because > While statisticians are generally quite good at estimating a correlation from a picture and vice versa, most people are not. The author says > Still not happy and absolutely want a number? You would do well to shun correlations even so. OK, so what else do you suggest?

stdbrouw 10y ago
> OK, so what else do you suggest? Slopes, a.k.a. the parameters in a regression analysis, ideally as a confidence interval or prediction interval to account for uncertainty. Interpreting and communicating regression analyses is a pretty big topic on its own so I chose to only hint at it, though I understand that might not be very satisfying for some readers.
apathy 10y ago
1) it is pretty amazing that normit transformations (map the quantiles of a non-normal distribution onto a Gaussian and use that) don't seem to be on this guy's radar. We use distributions with linearly additive and affine invariant properties (normal plus normal is normal, bernoulli plus bernoulli is bitwise bernoulli) because we find linear algebra very useful. Nonparametric tests and procedures erode your power; normit transformations usually increase it. I did part of my dissertation on this; it's partly to do with asymptotics, but also partly due to the robustness of Gaussian error assumptions thanks to the CLT. I realized recently that a lot of the trouble people have with training neural networks stems from their lack of training in basic model evaluation. If you stack a bunch of shitty penalized regressions (which is what linear/logistic + relU hinge loss represents) you now have one gigantic shitty regression which is harder to debug. If your early steps are thrown out of whack by outliers, your later steps will be too. Dropout is an attempt to remedy this, but you tend to lose power when you shrink your dataset or model, so (per usual) there really is no such thing as a free lunch. But most of the tradeoffs make more sense when you are able to evaluate each layer as a predictor/filter. Scaling this up to deep models is hard, therefore debugging deep models is hard. Not exactly a big leap. There is a reason people say "an expert is a master of the fundamentals". Building a castle on a swamp gives poor results. If you can't design an experiment to test your model and its assumptions, your model will suck. This is not rocket surgery. A GPU allows you to make more mistakes, faster, if that's what you want. If you have the fundamentals nailed down, and enough data to avoid overfitting, then nonlinear approaches can be incredibly powerful. Most of the time a simple logistic regression will offer 80-90% of the power of a DNN, a kernel regression will offer 80-90% of the power of a CNN, and an HMM or Kalman filter will offer 80-90% of the power of an RNN. It's when you need that 10-20% "extra" to compete, and have the data to do it, that deeper or trickier architectures help. If you can transform a bunch of correlated data so that it is 1) decorrelated and 2) close enough to Gaussian for government work, you suddenly get a tremendous amount of power from linear algebra and differential geometry "for free". This is one reason why Bayesian and graphical hierarchical mixed models work well -- you borrow information when you don't have enough to feed the model, and if you have some domain expertise, this allows you to keep the model from making stupid or impossible predictions. Anyways. I have had fun lately playing with various deep, recurrent, and adversarial architectures. I don't mean to imply they aren't tremendously powerful in the right hands. But so is a Hole Hawg. Don't use a Hole Hawg when a paper punch is all you really need. 2) What (good) statisticians excel at is catching faulty assumptions. (I'll leave it to the reader to decide whether this data-scientist-for-hire has done a good job of that in his piece) So we plot our data, marginally or via projections, all the damned time. If you don't, sooner or later it will bite you in the ass, and then you too can join the ranks of the always-plotting. However, choosing which margins or conditional distributions to plot in a high-dimensional or sparse dataset is important to avoid wasting a lot of time. So whether via transformation or penalization (e.g. graphical lasso) or both, we usually try to prune things down and home on on "the good stuff". Prioritizing what to do first is most easily done if you have a number and can rank the putative significance by that number. Use Spearman, use energy statistics (distance correlation), use marginal score tests -- IDGAF, just use these as guidelines and plot the damned data. Corollary: if someone shows you fancy plots and never simple ones containing clouds of individual data points, they're probably full of shit. Boxplots should be beeswarms, loess plots should have scatterplots (smoothed or otherwise) behind them. And for god's sake plot your residuals, either implicitly or explicitly. 3) see above. The author is good at fussing, and brings up some classical points. But they're not really his points. Median and MAD are more robust to outliers than mean and standard deviation, but that makes them less sensitive, too. Check your assumptions, plot everything, use the numbers as advisory quantities rather than final results.

OliverJones 10y ago

This is terrific. In the field of web ops, I've had great success tracking the 95th percentile of request time rather than the mean, median, mode, or any other descriptive metric. The systems I worked with were, like many systems, ordinarily very efficient. That meant the mean and median metrics washed away the occasional troublesome request and hid it from the metrics dashboard. But knowing the 95th percentile was out of bounds allowed my team to investigate and try to remediate the trouble spots. This was especially useful in troubleshooting fax delivery (yeah, I know, I know: it was healthcare stuff; fax is considered secure and email is not) through unbelievably flakey private branch exchanges (big city hospitals)

keithpeter 10y ago
Sounds fine to me: fits in with 'management by exception' - look for the extreme cases. Standard in the industrial quality control world (2*sd above or below).
telotortium 10y ago
Gil Tene has an entire talk about this point called "How NOT to measure latency" (https://www.youtube.com/watch?v=lJ8ydIuPFeU https://www.youtube.com/watch?v=lJ8ydIuPFeU). In many circumstances, a web service (either server or client) has large numbers of dependencies, so a user only has to hit 95th-percentile latency for one dependency to have their overall latency significantly hurt. In other words, users may encounter the "rare" bad case on a majority of requests. Thus, even higher percentiles should be tightly controlled in order to truly keep a web service's overall latency under control.

somestag 10y ago

I think the message of the article is great: move beyond the "standard" descriptions and pay more attention to what you're trying to show and who your audience is. That said, it's a slight pet peeve of mine when people recommend the median over the mean to describe center. The median, on its own, does not describe what is "typical" any more than the mean does; it just has a small advantage in that it will always map a real observation, so for discrete data you don't end up with things like "1.9 legs." (That said, a mean of 1.9 legs actually seems much more informative to me than a median of 2 legs, so even in that case I prefer the mean.) It's easy to envision many situations where the median's representation is wildly inaccurate, just as you can imagine ways in which the mean can be misleading. The median is also insensitive to skew, which is often cited as a good thing, but really that's something you should determine on a case-by-case basis. In many (most?) applications, there's no tangible benefit in having a measure of center that ignores skew. The median's insensitivity also creates strange situations where subgroups in the population end up unrepresented (e.g. if the poorest 20% become even poorer because of changes to the tax code, the median doesn't budge). In general, the median conveys remarkably little information. It's great at showing the center, but it gives zero indication of anything else. The mean certainly has its issues, but it's far from inferior. Use the median when the situation calls for it, certainly, but realize that it's just as limited as any other measure, if not more so.

closed 10y ago
Great points. Discrete data with few categories are a good case where the median might not cut it. As a contrived example.. data: 0 0 0 1 1 mean: 2 / 5 median: 0
- maxerickson 10y ago
  With a small number of categories a full summary of the data easily fits into text. It might still be desirable to report a summary statistic, but omitting the totals for 2 or 3 categories and reporting a summary statistic is pretty sloppy.
  - closed 10y ago

TorKlingberg 10y ago

I think "skew" is on of those things that are taught not because it is useful, but because it is easy to teach, easy to memorize and easy to test for.

heinrichhartman 10y ago

I wrote an article for the ACM Queue (also published in CACM) about this very issue called "Statistics for Engineers" http://queue.acm.org/detail.cfm?id=2903468 http://queue.acm.org/detail.cfm?id=2903468

bigger_cheese 10y ago
I took a "Statistics for Engineers" course while at university. It was mostly focused around ANOVA and hypothesis testing. By the end of the course I was sick of hearing the phrase "Null hypothesis". I wish the course had introduced some more practical applications instead it was very high level and theoretical. Working as an engineer now (Materials Engineering) here is a short list from the top of my head of what I feel are useful statistics applications that were not covered in uni. - Alternative probability distributions (Rosin-Ramler) - Harmonic and Geometric means - Multivariate regression - Statistical Sampling techniques (how to obtain a representative sample and avoid sampling bias) - Statistical process control - Time Series analysis
- heinrichhartman 10y ago
  Thanks for your list. In the IT Operations / Monitoring domain time series models play a large role. At the same time the level of sophistication is rather low. You definetly have to start with the basics when teaching such a course.

FabHK 10y ago

To be fair the article should not be called "Unlearning descriptive statistics", but "Learning just a bit more (and often better) descriptive statistics", but granted, it's not that snappy. Certainly better than "I took a statistics course, and you won't believe what happened next!!!1!!"

sevensor 10y ago
Agreed, but I've got to say from experience that quantile statistics like the author proposes are often way more informative. The only downside is that they can be considerably more expensive to compute for bigger data sets, whereas mean and standard deviation are cheap and can be updated without taking another pass over the data.

pyromine 10y ago

I don't feel particularly convinced by this article, I feel like it misses the bullet by going beyond a reasonable standard. The article is phrased in such a manner that it acts as if typical descriptive statistics are not useful, but the more important point is more reasonable and understand when to use what statistics. In fact, many of his examples of what to use are just other descriptive statistics, median is no less a descriptive statistic than the mean is. I think the problem is not so much that descriptive statistics are bad, so much as they are not particularly useful when they lack context. Unfortunately, statistics is a much more complex field than your average product designer doing an A/B test thinks. The majority of statistics and models the average person utilizes has assumptions and qualifications that they don't fully understand, and frankly that's not a slight to them but rather just the additional depth of the field. Just as an example, ask the average person to derive the degrees of freedom for their t-test without just utilizing the formula they were taught. As in most fields this is an issue of nuance, not that beginner techniques are bad, but beginner techniques alone are not nuanced enough to capture useful or rigorous insights.

ykler 10y ago
"In fact, many of his examples of what to use are just other descriptive statistics, median is no less a descriptive statistic than the mean is." I think the author would fully agree with this. He is saying we should use more easily understandable descriptive statistics. "I think the problem is not so much that descriptive statistics are bad, so much as they are not particularly useful when they lack context. Unfortunately, statistics is a much more complex field than your average product designer doing an A/B test thinks." Whatever you say publicly is likely to be stripped of its context and to be received by an audience that does not understand the complexities of your field. The answer to this is to be change what you say, not to change the world, which verges on impossible.

eanzenberg 10y ago

This is actually a great post. I think it's a wonderful idea to move away from worshipping summary statistics (ave, median) and start studying the underlying distributions.

yomritoyj 10y ago

I think that the article is unfair to statisticians in saying that they ignore descriptive statistics. The problem is that one cannot choose systematically between different descriptive statistics without clearly specifying our objectives and what we believe about the data-generating process. Otherwise on what grounds can you say that the median is 'better' than the mean or the other way round? But once you specify these things you are doing statistical estimation, which is a major branch of mathematical statistics.

TheGorramBatman 10y ago

Who the hell would consider Tukey a fringe statistician?

stdbrouw 10y ago
Lauded, sure, but just about all of his books are out of print and his work is rarely a part of the curriculum.

wodenokoto 10y ago

Great article overall, but I wish it went a little deeper in explaining how we should interpret the statistics it recommend. It basically says "don't use the normal numbers, use these instead, because they are closer to what you would expect them to mean" However I don't find all of these statistics super obvious.

em500 10y ago

To answer the author's postscript: "why did nobody tell me this?" Because you didn't pay attention at school? Means, medians, modes and percentiles were standard fare in my high school, and again in freshman university courses. And I didn't go to particularly expensive schools or elite universities. Then again, I also taught intro stat courses to BA students for a few years, and it was easy to tell that the vast majority of them cared very little, since understanding their uses wasn't exam material.

rankam 10y ago
I think the author is trying to point out that mean, median, and mode are things that people think they understand because they know how to calculate them - any one of them is trivial for the majority of the high school educated population (14-18 years old). However, just because you can calculate something doesn't mean you know how to properly apply it. When you taught your intro stats class, did you try to give the students guidance (and convey understanding) as to when and why they should apply one of the statistics? If so, good on you, but from my, obviously anecdotal experience, that has not been the case.

deleted 10y ago

[deleted]

jordigh 10y ago

I would like to advertise again my favourite measurement of skewness, the medcouple: https://en.wikipedia.org/wiki/Medcouple https://en.wikipedia.org/wiki/Medcouple It's robust (25% breakdown point) and is kind of the "optimal" measurement of skewness, as it's computed as the median of all possible interquartile range-like statistics. I hope someone implements the faster algorithm for Python's statsmodels.

wnoise 10y ago

Variance and standard deviation gets a lot of hate here, that I don't think it deserves. It really does have nice properties, generally nicer than the MAD. Squaring feels to me much less of a hack than the absolute value. The problem that most people don't quite understand it is certainly true, but that's as much a problem with the people as the method.

krick 10y ago

Can somebody recommend a good book (or any other resource) on statistics? I mean kind of stuff this post talks about: descriptives, tests, all the basic stuff. Despite being somewhat familiar with that, I sometimes feel I really lack the solid understanding of the subject and am longing for something explanatory, with real life examples and exercises.

stdbrouw 10y ago
Think Stats (http://greenteapress.com/wp/think-stats-2e/ http://greenteapress.com/wp/think-stats-2e/) and the follow-up Think Bayes are awesome if you're a coder. https://bblais.github.io/statistical-inference-for-everyone-sie.html https://bblais.github.io/statistical-inference-for-everyone-... is a complete rethink of how basic statistics should be taught and is excellent as well.
- krick 10y ago
  Thanks, looks awesome.

OscarCunningham 10y ago

What's the "ordinal center"?

stdbrouw 10y ago
If you've got a bunch of ordered categories, like "bad", "okay" and "good", the category in the middle. (So if humans can have 0, 1 or 2 legs, then 1 would be at the center.)

kriro 10y ago

On a somewhat related note, I think it's interesting that there is data on how people perceive information but that isn't really taught in descriptive statistics (at least it wasn't at my university). I feel like data representation and charts/graphs should be more of a focus if the title of the class is descriptive statistics. I hadn't even heard about "How to Lie with Statistics" or "The Visual Display of Quantitative Information" before randomly finding them mentioned somewhere. I simply learned about some standard graphs without much explanation on why you use them or how they impact the understanding of the audience.