4 ms·

The quality and depth of answers here is pretty inconsistent. But this in particular is a pet peeve of mine: > Plot a histogram out of the sampled data. If you

The quality and depth of answers here is pretty inconsistent. But this in particular is a pet peeve of mine: > Plot a histogram out of the sampled data. If you can fit the bell-shaped "normal" curve to the histogram, then the hypothesis that the underlying random variable follows the normal distribution can not be rejected. This is commonly taught in undergrad stats, but you shouldn't do this. I'm of the opinion that normality testing in general is usually a red herring, but this is specifically not a productive way of doing it. Use the other methods. A visual test that relies on how much the histogram approximates a bell curve is very prone to error, because a sample from a variety of other distributions can look visually normal even though it isn't. More broadly speaking, the reason I don't like this is because it's an example of the kind of formulaic, cargo-culted recipes that are often used in statistics without critical thinking. You should strive to obtain a deep understanding of your data and its distribution, and you should be deeply skeptical if the sample you happen to have looks normal. Nature abhors normality, and the central limit theorem can only promise a tendency towards normality as n approaches infinity. It says nothing about what size sample you'll practically need for your specific data to be able to treat it as normal.

tgv 6y ago
There's a test for normality. Well, I knew one, Kolmogorov, but Wikipedia already lists 8. What data scientist doesn't know of such a test? And how would you fit the data? There's not one, unique way to fit. And as you say, a small deviation can mean a lot: if you use L2 distance, the errors at the outer regions of a normal distribution are probably dwarfed by any deviation that occurs more towards the center.
- srean 6y ago
  There are myriads of ways for two things to be unequal. In fact its fair to state that unless you are taking two sets of measurements of some fundamental physical constant or measuring artificially generated data, they are not going to be from the same exact distribution -- you cannot bathe in the same river twice is very true. The way to address the problem you speak of is to think hard about the kind deviations from equality that would be most damaging to the application (were it to slip in undetected). Once you know what's the most damaging deviation, you can then select a test that is very powerful in detecting that specific type of deviation. For example, you mention Kolmogorov(Smirnov) test. Those are daft blind at the tails of the distribution. So in case you need to catch deviations at the tail, then you can safely skip KS test, they are no good. On the other hand KS tests are very powerful around the median, if your application requires high resolution there, KS is the test you want.
  - tgv 6y ago
    I would have blindly gone for KS...
jpeloquin 6y ago
> You should strive to obtain a deep understanding of your data and its distribution, and you should be deeply skeptical if the sample you happen to have looks normal. Although normality testing is useless in many situations, the parent comment somewhat overstates the degree of caution required. In many contexts the exact distribution doesn't matter; sort-of-normal is good enough. For example, the t-test is used ubiquitously. It assumes normality, so we would expect possible non-normality to be a major problem, right? Not so. The t-test is extremely robust to departures from normality given equal sample sizes [1,2,3]. Or you can use a so-called non-parametric test. Rather than investing great effort in specifying exactly what distribution you're dealing with, it's more productive to simply use a test that is robust against your unknowns and move on to pursuing your actual objectives. It's true that if you are interested in predicting events in the tails of the distribution, you really do need to study the distribution in detail. Predicting rare events is very difficult. But if you're just interested in differences between group means, don't overthink it. [1] http://www.jerrydallal.com/LHSP/student3.htm http://www.jerrydallal.com/LHSP/student3.htm [2] Posten, H.O., Yeh, H.C., and Owen, D.B. (1977). Robustness of the two-sample t-test under violations of the homogeneity of variance assumption. Communications in Statistics - Theory and Methods 11, 109–126. [3] Posten, H.O. (1992). Robustness of the two-sample t-test under violations of the homogeneity of variance assumption, part ii. Communications in Statistics - Theory and Methods 21, 2169–2184.
- srean 6y ago
  > The t-test is extremely robust to departures from normality given equal sample sizes That's over selling it. Its very sensitive to fat tails and skew. That's the reason robust testing and estimation is a thing. Wilcox's research would be a good near contemporary place to start [0][1]. [0] Wilcox, Robustness of Standard Tests https://onlinelibrary.wiley.com/doi/abs/10.1002/9781118445112.stat06350 https://onlinelibrary.wiley.com/doi/abs/10.1002/978111844511... (paywalled) Abstract: Conventional hypothesis‐testing methods such as student's t, the ANOVA F test, and methods based on the ordinary least squares regression estimator, are not robust to violations of assumptions. In fact, there are general conditions under which these methods can provide poor control over the probability of a type I error and inaccurate confidence intervals, no matter how large the sample sizes might be. Relatively poor power is yet another concern. [1] Fundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy https://books.google.co.in/books?id=uUNGzhdxk0kC https://books.google.co.in/books?id=uUNGzhdxk0kC Conventional statistical methods have a very serious flaw. They routinely miss differences among groups or associations among variables that are detected by more modern techniques - even under very small departures from normality. Hundreds of journal articles have described the reasons standard techniques can be unsatisfactory, but simple, intuitive explanations are generally unavailable. Improved methods have been derived, but they are far from obvious or intuitive based on the training most researchers receive. Situations arise where even highly nonsignificant results become significant when analyzed with more modern methods. Without assuming any prior training in statistics, Part I of this book describes basic statistical principles from a point of view that makes their shortcomings intuitive and easy to understand.
stagehn 6y ago
The visual test is the best. By far the biggest problem among data scientists is being too theoretical and relying too much on statistical tests and metrics instead of common sense and visualizations. I've worked with many quantitative researchers in the finance industry and those that are successful are always the least theoretical, do not rely on tests, and everything is visualized heavily and conclusions inferred from that. Gut feeling, intuition and visuals are the best protection against over-fitting, p-hacking and self-deception.

tgv 6y ago

There's a test for normality. Well, I knew one, Kolmogorov, but Wikipedia already lists 8. What data scientist doesn't know of such a test? And how would you fit the data? There's not one, unique way to fit. And as you say, a small deviation can mean a lot: if you use L2 distance, the errors at the outer regions of a normal distribution are probably dwarfed by any deviation that occurs more towards the center.

srean 6y ago
There are myriads of ways for two things to be unequal. In fact its fair to state that unless you are taking two sets of measurements of some fundamental physical constant or measuring artificially generated data, they are not going to be from the same exact distribution -- you cannot bathe in the same river twice is very true. The way to address the problem you speak of is to think hard about the kind deviations from equality that would be most damaging to the application (were it to slip in undetected). Once you know what's the most damaging deviation, you can then select a test that is very powerful in detecting that specific type of deviation. For example, you mention Kolmogorov(Smirnov) test. Those are daft blind at the tails of the distribution. So in case you need to catch deviations at the tail, then you can safely skip KS test, they are no good. On the other hand KS tests are very powerful around the median, if your application requires high resolution there, KS is the test you want.
- tgv 6y ago
  I would have blindly gone for KS...

jpeloquin 6y ago

> You should strive to obtain a deep understanding of your data and its distribution, and you should be deeply skeptical if the sample you happen to have looks normal. Although normality testing is useless in many situations, the parent comment somewhat overstates the degree of caution required. In many contexts the exact distribution doesn't matter; sort-of-normal is good enough. For example, the t-test is used ubiquitously. It assumes normality, so we would expect possible non-normality to be a major problem, right? Not so. The t-test is extremely robust to departures from normality given equal sample sizes [1,2,3]. Or you can use a so-called non-parametric test. Rather than investing great effort in specifying exactly what distribution you're dealing with, it's more productive to simply use a test that is robust against your unknowns and move on to pursuing your actual objectives. It's true that if you are interested in predicting events in the tails of the distribution, you really do need to study the distribution in detail. Predicting rare events is very difficult. But if you're just interested in differences between group means, don't overthink it. [1] http://www.jerrydallal.com/LHSP/student3.htm http://www.jerrydallal.com/LHSP/student3.htm [2] Posten, H.O., Yeh, H.C., and Owen, D.B. (1977). Robustness of the two-sample t-test under violations of the homogeneity of variance assumption. Communications in Statistics - Theory and Methods 11, 109–126. [3] Posten, H.O. (1992). Robustness of the two-sample t-test under violations of the homogeneity of variance assumption, part ii. Communications in Statistics - Theory and Methods 21, 2169–2184.

srean 6y ago
> The t-test is extremely robust to departures from normality given equal sample sizes That's over selling it. Its very sensitive to fat tails and skew. That's the reason robust testing and estimation is a thing. Wilcox's research would be a good near contemporary place to start [0][1]. [0] Wilcox, Robustness of Standard Tests https://onlinelibrary.wiley.com/doi/abs/10.1002/9781118445112.stat06350 https://onlinelibrary.wiley.com/doi/abs/10.1002/978111844511... (paywalled) Abstract: Conventional hypothesis‐testing methods such as student's t, the ANOVA F test, and methods based on the ordinary least squares regression estimator, are not robust to violations of assumptions. In fact, there are general conditions under which these methods can provide poor control over the probability of a type I error and inaccurate confidence intervals, no matter how large the sample sizes might be. Relatively poor power is yet another concern. [1] Fundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy https://books.google.co.in/books?id=uUNGzhdxk0kC https://books.google.co.in/books?id=uUNGzhdxk0kC Conventional statistical methods have a very serious flaw. They routinely miss differences among groups or associations among variables that are detected by more modern techniques - even under very small departures from normality. Hundreds of journal articles have described the reasons standard techniques can be unsatisfactory, but simple, intuitive explanations are generally unavailable. Improved methods have been derived, but they are far from obvious or intuitive based on the training most researchers receive. Situations arise where even highly nonsignificant results become significant when analyzed with more modern methods. Without assuming any prior training in statistics, Part I of this book describes basic statistical principles from a point of view that makes their shortcomings intuitive and easy to understand.

stagehn 6y ago

The visual test is the best. By far the biggest problem among data scientists is being too theoretical and relying too much on statistical tests and metrics instead of common sense and visualizations. I've worked with many quantitative researchers in the finance industry and those that are successful are always the least theoretical, do not rely on tests, and everything is visualized heavily and conclusions inferred from that. Gut feeling, intuition and visuals are the best protection against over-fitting, p-hacking and self-deception.