4 ms·

> You should strive to obtain a deep understanding of your data and its distribution, and you should be deeply skeptical if the sample you happen to have looks

> You should strive to obtain a deep understanding of your data and its distribution, and you should be deeply skeptical if the sample you happen to have looks normal. Although normality testing is useless in many situations, the parent comment somewhat overstates the degree of caution required. In many contexts the exact distribution doesn't matter; sort-of-normal is good enough. For example, the t-test is used ubiquitously. It assumes normality, so we would expect possible non-normality to be a major problem, right? Not so. The t-test is extremely robust to departures from normality given equal sample sizes [1,2,3]. Or you can use a so-called non-parametric test. Rather than investing great effort in specifying exactly what distribution you're dealing with, it's more productive to simply use a test that is robust against your unknowns and move on to pursuing your actual objectives. It's true that if you are interested in predicting events in the tails of the distribution, you really do need to study the distribution in detail. Predicting rare events is very difficult. But if you're just interested in differences between group means, don't overthink it. [1] http://www.jerrydallal.com/LHSP/student3.htm http://www.jerrydallal.com/LHSP/student3.htm [2] Posten, H.O., Yeh, H.C., and Owen, D.B. (1977). Robustness of the two-sample t-test under violations of the homogeneity of variance assumption. Communications in Statistics - Theory and Methods 11, 109–126. [3] Posten, H.O. (1992). Robustness of the two-sample t-test under violations of the homogeneity of variance assumption, part ii. Communications in Statistics - Theory and Methods 21, 2169–2184.

srean 6y ago
> The t-test is extremely robust to departures from normality given equal sample sizes That's over selling it. Its very sensitive to fat tails and skew. That's the reason robust testing and estimation is a thing. Wilcox's research would be a good near contemporary place to start [0][1]. [0] Wilcox, Robustness of Standard Tests https://onlinelibrary.wiley.com/doi/abs/10.1002/9781118445112.stat06350 https://onlinelibrary.wiley.com/doi/abs/10.1002/978111844511... (paywalled) Abstract: Conventional hypothesis‐testing methods such as student's t, the ANOVA F test, and methods based on the ordinary least squares regression estimator, are not robust to violations of assumptions. In fact, there are general conditions under which these methods can provide poor control over the probability of a type I error and inaccurate confidence intervals, no matter how large the sample sizes might be. Relatively poor power is yet another concern. [1] Fundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy https://books.google.co.in/books?id=uUNGzhdxk0kC https://books.google.co.in/books?id=uUNGzhdxk0kC Conventional statistical methods have a very serious flaw. They routinely miss differences among groups or associations among variables that are detected by more modern techniques - even under very small departures from normality. Hundreds of journal articles have described the reasons standard techniques can be unsatisfactory, but simple, intuitive explanations are generally unavailable. Improved methods have been derived, but they are far from obvious or intuitive based on the training most researchers receive. Situations arise where even highly nonsignificant results become significant when analyzed with more modern methods. Without assuming any prior training in statistics, Part I of this book describes basic statistical principles from a point of view that makes their shortcomings intuitive and easy to understand.
- jpeloquin 6y ago
  > Its very sensitive to fat tails and skew. That's the reason robust testing and estimation is a thing. Yeah, I shouldn't have said "extremely". Its much more robust than is commonly perceived, but that does not make it "extremely" robust. Thank you for the correction. But please note that I specified equal sample sizes, so the fact that "there are general conditions under which these methods can provide poor control over the probability of a type I error" is not really a refutation—I already implied that the t-test is not generally (= in all cases) robust. I also mentioned non-parametric tests as a potential alternative. I do not wish to imply that the t-test is always the right choice. But I stand by the assertion that approximate knowledge of the distribution, such as obtained by inspecting a histogram, is perfectly adequate to choose a test. The main point remains — list what you know about your data, and pick a test that tells you what you want to know with a tolerable error level for your application. You can spend an awful lot of time picking the "correct" statistical test (if there is such a thing; tradeoffs exist) with little gain. Worse, making a lot of decisions about what test to use after looking at the data leads to p-hacking, potentially leaving you with more bias than if you naively used a slightly-wrong test from the start. To elaborate on the t-test discussion: From your ref [1], "If sampling is from nonnormal distributions that are absolutely identical, so in particular the variances are equal, the probability of a Type I error will not exceed 0.05 by very much, assuming the method is applied with the desired probability of a Type I error set at α = 0.05. These two results have been known for some time and have been verified in various studies conducted in more recent years" (page 79). I think Wilcox makes this statement under the equal sample size caveat, but the text is unclear on this point. This does not guarantee robustness under nonnormality, but it does refute the idea that nonnormality is always fatal to the t-test. Yes, a t-test can be affected by skew even with equal sample sizes. Whether this is a problem depends on the application. The example in [1] uses a lognormal distribution and ends up with actual α = 0.15 for n = 20, desired α = 0.05. Which isn't great, but is somewhat tolerable. Also, a lognormal distribution is strongly skew right and left-truncated, and this is easily noticeable on a plot and can be transformed to a normal distribution. So this does not refute the idea that a sort-of-normal distribution is fatal to the t-test. You can look up the potential α level and β level errors from violating the assumptions of a particular test—they're tabulated. E.g., applying the t-test to a Pearson distribution produces a typical α level error of 0.005 at desired α = 0.05 [2]. That's definitely tolerable. This is why I stated that sort-of-normal is good enough for a t-test. It's also fairly straightforward to calculate the expected errors yourself for a particular situation, if reassurance is needed. If your statistics are intended to support a high-stakes decision this may be a good use of time. Working to validate your finding by other means is better, though. I'm not sure what your ref [0] (the "Robustness of Standard Tests" book chapter) is arguing for or against from the abstract alone. I don't have a copy of that book. The abstract mentions most flavors of maximum likelihood ratio tests, which is too broad a set of topics to discuss effectively. The implication seems to be that null hypothesis statistical tests are bad and something else (Bayesian analysis? visual inspection?) is better, which I don't necessarily disagree with. If you could please clarify its contents and if it is worth tracking down, I would appreciate it. [0] Wilcox, Robustness of Standard Tests [1] Fundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy [2] Posten, H.O. (1984). Robustness of the Two-Sample T-Test. In Robustness of Statistical Methods and Nonparametric Statistics, D. Rasch, and M.L. Tiku, eds. (Springer, Dordrecht), pp. 92–99.
BeetleB 6y ago
I'm guessing his concern is visually looking at a histogram and gauging its normality vs doing so with a q-q plot, which will likely show deviations from normality much better.

srean 6y ago

> The t-test is extremely robust to departures from normality given equal sample sizes That's over selling it. Its very sensitive to fat tails and skew. That's the reason robust testing and estimation is a thing. Wilcox's research would be a good near contemporary place to start [0][1]. [0] Wilcox, Robustness of Standard Tests https://onlinelibrary.wiley.com/doi/abs/10.1002/9781118445112.stat06350 https://onlinelibrary.wiley.com/doi/abs/10.1002/978111844511... (paywalled) Abstract: Conventional hypothesis‐testing methods such as student's t, the ANOVA F test, and methods based on the ordinary least squares regression estimator, are not robust to violations of assumptions. In fact, there are general conditions under which these methods can provide poor control over the probability of a type I error and inaccurate confidence intervals, no matter how large the sample sizes might be. Relatively poor power is yet another concern. [1] Fundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy https://books.google.co.in/books?id=uUNGzhdxk0kC https://books.google.co.in/books?id=uUNGzhdxk0kC Conventional statistical methods have a very serious flaw. They routinely miss differences among groups or associations among variables that are detected by more modern techniques - even under very small departures from normality. Hundreds of journal articles have described the reasons standard techniques can be unsatisfactory, but simple, intuitive explanations are generally unavailable. Improved methods have been derived, but they are far from obvious or intuitive based on the training most researchers receive. Situations arise where even highly nonsignificant results become significant when analyzed with more modern methods. Without assuming any prior training in statistics, Part I of this book describes basic statistical principles from a point of view that makes their shortcomings intuitive and easy to understand.

jpeloquin 6y ago
> Its very sensitive to fat tails and skew. That's the reason robust testing and estimation is a thing. Yeah, I shouldn't have said "extremely". Its much more robust than is commonly perceived, but that does not make it "extremely" robust. Thank you for the correction. But please note that I specified equal sample sizes, so the fact that "there are general conditions under which these methods can provide poor control over the probability of a type I error" is not really a refutation—I already implied that the t-test is not generally (= in all cases) robust. I also mentioned non-parametric tests as a potential alternative. I do not wish to imply that the t-test is always the right choice. But I stand by the assertion that approximate knowledge of the distribution, such as obtained by inspecting a histogram, is perfectly adequate to choose a test. The main point remains — list what you know about your data, and pick a test that tells you what you want to know with a tolerable error level for your application. You can spend an awful lot of time picking the "correct" statistical test (if there is such a thing; tradeoffs exist) with little gain. Worse, making a lot of decisions about what test to use after looking at the data leads to p-hacking, potentially leaving you with more bias than if you naively used a slightly-wrong test from the start. To elaborate on the t-test discussion: From your ref [1], "If sampling is from nonnormal distributions that are absolutely identical, so in particular the variances are equal, the probability of a Type I error will not exceed 0.05 by very much, assuming the method is applied with the desired probability of a Type I error set at α = 0.05. These two results have been known for some time and have been verified in various studies conducted in more recent years" (page 79). I think Wilcox makes this statement under the equal sample size caveat, but the text is unclear on this point. This does not guarantee robustness under nonnormality, but it does refute the idea that nonnormality is always fatal to the t-test. Yes, a t-test can be affected by skew even with equal sample sizes. Whether this is a problem depends on the application. The example in [1] uses a lognormal distribution and ends up with actual α = 0.15 for n = 20, desired α = 0.05. Which isn't great, but is somewhat tolerable. Also, a lognormal distribution is strongly skew right and left-truncated, and this is easily noticeable on a plot and can be transformed to a normal distribution. So this does not refute the idea that a sort-of-normal distribution is fatal to the t-test. You can look up the potential α level and β level errors from violating the assumptions of a particular test—they're tabulated. E.g., applying the t-test to a Pearson distribution produces a typical α level error of 0.005 at desired α = 0.05 [2]. That's definitely tolerable. This is why I stated that sort-of-normal is good enough for a t-test. It's also fairly straightforward to calculate the expected errors yourself for a particular situation, if reassurance is needed. If your statistics are intended to support a high-stakes decision this may be a good use of time. Working to validate your finding by other means is better, though. I'm not sure what your ref [0] (the "Robustness of Standard Tests" book chapter) is arguing for or against from the abstract alone. I don't have a copy of that book. The abstract mentions most flavors of maximum likelihood ratio tests, which is too broad a set of topics to discuss effectively. The implication seems to be that null hypothesis statistical tests are bad and something else (Bayesian analysis? visual inspection?) is better, which I don't necessarily disagree with. If you could please clarify its contents and if it is worth tracking down, I would appreciate it. [0] Wilcox, Robustness of Standard Tests [1] Fundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy [2] Posten, H.O. (1984). Robustness of the Two-Sample T-Test. In Robustness of Statistical Methods and Nonparametric Statistics, D. Rasch, and M.L. Tiku, eds. (Springer, Dordrecht), pp. 92–99.

BeetleB 6y ago

I'm guessing his concern is visually looking at a histogram and gauging its normality vs doing so with a q-q plot, which will likely show deviations from normality much better.