4 ms·

If you're looking for things to add big-picture-wise, it might be helpful to specify what assumptions go into various tests/methods. In my experience, this is t

If you're looking for things to add big-picture-wise, it might be helpful to specify what assumptions go into various tests/methods. In my experience, this is the biggest hangup and mistake, because a) it's more difficult to understand and b) ignoring it gives the appearance of rigor even if the test used is inappropriate for the data. I should emphasize that this is not a nitpick or even a criticism, just a feature I would love to see. It's also what I spend a large portion of my time trying to track down, so having it in a convenient location would be nice.

scottedwards 13y ago
Great suggestion. I've been amazed to find out that many coders and amateur "data scientists" don't realize that testing the assumptions is an important part of conducting statistical analyses. Part of this may be due to the recent emphasis on machine learning techniques, which tend to be assumption-free (often just assuming independence of cases in the sample).
- yummyfajitas 13y ago
  Part of this may be due to the recent emphasis on machine learning techniques, which tend to be assumption-free... No statistical technique is assumption free, unless it is purely descriptive. Some of them are free of explicit assumptions known by the practitioner, but that's not the same thing. In much the same way, my code is all bug-free.
- gtani 13y ago
  machine learning techniques, which tend to be assumption-free ML should be a rigorous exercise in Bayesian and classical/frequentist stats, computational methods, dataset integrity, visualization etc, if you've been thru the texts by Murphy or Bishop. It often happens that people a couple years out of their last stats class only retain that high R-squared, p-, t- and f-values are what they're looking for, and heteroskedasticity and sphericity are just big words. My evidence that ML is a rigorous exercise: the free texts listed (Barber, Mackay and Smola's are excellent, ESL not as accessible) http://metaoptimize.com/qa/questions/186/good-freely-available-textbooks-on-machine-learning http://metaoptimize.com/qa/questions/186/good-freely-availab...
  - scottedwards 13y ago
    Thanks, @gtani, great resource. Yeah didn't mean to imply that ML techniques are free of ANY assumptions, just that several of the popular ones like logistic regression don't have distributional assumptions. (actually, I really want to understand the VC Inequality at some point, as it seems to allow us to make conclusions about out of sample error rates without depending on distributional assumptions)
timr 13y ago
This guy has written up many of the most common statistical tests, their interpretations, and their assumptions in a very human-readable way: http://udel.edu/~mcdonald/statintro.html http://udel.edu/~mcdonald/statintro.html I point a lot of newbs to pages on that site so that they can develop a better intuition for the methods.
- buzzkillr2 13y ago
  This is an amazing resource thank you. I wish I had this last semester during my statistics course.
  - timr 13y ago
    If you're looking for more of this, but in greater depth, pick up a copy of Biometry. It's written for biologists, but you don't really need to know much biology to work through the examples, and the focus is inherently practical.
scottfr 13y ago
Just as a rule of thumb, in general the most important assumption that is violated is independence. If the data is not independent and is instead positively correlated (the most likely form of correlation in practice) most equations will overestimate statistical significance. A small amount of correlation can lead to very biased estimates of significance. Correlation between observations will generally appear in any time series data or data that is arranged spatially. Again, just a rule of thumb, but you should be very wary of the lack of independence between observations.
- vwinsyee 13y ago
  To add to this comment, large correlation between (or among) variables will often inflate estimates of standard error, leading to underestimates of statistical significance.

scottedwards 13y ago

Great suggestion. I've been amazed to find out that many coders and amateur "data scientists" don't realize that testing the assumptions is an important part of conducting statistical analyses. Part of this may be due to the recent emphasis on machine learning techniques, which tend to be assumption-free (often just assuming independence of cases in the sample).

yummyfajitas 13y ago
Part of this may be due to the recent emphasis on machine learning techniques, which tend to be assumption-free... No statistical technique is assumption free, unless it is purely descriptive. Some of them are free of explicit assumptions known by the practitioner, but that's not the same thing. In much the same way, my code is all bug-free.
gtani 13y ago
machine learning techniques, which tend to be assumption-free ML should be a rigorous exercise in Bayesian and classical/frequentist stats, computational methods, dataset integrity, visualization etc, if you've been thru the texts by Murphy or Bishop. It often happens that people a couple years out of their last stats class only retain that high R-squared, p-, t- and f-values are what they're looking for, and heteroskedasticity and sphericity are just big words. My evidence that ML is a rigorous exercise: the free texts listed (Barber, Mackay and Smola's are excellent, ESL not as accessible) http://metaoptimize.com/qa/questions/186/good-freely-available-textbooks-on-machine-learning http://metaoptimize.com/qa/questions/186/good-freely-availab...
- scottedwards 13y ago
  Thanks, @gtani, great resource. Yeah didn't mean to imply that ML techniques are free of ANY assumptions, just that several of the popular ones like logistic regression don't have distributional assumptions. (actually, I really want to understand the VC Inequality at some point, as it seems to allow us to make conclusions about out of sample error rates without depending on distributional assumptions)

timr 13y ago

This guy has written up many of the most common statistical tests, their interpretations, and their assumptions in a very human-readable way: http://udel.edu/~mcdonald/statintro.html http://udel.edu/~mcdonald/statintro.html I point a lot of newbs to pages on that site so that they can develop a better intuition for the methods.

buzzkillr2 13y ago
This is an amazing resource thank you. I wish I had this last semester during my statistics course.
- timr 13y ago
  If you're looking for more of this, but in greater depth, pick up a copy of Biometry. It's written for biologists, but you don't really need to know much biology to work through the examples, and the focus is inherently practical.

scottfr 13y ago

Just as a rule of thumb, in general the most important assumption that is violated is independence. If the data is not independent and is instead positively correlated (the most likely form of correlation in practice) most equations will overestimate statistical significance. A small amount of correlation can lead to very biased estimates of significance. Correlation between observations will generally appear in any time series data or data that is arranged spatially. Again, just a rule of thumb, but you should be very wary of the lack of independence between observations.

vwinsyee 13y ago
To add to this comment, large correlation between (or among) variables will often inflate estimates of standard error, leading to underestimates of statistical significance.