4 ms·
Dropping outliers is common in statistical analysis.
by ceph_ 11y ago
Dropping outliers is common in statistical analysis.
- deleted 11y ago[deleted]
- jamiequint 11y agoDropping outliers can be done when outliers cloud the analysis, but doing this in an analysis of startups is inane since startup investors' entire goal is to find outliers.
- slugg 11y agoDon't just throw around some Peter Thiel shit like it justifies any argument you want it to.
- angelbob 11y agoPossible. In this case, we're not looking for outliers or measuring based on financial success, but trying to tell if the VC is systematically biased anti-woman. It's not clear that dropping outliers is a bad idea there. It's also not clear it's a good idea, granted.
- vasilipupkin 11y agoWell, if you are trying to measure whether men founders or women founders you have funded on average make more, then you would have to include Uber. The real issue with the analysis is that results are unlikely to be statistically significant due to small samples and high variance, which means they are useless.
- vasilipupkin 11y agoWell, if you are trying to measure whether men founders or women founders you have funded on average make more, then you would have to include Uber. The real issue with the analysis is that results are unlikely to be statistically significant due to small samples and high variance, which means they are useless.
- Pyxl101 11y agoWhy is it appropriate to drop outliers? (The fact that something is common does not make it a good thing.)
- venomsnake 11y agoStatistics 101. When you have samples you throw away the highest and lowest member, to counteract some random occurrence. The mean net worth of the patrons in any restaurant carlos slim frequents rises substantially when he is there.
- stkni 11y agoThat, and the fact that outliers can often be discounted due to measurement/instrumentation error. Moreover, the fact that Carlos entered your restaurant may be a significant event depending on the analysis that you're attempting to do. So you need to have to have a good rationale for dropping outliers, and you should probably also watch for bias when dropping outliers that don't support your hypothesis!
- Pyxl101 11y agoYes, because that is how mean net worth is defined. I don't see specifically what that argues against, except that mean is not the best indicator to use in all situations; perhaps a different indicator is appropriate, such as the income per patron by percentile. 100th percentile will be Carlos Slim, but 99th percentile and lower will be other patrons. If Carlos Slim actually does frequent the casino, then his attendance is an important part of understanding the situation.
- yummyfajitas 11y agoIt's still BS. Outliers are a signal that you don't have a simple, nicely decaying distribution. The right way to deal with outliers is to use a method that acknowledges their existence, not to ignore them. For example, if outliers destroy your OLS linear regression, it's because your error is not normal. That means you need to do Bayesian linear regression with a non-normal error term, not just throw them away.
- Scea91 11y agoDepends. Throwing outliers out without thinking is obviously wrong. In many instances outliers can be just invalid measurements and you should ignore them.
- yummyfajitas 11y agoA much better approach is to incorporate measurement error into your statistical procedure. Of course that's usually a lot easier to do with Bayesian techniques...
- dlss 11y ago> In many instances outliers can be just invalid measurements and you should ignore them. signal[i] = value[i] + noise[i]. If you know that value[i] == NaN, then by all means throw out signal[i]. If value[i] != NaN, then you're better off modeling error[i], and using that model to give you information about value[i] as yummyfajitas suggests. This is trivial to see if noise[i] == 0, but for some reason becomes progressively harder for people as noise[i] increases.
- bsder 11y agoDropping outliers is not common in good statistical analysis. In many labs, your data is looked at very suspiciously if you don't have any outliers. An outlier may not be thrown out without good reason. Preferably an a priori reason before you do the analysis.