3 ms·

I may be wrong, but it seems to me that 20th century (theoretical) statistics research overemphasized efficiency at the expense of robustness. My guess is that

I may be wrong, but it seems to me that 20th century (theoretical) statistics research overemphasized efficiency at the expense of robustness. My guess is that this has to do with the (over-)mathematization of statistics in the past century, as opposed to a more empirical/engineering viewpoint. Efficiency typically only holds under extremely narrow (and often impossible to check) assumptions, which is great for mathematicians proving theorems and creating theories of efficiency. On the other hand, robustness is ideally about unknown unknowns and weak assumptions, which is hard to deal with mathematically. It seems already the 21st century is seeing a more balanced emphasis on theory vs. real world applications though.

chalst 6y ago
Not my impression. The kind of outlier-culling technique suggested by civilized is not recommended these days because it adds unprincipled choice points to what Andrew Gelman calls the 'Garden of Forking Paths' [1, 2]. Thus they are bad for hypothesis testing, which tends to be what most statisticians care about. Additionally the technique obscures the relationship between the variance of the sample and the population variance if we do not have reliable knowledge of the population distribution; likewise for the mean if the mean is not close to the mode. These problems can be quite dramatic for long-tailed distributions. [1]: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/p_... [2]: https://statmodeling.stat.columbia.edu/2016/09/30/why-the-garden-of-forking-paths-criticism-of-p-values-is-not-like-a-famous-borscht-belt-comedy-bit/ https://statmodeling.stat.columbia.edu/2016/09/30/why-the-ga...
- civilized 6y ago
  This post seems to be conflating a few different things: 1. Trimming for mean estimation, which removes extreme values in an algorithmic fashion 2. Subjective removal of outliers based on researcher judgment (this is the garden of forking paths Gelman talks about) 3. Estimating other distributional properties, such as the variance, with trimmed estimators These are all different things and come with different theoretical and practical risks and benefits. Trimmed means are perfectly good statistical tools, although they have their limitations like anything else.
  - chalst 6y ago
    I made two separate points. The choice of N used in cutting out the N% most extreme results is not determined by widely accepted statistical best practice. Hence it is a source of forks. The algorithm might be deterministic but the choice of this parameter isn't. My discussion of distributional properties was another issue concerning this technique. You seemed to have missed the point that dropping extreme points can also lead to biased estimates of the mean. Ten years ago, dropping outliers was considered good practice in the social sciences. Today, it has become a reason for rejection in peer review. There are better techniques for dealing with noisy data, such as adding measurements to data points to measure "badness" that can then be adjusted for in a multi-level model.
    - civilized 6y ago
      All but the simplest statistical estimators have researcher degrees of freedom (certainly including multilevel models) so it seems arbitrary to criticize the trimmed mean in particular for that "fault". Similarly, any estimator can be biased if its assumptions are violated, so I'm not sure why the potential bias of the trimmed mean in particular is an interesting point. I'm sure that social science peer reviewers have their reasons for their methodological preferences, but trimmed means are great workhorses in other areas of science, like signal processing. The critique strikes me as potentially valid in its subfield but a bit parochial if it is attempting generality.
      - chalst 6y ago
        I don't deny the technique has its uses. The point is it is a poor technique to use if your goal is hypothesis testing, which, as I said, is what most statisticians care about. I didn't reply to you, but to goodsector, who claimed that statisticians cared focus on efficiency at the expense of reliability. I dispute this.
        civilized 6y ago
        I agree! Statisticians have largely led the development of robust methods, so I don't see how they can be characterized as ignoring the concern.

chalst 6y ago

Not my impression. The kind of outlier-culling technique suggested by civilized is not recommended these days because it adds unprincipled choice points to what Andrew Gelman calls the 'Garden of Forking Paths' [1, 2]. Thus they are bad for hypothesis testing, which tends to be what most statisticians care about. Additionally the technique obscures the relationship between the variance of the sample and the population variance if we do not have reliable knowledge of the population distribution; likewise for the mean if the mean is not close to the mode. These problems can be quite dramatic for long-tailed distributions. [1]: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/p_... [2]: https://statmodeling.stat.columbia.edu/2016/09/30/why-the-garden-of-forking-paths-criticism-of-p-values-is-not-like-a-famous-borscht-belt-comedy-bit/ https://statmodeling.stat.columbia.edu/2016/09/30/why-the-ga...

civilized 6y ago
This post seems to be conflating a few different things: 1. Trimming for mean estimation, which removes extreme values in an algorithmic fashion 2. Subjective removal of outliers based on researcher judgment (this is the garden of forking paths Gelman talks about) 3. Estimating other distributional properties, such as the variance, with trimmed estimators These are all different things and come with different theoretical and practical risks and benefits. Trimmed means are perfectly good statistical tools, although they have their limitations like anything else.
- chalst 6y ago
  I made two separate points. The choice of N used in cutting out the N% most extreme results is not determined by widely accepted statistical best practice. Hence it is a source of forks. The algorithm might be deterministic but the choice of this parameter isn't. My discussion of distributional properties was another issue concerning this technique. You seemed to have missed the point that dropping extreme points can also lead to biased estimates of the mean. Ten years ago, dropping outliers was considered good practice in the social sciences. Today, it has become a reason for rejection in peer review. There are better techniques for dealing with noisy data, such as adding measurements to data points to measure "badness" that can then be adjusted for in a multi-level model.
  - civilized 6y ago
    All but the simplest statistical estimators have researcher degrees of freedom (certainly including multilevel models) so it seems arbitrary to criticize the trimmed mean in particular for that "fault". Similarly, any estimator can be biased if its assumptions are violated, so I'm not sure why the potential bias of the trimmed mean in particular is an interesting point. I'm sure that social science peer reviewers have their reasons for their methodological preferences, but trimmed means are great workhorses in other areas of science, like signal processing. The critique strikes me as potentially valid in its subfield but a bit parochial if it is attempting generality.
    - chalst 6y ago
      I don't deny the technique has its uses. The point is it is a poor technique to use if your goal is hypothesis testing, which, as I said, is what most statisticians care about. I didn't reply to you, but to goodsector, who claimed that statisticians cared focus on efficiency at the expense of reliability. I dispute this.
      - civilized 6y ago
        I agree! Statisticians have largely led the development of robust methods, so I don't see how they can be characterized as ignoring the concern.