5 ms·

> efficiency as the opposite of stability In statistics, there is a slight variant of this thesis that is true in a precise formal sense: the tradeoff between

> efficiency as the opposite of stability In statistics, there is a slight variant of this thesis that is true in a precise formal sense: the tradeoff between efficiency and "robustness" (stability in a non-ideal situation). For example, if you have a population sample, the most efficient way to estimate the population mean from your sample is the sample mean. But if some of the data are corrupted, you're better off with a robust estimator - in this case, a trimmed mean, where the extreme N% of high and low values are discarded. The trimmed mean is less efficient in the sense that, if none of the data are corrupted, it discards information and is less accurate than the full mean. But it's more robust in the sense that it remains accurate even when a small-to-moderate % of the data are corrupted.

clairity 6y ago
i stumbled on "stability' too, because it's a static quality. rather than robustness, i prefer to use the term resilience, a dynamic quality, since efficiency is also a dynamic quality. you can trade efficiency for resilience and vice versa (as the parent poster switched to later). edit: i should add that i don't entirely agree with the thesis of the article, which exhorts us to slow down, thereby trading efficiency away for resilience. there are a number of ways to add resilience (and trade away efficiency), and in some cases, slowing down might be the best, but it's certainly not the only, or best, option in most cases. for housing, an example used in the article, we could add more housing to create resilience, which requires reducing friction, like increasing the throughput of permitting/inspections while generally reducing zoning/regulations.
- throwaway815190 6y ago
  I'm completely ignorant on this topic, so I apologize for asking what must be an extremely stupid question to you, but: what makes stability a static quality whereas resilience is a dynamic quality? Are these statistical definitions that I can look up somewhere?
  - vulpesx2 6y ago
    Not OP, but my take is that stability is usually defined as a base-state that will continue onto perpetuity unless some outside force disrupts it. Resiliency is more closely defined as the ability to recover from disruptions back to the base-state quickly. Since the context here is that efficiency can remove layers of redundancy therefore allowing disruptions to wreck more havoc - I believe that's what OP was getting at.
- deleted 6y ago
  [deleted]
- civilized 6y ago
  I like resilience better as well - here it just happens that the technical terms of statistics match up fairly nicely with what we're trying to say.
- akendo 6y ago
  Do you have any reference or source of this statement? Just asking out of curiosity.
javert 6y ago
Wouldn't this be better described as a tradeoff between accuracy and robustness? Interesting concept.
- civilized 6y ago
  It's a bit different aspect of the same thing. If B can perform better than A given the same data, it usually means B can perform equal to A with less data. From wikipedia: > Essentially, a more efficient estimator, experiment, or test needs fewer observations than a less efficient one to achieve a given performance.
  - javert 6y ago
    Aha. That makes sense.
cmonnow 6y ago
it's called bias vs. variance tradeoff, or over-fitting, in stats/machine learning lingo.
- BeatLeJuce 6y ago
  No, it isn't. The b/v tradeoff is not the same thing. The efficiency of an estimator is a different from its bias.
  - ianhorn 6y ago
    > The efficiency of an estimator is a different from its bias. I think the comment is drawing a parallel to variance (better efficiency = lower variance). Still not exactly the same, I think, but pretty damn similar.
    - xapata 6y ago
      They're related in that less complex models will degrade more gracefully when making predictions on novel anomalies, and that in general model complexity drives the bias-variance trade-off. But, erring on the side of efficiency in this discussion is more like over-fitting, which implies an overly complex model. It's making your model too good for one situation, such that it fails to generalize. You'd rather pull back on accuracy and choose a simpler model, in the hopes that it's more resilient to novel observations.
mrtnmcc 6y ago
Another example would be forward error correction (adding parity bits to improve robustness at the expense of efficiency). But inefficiency isn't necessarally more robust unless the extra bits serve some purpose.
zzz95 6y ago
This also exists in control theory as the tradeoff between performance and robustness!
goodsector 6y ago
I may be wrong, but it seems to me that 20th century (theoretical) statistics research overemphasized efficiency at the expense of robustness. My guess is that this has to do with the (over-)mathematization of statistics in the past century, as opposed to a more empirical/engineering viewpoint. Efficiency typically only holds under extremely narrow (and often impossible to check) assumptions, which is great for mathematicians proving theorems and creating theories of efficiency. On the other hand, robustness is ideally about unknown unknowns and weak assumptions, which is hard to deal with mathematically. It seems already the 21st century is seeing a more balanced emphasis on theory vs. real world applications though.
- chalst 6y ago
  Not my impression. The kind of outlier-culling technique suggested by civilized is not recommended these days because it adds unprincipled choice points to what Andrew Gelman calls the 'Garden of Forking Paths' [1, 2]. Thus they are bad for hypothesis testing, which tends to be what most statisticians care about. Additionally the technique obscures the relationship between the variance of the sample and the population variance if we do not have reliable knowledge of the population distribution; likewise for the mean if the mean is not close to the mode. These problems can be quite dramatic for long-tailed distributions. [1]: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/p_... [2]: https://statmodeling.stat.columbia.edu/2016/09/30/why-the-garden-of-forking-paths-criticism-of-p-values-is-not-like-a-famous-borscht-belt-comedy-bit/ https://statmodeling.stat.columbia.edu/2016/09/30/why-the-ga...
  - civilized 6y ago
    This post seems to be conflating a few different things: 1. Trimming for mean estimation, which removes extreme values in an algorithmic fashion 2. Subjective removal of outliers based on researcher judgment (this is the garden of forking paths Gelman talks about) 3. Estimating other distributional properties, such as the variance, with trimmed estimators These are all different things and come with different theoretical and practical risks and benefits. Trimmed means are perfectly good statistical tools, although they have their limitations like anything else.

clairity 6y ago

i stumbled on "stability' too, because it's a static quality. rather than robustness, i prefer to use the term resilience, a dynamic quality, since efficiency is also a dynamic quality. you can trade efficiency for resilience and vice versa (as the parent poster switched to later). edit: i should add that i don't entirely agree with the thesis of the article, which exhorts us to slow down, thereby trading efficiency away for resilience. there are a number of ways to add resilience (and trade away efficiency), and in some cases, slowing down might be the best, but it's certainly not the only, or best, option in most cases. for housing, an example used in the article, we could add more housing to create resilience, which requires reducing friction, like increasing the throughput of permitting/inspections while generally reducing zoning/regulations.

throwaway815190 6y ago
I'm completely ignorant on this topic, so I apologize for asking what must be an extremely stupid question to you, but: what makes stability a static quality whereas resilience is a dynamic quality? Are these statistical definitions that I can look up somewhere?
- vulpesx2 6y ago
  Not OP, but my take is that stability is usually defined as a base-state that will continue onto perpetuity unless some outside force disrupts it. Resiliency is more closely defined as the ability to recover from disruptions back to the base-state quickly. Since the context here is that efficiency can remove layers of redundancy therefore allowing disruptions to wreck more havoc - I believe that's what OP was getting at.
deleted 6y ago
[deleted]
civilized 6y ago
I like resilience better as well - here it just happens that the technical terms of statistics match up fairly nicely with what we're trying to say.
akendo 6y ago
Do you have any reference or source of this statement? Just asking out of curiosity.

javert 6y ago

Wouldn't this be better described as a tradeoff between accuracy and robustness? Interesting concept.

civilized 6y ago
It's a bit different aspect of the same thing. If B can perform better than A given the same data, it usually means B can perform equal to A with less data. From wikipedia: > Essentially, a more efficient estimator, experiment, or test needs fewer observations than a less efficient one to achieve a given performance.
- javert 6y ago
  Aha. That makes sense.

cmonnow 6y ago

it's called bias vs. variance tradeoff, or over-fitting, in stats/machine learning lingo.

BeatLeJuce 6y ago
No, it isn't. The b/v tradeoff is not the same thing. The efficiency of an estimator is a different from its bias.
- ianhorn 6y ago
  > The efficiency of an estimator is a different from its bias. I think the comment is drawing a parallel to variance (better efficiency = lower variance). Still not exactly the same, I think, but pretty damn similar.
  - xapata 6y ago
    They're related in that less complex models will degrade more gracefully when making predictions on novel anomalies, and that in general model complexity drives the bias-variance trade-off. But, erring on the side of efficiency in this discussion is more like over-fitting, which implies an overly complex model. It's making your model too good for one situation, such that it fails to generalize. You'd rather pull back on accuracy and choose a simpler model, in the hopes that it's more resilient to novel observations.

mrtnmcc 6y ago

Another example would be forward error correction (adding parity bits to improve robustness at the expense of efficiency). But inefficiency isn't necessarally more robust unless the extra bits serve some purpose.

zzz95 6y ago

This also exists in control theory as the tradeoff between performance and robustness!

goodsector 6y ago

I may be wrong, but it seems to me that 20th century (theoretical) statistics research overemphasized efficiency at the expense of robustness. My guess is that this has to do with the (over-)mathematization of statistics in the past century, as opposed to a more empirical/engineering viewpoint. Efficiency typically only holds under extremely narrow (and often impossible to check) assumptions, which is great for mathematicians proving theorems and creating theories of efficiency. On the other hand, robustness is ideally about unknown unknowns and weak assumptions, which is hard to deal with mathematically. It seems already the 21st century is seeing a more balanced emphasis on theory vs. real world applications though.

chalst 6y ago
Not my impression. The kind of outlier-culling technique suggested by civilized is not recommended these days because it adds unprincipled choice points to what Andrew Gelman calls the 'Garden of Forking Paths' [1, 2]. Thus they are bad for hypothesis testing, which tends to be what most statisticians care about. Additionally the technique obscures the relationship between the variance of the sample and the population variance if we do not have reliable knowledge of the population distribution; likewise for the mean if the mean is not close to the mode. These problems can be quite dramatic for long-tailed distributions. [1]: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/p_... [2]: https://statmodeling.stat.columbia.edu/2016/09/30/why-the-garden-of-forking-paths-criticism-of-p-values-is-not-like-a-famous-borscht-belt-comedy-bit/ https://statmodeling.stat.columbia.edu/2016/09/30/why-the-ga...
- civilized 6y ago
  This post seems to be conflating a few different things: 1. Trimming for mean estimation, which removes extreme values in an algorithmic fashion 2. Subjective removal of outliers based on researcher judgment (this is the garden of forking paths Gelman talks about) 3. Estimating other distributional properties, such as the variance, with trimmed estimators These are all different things and come with different theoretical and practical risks and benefits. Trimmed means are perfectly good statistical tools, although they have their limitations like anything else.