4 ms·

I agree with much of this, and tend to be fairly ecumenical/pragmatic in my own choice of tools, but there are two things that lead to the "identity statistics"

by mjn 14y ago

I agree with much of this, and tend to be fairly ecumenical/pragmatic in my own choice of tools, but there are two things that lead to the "identity statistics" that are only briefly covered here, I think. One is the entire philosophical debate, e.g. at least some Bayesians think arguments against the coherence of frequentist statistics are damning enough to make it questionable whether the methods should be considered rigorous statistics at all (admittedly this is basically the hardline view) [1]. The other is that it's not always agreed when it's appropriate to look for coverage versus to analyze beliefs, partly due to the philosophical debate, and partly because often what you ultimately want is a decision, and there are arguments for whether you should base decisions on frequentist-coverage machinery, or on belief-update machinery. For example, to move slightly afield from bounding a parameter, let's say we want an estimate of the region in which bombs are likely to fall. This can be formulated in frequentist statistics as a tolerance interval, with two decision thresholds, one for how many bombs we want to bound, and one for how confident we want to be in the bound: we want an interval that includes at least x% of the population with y% confidence, e.g. that with 99% confidence we'll bound 99% of bombs [2]. On the other hand, it can be formulated as a question about belief: essentially, we want to find the range in which we believe (for some suitably conservative definition of belief) we are going to find falling bombs, which Bayesian predictive statistics looks at. [1] One famous/infamous such argument: http://en.wikipedia.org/wiki/Likelihood_principle http://en.wikipedia.org/wiki/Likelihood_principle [2] I wrote a bit on why tolerance intervals should really be a more prominent part of the frequentist toolbox: http://www.kmjn.org/notes/tolerance_intervals.html http://www.kmjn.org/notes/tolerance_intervals.html

shardling 14y ago
>let's say we want an estimate of the region in which bombs are likely to fall. That seems like fundamentally the wrong sort of question. You never actually care directly about something like -- the point of statistics is to inform some decision-making process. And it almost always becomes more obvious how to proceed when you keep in mind what you're actually using the stats for.
- sillysaurus 14y ago
  the point of statistics is to inform some decision-making process. An estimate of the region in which bombs are likely to fall would directly inform your decision-making process ("Don't go over there!") so I don't understand your objection.
  - lisper 14y ago
    I don't know what shardling intended, but here's my take on it: the kind of estimate you want depends on the question you want to answer. I live in California, so the analysis I'm doing on the rockets coming out of Gaza is likely very different from the one being performed by the people living in Tel Aviv. And both of those analyses are different from the ones being performed by Hamas. So there is no such thing as a reliable "estimate of the region in which bombs are likely to fall" independent of the particular question you want to answer. Another example, from the original article: "a weather forecaster is good if it rains 95 percent of the times he says there is a 95 percent chance of rain" It's not clear whether or not there's something special about the number 95, or whether the intent is that a forecaster is good if it rains X% of the time he says there's an X% chance of rain for any X. So consider a 100 day period during which it rains 10 days, and a forecaster who every day predicts a 10% chance of rain. Is that a "good" forecast? If you're a farmer, it might be. If you're planning a picnic, not so much.
  - shardling 14y ago
    Because boiling everything down to a simple model like "regions where bombs are likely" is throwing away information. We do so because it's easier to analyze and think about a simple model than to try to deal with the whole dataset all at once -- except that if you have a specific question, it might be possible to treat that question directly. Because you're not losing information, doing so provides a more accurate result. In practice this might not matter, and simplified models might be good enough. But as computers become more powerful, it becomes easier to query the data directly.

by mjn 14y ago

shardling 14y ago

>let's say we want an estimate of the region in which bombs are likely to fall. That seems like fundamentally the wrong sort of question. You never actually care directly about something like -- the point of statistics is to inform some decision-making process. And it almost always becomes more obvious how to proceed when you keep in mind what you're actually using the stats for.

sillysaurus 14y ago
the point of statistics is to inform some decision-making process. An estimate of the region in which bombs are likely to fall would directly inform your decision-making process ("Don't go over there!") so I don't understand your objection.
- lisper 14y ago
  I don't know what shardling intended, but here's my take on it: the kind of estimate you want depends on the question you want to answer. I live in California, so the analysis I'm doing on the rockets coming out of Gaza is likely very different from the one being performed by the people living in Tel Aviv. And both of those analyses are different from the ones being performed by Hamas. So there is no such thing as a reliable "estimate of the region in which bombs are likely to fall" independent of the particular question you want to answer. Another example, from the original article: "a weather forecaster is good if it rains 95 percent of the times he says there is a 95 percent chance of rain" It's not clear whether or not there's something special about the number 95, or whether the intent is that a forecaster is good if it rains X% of the time he says there's an X% chance of rain for any X. So consider a 100 day period during which it rains 10 days, and a forecaster who every day predicts a 10% chance of rain. Is that a "good" forecast? If you're a farmer, it might be. If you're planning a picnic, not so much.
- shardling 14y ago
  Because boiling everything down to a simple model like "regions where bombs are likely" is throwing away information. We do so because it's easier to analyze and think about a simple model than to try to deal with the whole dataset all at once -- except that if you have a specific question, it might be possible to treat that question directly. Because you're not losing information, doing so provides a more accurate result. In practice this might not matter, and simplified models might be good enough. But as computers become more powerful, it becomes easier to query the data directly.