4 ms·

Type I and type II errors

17 points by aburan28 13y ago

yiransheng 13y ago
In main street interpretation of statistical findings, type II errors are often swept under the rug. A good illustration of the danger in only telling the type I side of the story is [xkcd/1132](http://xkcd.com/1132/ http://xkcd.com/1132/), the frequentist looks hilariously stupid, and we all hail the Bayesian smartness. Yet a closer look from the frequentist view, the type II error of the test: H_0: the sun has gone nova is Pr[detector reports no|the sun has indeed exploded] = 1 - 1/36 > 95%; in other words the power of the test(1-Pr[type II error]) is extremely low (1/36). Even though the test has a reasonably low level alpha=0.05(type I error), it's still pretty useless as its power is also pathetic. The comic made the frequentist look idiotic, however, he's just not a good frequentist with a sufficient grasp of type II error. In reality, many people are like the freqentist, easily satisfied with a low p-value without thinking about the fundamental usefulness of their tests (power), becoming victims of type II errors. EDIT: I made a mistake, see eli_gottlieb's comment below for correction.
- eli_gottlieb 13y ago
  Actually, the comic specifies a 1/36 chance that the detector lies, no matter which result is true. So the P(detector reports no | sun has in fact exploded) = 1/36, making it a very strong test. The real problem there is that the p-value just doesn't mean what he thinks it means, and he in fact hasn't factored in any prior for how often the sun actually explodes.
  - yiransheng 13y ago
    Oops, you are right, I am a bad freqentist. Just had the discussion about the comic with a friend verbally, thought we figured it out. I guess it is really easy to get even the most basic concepts applied wrong.
    - eli_gottlieb 13y ago
      And this is why I am currently in my office at 8:15pm reading Jayne's probability theory book.
HCIdivision17 13y ago
And my favorite: Type III Errors, wherein the very premise for the calculation is flawed. Always be wary of precisely answering the wrong question. http://www.tandfonline.com/doi/abs/10.1080/01621459.1957.10501374 http://www.tandfonline.com/doi/abs/10.1080/01621459.1957.105...
dccoolgai 13y ago
This is one of those fundamental things they should just make mandatory in grade school. I constantly find myself explaining this to people I work with - and even though it's kind of intuitive ("penny wise, pound foolish", "straining gnats to swallow camels"), it helps to have a formal concept like this to hang the concept on.
nimrody 13y ago
The terms "false positive" and "false negative" are so much clear.
- mewwts 13y ago
  True, but we get to sound a lot smarter saying stuff like "type I error" and "type II error".

17 points by aburan28 13y ago

In main street interpretation of statistical findings, type II errors are often swept under the rug. A good illustration of the danger in only telling the type I side of the story is [xkcd/1132](http://xkcd.com/1132/ http://xkcd.com/1132/), the frequentist looks hilariously stupid, and we all hail the Bayesian smartness. Yet a closer look from the frequentist view, the type II error of the test: H_0: the sun has gone nova is Pr[detector reports no|the sun has indeed exploded] = 1 - 1/36 > 95%; in other words the power of the test(1-Pr[type II error]) is extremely low (1/36). Even though the test has a reasonably low level alpha=0.05(type I error), it's still pretty useless as its power is also pathetic. The comic made the frequentist look idiotic, however, he's just not a good frequentist with a sufficient grasp of type II error. In reality, many people are like the freqentist, easily satisfied with a low p-value without thinking about the fundamental usefulness of their tests (power), becoming victims of type II errors. EDIT: I made a mistake, see eli_gottlieb's comment below for correction.

eli_gottlieb 13y ago
Actually, the comic specifies a 1/36 chance that the detector lies, no matter which result is true. So the P(detector reports no | sun has in fact exploded) = 1/36, making it a very strong test. The real problem there is that the p-value just doesn't mean what he thinks it means, and he in fact hasn't factored in any prior for how often the sun actually explodes.
- yiransheng 13y ago
  Oops, you are right, I am a bad freqentist. Just had the discussion about the comic with a friend verbally, thought we figured it out. I guess it is really easy to get even the most basic concepts applied wrong.
  - eli_gottlieb 13y ago
    And this is why I am currently in my office at 8:15pm reading Jayne's probability theory book.

HCIdivision17 13y ago

And my favorite: Type III Errors, wherein the very premise for the calculation is flawed. Always be wary of precisely answering the wrong question. http://www.tandfonline.com/doi/abs/10.1080/01621459.1957.10501374 http://www.tandfonline.com/doi/abs/10.1080/01621459.1957.105...

dccoolgai 13y ago

This is one of those fundamental things they should just make mandatory in grade school. I constantly find myself explaining this to people I work with - and even though it's kind of intuitive ("penny wise, pound foolish", "straining gnats to swallow camels"), it helps to have a formal concept like this to hang the concept on.

nimrody 13y ago

The terms "false positive" and "false negative" are so much clear.

mewwts 13y ago
True, but we get to sound a lot smarter saying stuff like "type I error" and "type II error".