6 ms·

This is an excellent example of reasoning that Dawkins refers to as 'PETWAC' (Population of Events That Would happen to Appear Coincidental). Consider three po

This is an excellent example of reasoning that Dawkins refers to as 'PETWAC' (Population of Events That Would happen to Appear Coincidental). Consider three possible scenarios: 1) female hurricanes, as a set, are deadlier than male hurricanes 2) male hurricanes, as a set, are deadlier than female hurricanes 3) neither male nor female hurricanes are deadlier Surprisingly, if you assume a uniform distribution of deadliness independent of the names, you actually find that it's more likely that either 1 or 2 is true than that 3 is true; 3 being true would require that an arbitrary partitioning of the hurricanes would have resulted in equal deadliness, but if the partition is arbitrary, it's more likely that it creates two unequal groups. However, a quick appeal to symmetry shows that either 1 or 2 being true are equally newsworthy. It's like the news getting excited that a coin-toss was heads... "Wow, look at how heads that coin toss was!"... When in reality, the odds of heads OR tails were exceptionally high and the newsworthy item would be the coin landing on edge.

jjoonathan 12y ago
Given that this was published in PNAS and edited by someone from Princeton, it's safe to assume (unfortunately it's paywalled so I can only assume) that the authors know how to perform a significance test.
- Anderkent 12y ago
  http://xkcd.com/882/ http://xkcd.com/882/
  - jjoonathan 12y ago
    Exactly. What's at play is a combination of publication bias and (possibly) a slightly-too-narrow null hypothesis. This kind of silliness can squeeze through without any need for the null hypothesis to be entirely absent, which is the aspect of fixermark's post that I object to. I'd bet good money at 5:1 odds that the null hypothesis the paper used was more sophisticated than his uniform distribution let alone the delta distribution he accuses them of using.
    - ronaldx 12y ago
      You may have missed that the result is not significant, despite the headline, when considering the real data. Only laboratory questioning showed a significant psychological bias (and there an unnamed hurricane was rated less risky than either).
      - jjoonathan 12y ago
        > We use more than six decades of death rates from US hurricanes to show that feminine-named hurricanes cause significantly more deaths What am I missing?
        6d0debc071 12y ago
        From post down the page by tekp2 - https://news.ycombinator.com/item?id=7836326 https://news.ycombinator.com/item?id=7836326 http://phenomena.nationalgeographic.com/2014/06/02/why-have-female-hurricanes-killed-more-people-than-male-ones/ http://phenomena.nationalgeographic.com/2014/06/02/why-have-... "But Lazo thinks that neither the archival analysis nor the psychological experiments support the team’s conclusions. For a start, they analysed hurricane data from 1950, but hurricanes all had female names at first. They only started getting male names on alternate years in 1979. This matters because hurricanes have also, on average, been getting less deadly over time. “It could be that more people die in female-named hurricanes, simply because more people died in hurricanes on average before they started getting male names,” says Lazo. Jung’s team tried to address this problem by separately analysing the data for hurricanes before and after 1979. They claim that the findings “directionally replicated those in the full dataset” but that’s a bit of a fudge. The fact is they couldn’t find a significant link between the femininity of a hurricane’s name and the damage it caused for either the pre-1979 set or the post-1979 one (and a “marginally significant interaction” of p=0.073 doesn’t really count [this is a link in the actual article - http://mchankins.wordpress.com/2013/04/21/still-not-significant-2/ http://mchankins.wordpress.com/2013/04/21/still-not-signific... ]). The team argues that splitting the data meant there weren’t enough hurricanes in each subset to provide enough statistical power. But that only means we can’t rule out a connection between gender and damage; we can’t soundly confirm one either."
Retric 12y ago
Which is more likely everyone involved in publishing this study is an idiot or the difference is non trivial. IMO, this is much more likely that fixermark is suffering from http://en.wikipedia.org/wiki/Dunning–Kruger_effect http://en.wikipedia.org/wiki/Dunning–Kruger_effect, but I have also not read the study. Edit: That came off as overly harsh my point was the chance of 3 is not close to zero.
- fixermark 12y ago
  Nobody has to be an idiot for this study to get published; it's all about incentives. My point is that if the data showed that male hurricanes were deadlier than female hurricanes, this team could very will still have published a very similar paper reaching very similar but gender-reversed conclusions based upon the same laboratory experiments. That still definitely leaves open the question of how possible (3) was. Since I'm not a peer (i.e. the study is paywalled and I don't have access), I can't peer-review their figures, so I'm left critiquing from the outside based on hypothesis and conjecture. If anyone has access to the raw numbers, I'd be happy to follow their research.
  - deleted 12y ago
    [deleted]
deleted 12y ago
[deleted]
Houshalter 12y ago
If you flip a coin a thousand times and it always comes up heads, it's fairly safe to assume it will come up heads the thousandth and one time. I know we should be skeptical at everything, but I'm going to assume the journal and authors aren't completely incompetent to make a mistake like that. Of course there is prior probability to factor in, and it doesn't seem very likely a priori that this would be the case. So the effect size would have to be fairly large or have a large number of examples to convince me of this.
- baddox 12y ago
  > If you flip a coin a thousand times and it always comes up heads, it's fairly safe to assume it will come up heads the thousandth and one time. Unless you have reason to believe that it's a fair coin.
  - Houshalter 12y ago
    After a thousand times it's pretty unlikely it's a fair coin. If it came up heads a million times it's pretty much impossible. Even very small prior probabilities will become dominant with enough evidence.
    - coherentpony 12y ago
      If you're given a fair coin, then it's impossible for it to also be a biased coin. If you're given a coin, and you don't know whether it's fair or biased, then you can do an inference to give an estimated probability of how biased the coin is. Those are the two situations. This is a problem about perfect versus imperfect information, not about probabilities.
      - deleted 12y ago
        [deleted]
      - Houshalter 12y ago
        Nowhere was it stated the coin was 100% guaranteed to be fair. You are missing the point of my original example. My point was you can estimate the probability of a hypothesis by looking at the data. E.g. is it a fair or biased coin, and see how many times it came up heads or tails. Or are female named hurricanes more likely to kill than male ones or not?
aaron695 12y ago
This is totally and utterly wrong. This experiment has been found statistical true. One side has been found to go it's way NOT by chance. This is how peer review / science works. That said there are issues, this is not one of them. If you can read the article they also performed 6 other experiments where they for instance show people a weather map, tell them the name (M/F) and ask if they should evacuate. But I have no idea how they justify including years where male names weren't allocated.
- cwyers 12y ago
  That's... not what statistical significance means. All statistical significance means is that there's a less than five percent chance of getting that result at random, if the null hypothesis were true. Now, granted, most studies reported on find p-values well below the .05 threshold. But still, if you go out and count the number of studies published in a year, one percent of them is still a LOT of wrong studies. And remember that, due to publication bias, it's a biased sample of the population of studies. So for every 100 studies that are examining a hypothesis that is in reality false, you'll get 4-5 that get a statistically significant finding instead. And it's those few studies that end up getting published.
  - aaron695 12y ago
    This is also not true. For starters many studies don't use a p value of .05. People seem to have read one xkcd comic and think now all studies are false. There are issues, especially were the people doing the studies want a particular outcome for financial gain. But if there are issues with this study talk about them. Were are the other 19 studies they have hidden? Where have they not followed proper statistical practice? How does the other 6 theoretical experiments on the same paper which some also have significance come into it?
quacker 12y ago
Surprisingly, if you assume a uniform distribution of deadliness independent of the names, you actually find that it's more likely that either 1 or 2 is true than that 3 is true Right, but if you are a statistician what you should care about is the "distance" of either 1 or 2 from 3. It would not be surprising that one group is deadlier than the other, but it should be surprising if female hurricanes are "quite a bit" deadlier than male hurricanes (or vice versa), assuming this uniform distribution that you mentioned.
- coherentpony 12y ago
  Would you be surprised if you won the lottery? Yes. I imagine you'd also be ecstatic. However, the likelihood of your numbers being chosen had equal probability (assuming the numbers are drawn uniformly at random) with any other set of numbers anybody else picked. If someone else had won the lottery instead of you, they would also be surprised. Female hurricanes being 'quite a bit deadlier' than male hurricanes may well be surprising but it has equal probability with the event that male hurricanes are 'quite a bit deadlier' than female hurricanes. What you're describing is a rare event. Rare events happen, just rarely.
  - quacker 12y ago
    Female hurricanes being 'quite a bit deadlier' than male hurricanes may well be surprising but it has equal probability with the event that male hurricanes are 'quite a bit deadlier' than female hurricanes. I definitely agree with this. What you're describing is a rare event. Rare events happen, just rarely. I'll restate this hurricane problem. Suppose we're flipping a fair coin. We should expect to see roughly equal numbers of heads and tails. If I flip the same coin a thousand times and see 523 heads, that seems "reasonable". It's certainly possible to get 873 heads, but I might start to wonder if the coin is really fair not because it's very unlikely to get 873 heads with a fair coin (and it is very unlikely) but because it's more plausible that the coin is biased. That is, it's more likely that a coin that gets heads with some probability near 0.873 produces 873 heads in a thousand flips than it is for fair coin to produce 873 heads in a thousand flips. In the context of hurricanes, if we assign genders randomly we expect close to equal deadliness between male and female hurricanes. If there have been a thousand hurricanes, and female hurricanes cause 87.3% of all hurricane deaths, well... I might start wondering if giving hurricanes female names somehow makes a hurricane more deadly. Of course, this is all in the context of a uniform distribution of names on hurricanes.

jjoonathan 12y ago

Given that this was published in PNAS and edited by someone from Princeton, it's safe to assume (unfortunately it's paywalled so I can only assume) that the authors know how to perform a significance test.

Anderkent 12y ago
http://xkcd.com/882/ http://xkcd.com/882/
- jjoonathan 12y ago
  Exactly. What's at play is a combination of publication bias and (possibly) a slightly-too-narrow null hypothesis. This kind of silliness can squeeze through without any need for the null hypothesis to be entirely absent, which is the aspect of fixermark's post that I object to. I'd bet good money at 5:1 odds that the null hypothesis the paper used was more sophisticated than his uniform distribution let alone the delta distribution he accuses them of using.
  - ronaldx 12y ago
    You may have missed that the result is not significant, despite the headline, when considering the real data. Only laboratory questioning showed a significant psychological bias (and there an unnamed hurricane was rated less risky than either).
    - jjoonathan 12y ago
      > We use more than six decades of death rates from US hurricanes to show that feminine-named hurricanes cause significantly more deaths What am I missing?
      - 6d0debc071 12y ago
        From post down the page by tekp2 - https://news.ycombinator.com/item?id=7836326 https://news.ycombinator.com/item?id=7836326 http://phenomena.nationalgeographic.com/2014/06/02/why-have-female-hurricanes-killed-more-people-than-male-ones/ http://phenomena.nationalgeographic.com/2014/06/02/why-have-... "But Lazo thinks that neither the archival analysis nor the psychological experiments support the team’s conclusions. For a start, they analysed hurricane data from 1950, but hurricanes all had female names at first. They only started getting male names on alternate years in 1979. This matters because hurricanes have also, on average, been getting less deadly over time. “It could be that more people die in female-named hurricanes, simply because more people died in hurricanes on average before they started getting male names,” says Lazo. Jung’s team tried to address this problem by separately analysing the data for hurricanes before and after 1979. They claim that the findings “directionally replicated those in the full dataset” but that’s a bit of a fudge. The fact is they couldn’t find a significant link between the femininity of a hurricane’s name and the damage it caused for either the pre-1979 set or the post-1979 one (and a “marginally significant interaction” of p=0.073 doesn’t really count [this is a link in the actual article - http://mchankins.wordpress.com/2013/04/21/still-not-significant-2/ http://mchankins.wordpress.com/2013/04/21/still-not-signific... ]). The team argues that splitting the data meant there weren’t enough hurricanes in each subset to provide enough statistical power. But that only means we can’t rule out a connection between gender and damage; we can’t soundly confirm one either."

Retric 12y ago

Which is more likely everyone involved in publishing this study is an idiot or the difference is non trivial. IMO, this is much more likely that fixermark is suffering from http://en.wikipedia.org/wiki/Dunning–Kruger_effect http://en.wikipedia.org/wiki/Dunning–Kruger_effect, but I have also not read the study. Edit: That came off as overly harsh my point was the chance of 3 is not close to zero.

fixermark 12y ago
Nobody has to be an idiot for this study to get published; it's all about incentives. My point is that if the data showed that male hurricanes were deadlier than female hurricanes, this team could very will still have published a very similar paper reaching very similar but gender-reversed conclusions based upon the same laboratory experiments. That still definitely leaves open the question of how possible (3) was. Since I'm not a peer (i.e. the study is paywalled and I don't have access), I can't peer-review their figures, so I'm left critiquing from the outside based on hypothesis and conjecture. If anyone has access to the raw numbers, I'd be happy to follow their research.
- deleted 12y ago
  [deleted]

deleted 12y ago

[deleted]

Houshalter 12y ago

If you flip a coin a thousand times and it always comes up heads, it's fairly safe to assume it will come up heads the thousandth and one time. I know we should be skeptical at everything, but I'm going to assume the journal and authors aren't completely incompetent to make a mistake like that. Of course there is prior probability to factor in, and it doesn't seem very likely a priori that this would be the case. So the effect size would have to be fairly large or have a large number of examples to convince me of this.

baddox 12y ago
> If you flip a coin a thousand times and it always comes up heads, it's fairly safe to assume it will come up heads the thousandth and one time. Unless you have reason to believe that it's a fair coin.
- Houshalter 12y ago
  After a thousand times it's pretty unlikely it's a fair coin. If it came up heads a million times it's pretty much impossible. Even very small prior probabilities will become dominant with enough evidence.
  - coherentpony 12y ago
    If you're given a fair coin, then it's impossible for it to also be a biased coin. If you're given a coin, and you don't know whether it's fair or biased, then you can do an inference to give an estimated probability of how biased the coin is. Those are the two situations. This is a problem about perfect versus imperfect information, not about probabilities.
    - deleted 12y ago
      [deleted]
    - Houshalter 12y ago
      Nowhere was it stated the coin was 100% guaranteed to be fair. You are missing the point of my original example. My point was you can estimate the probability of a hypothesis by looking at the data. E.g. is it a fair or biased coin, and see how many times it came up heads or tails. Or are female named hurricanes more likely to kill than male ones or not?

aaron695 12y ago

This is totally and utterly wrong. This experiment has been found statistical true. One side has been found to go it's way NOT by chance. This is how peer review / science works. That said there are issues, this is not one of them. If you can read the article they also performed 6 other experiments where they for instance show people a weather map, tell them the name (M/F) and ask if they should evacuate. But I have no idea how they justify including years where male names weren't allocated.

cwyers 12y ago
That's... not what statistical significance means. All statistical significance means is that there's a less than five percent chance of getting that result at random, if the null hypothesis were true. Now, granted, most studies reported on find p-values well below the .05 threshold. But still, if you go out and count the number of studies published in a year, one percent of them is still a LOT of wrong studies. And remember that, due to publication bias, it's a biased sample of the population of studies. So for every 100 studies that are examining a hypothesis that is in reality false, you'll get 4-5 that get a statistically significant finding instead. And it's those few studies that end up getting published.
- aaron695 12y ago
  This is also not true. For starters many studies don't use a p value of .05. People seem to have read one xkcd comic and think now all studies are false. There are issues, especially were the people doing the studies want a particular outcome for financial gain. But if there are issues with this study talk about them. Were are the other 19 studies they have hidden? Where have they not followed proper statistical practice? How does the other 6 theoretical experiments on the same paper which some also have significance come into it?

quacker 12y ago

Surprisingly, if you assume a uniform distribution of deadliness independent of the names, you actually find that it's more likely that either 1 or 2 is true than that 3 is true Right, but if you are a statistician what you should care about is the "distance" of either 1 or 2 from 3. It would not be surprising that one group is deadlier than the other, but it should be surprising if female hurricanes are "quite a bit" deadlier than male hurricanes (or vice versa), assuming this uniform distribution that you mentioned.

coherentpony 12y ago
Would you be surprised if you won the lottery? Yes. I imagine you'd also be ecstatic. However, the likelihood of your numbers being chosen had equal probability (assuming the numbers are drawn uniformly at random) with any other set of numbers anybody else picked. If someone else had won the lottery instead of you, they would also be surprised. Female hurricanes being 'quite a bit deadlier' than male hurricanes may well be surprising but it has equal probability with the event that male hurricanes are 'quite a bit deadlier' than female hurricanes. What you're describing is a rare event. Rare events happen, just rarely.
- quacker 12y ago
  Female hurricanes being 'quite a bit deadlier' than male hurricanes may well be surprising but it has equal probability with the event that male hurricanes are 'quite a bit deadlier' than female hurricanes. I definitely agree with this. What you're describing is a rare event. Rare events happen, just rarely. I'll restate this hurricane problem. Suppose we're flipping a fair coin. We should expect to see roughly equal numbers of heads and tails. If I flip the same coin a thousand times and see 523 heads, that seems "reasonable". It's certainly possible to get 873 heads, but I might start to wonder if the coin is really fair not because it's very unlikely to get 873 heads with a fair coin (and it is very unlikely) but because it's more plausible that the coin is biased. That is, it's more likely that a coin that gets heads with some probability near 0.873 produces 873 heads in a thousand flips than it is for fair coin to produce 873 heads in a thousand flips. In the context of hurricanes, if we assign genders randomly we expect close to equal deadliness between male and female hurricanes. If there have been a thousand hurricanes, and female hurricanes cause 87.3% of all hurricane deaths, well... I might start wondering if giving hurricanes female names somehow makes a hurricane more deadly. Of course, this is all in the context of a uniform distribution of names on hurricanes.