4 ms·

I very much doubt that this is true. Raw data by itself is almost never definitive. Meaningful conclusions often require that data be manipulated in fairly extr

I very much doubt that this is true. Raw data by itself is almost never definitive. Meaningful conclusions often require that data be manipulated in fairly extreme ways. Each one of those manipulations provides fertile ground for denialist critique. To give you one example with which I'm familiar, several research groups have attempted to measure the number of fatalities associated with the Iraq War. One of the standard techniques for doing so involves cluster sampling, i.e., picking a small number of locations and conducting multiple interviews near each cluster site. One research group doing this noticed that the mortality rate from their Fallujah cluster was much much much higher than any of the other clusters. So they purposefully excluded it from their analysis although they kept it as part of their raw data and explained why they removed it. As a result, their total death toll was much lower than it would have been. Denialists immediately criticized them for manipulating data so as to inflate the death toll. How did they do that? Very simply: if you omit the outlier, then the spread of the probability distribution shrinks dramatically. If you include the outlier, then the width of the distribution expands so far out that it encompasses zero excess deaths. Now in either case, the distribution was screaming that there were many excess deaths, but these denialists claimed that scientists were reducing the average fatality count just so they could exclude the minute possibility that the war decreased death rates. Obviously, there is no way these scientists could win: if they include the outlier, then they're deliberately inflating the death toll, but if they exclude the outlier, then they're manipulating data to obscure the possibility that the war saved lives. If the scientists had just quietly dropped their Falujah data, this controversy never would have happened. In this case, releasing data just gave denialists fodor for crazy attacks.

MichaelSalib 16y ago
Downvoters: I'm curious, what is wrong with this comment?
- keefe 16y ago
  I often wonder this myself, something seems to make people more inclined to disagree than agree.
  - ars 16y ago
    It's not people - it's person. There are a lot more readers than modders.
- ars 16y ago
  I wouldn't downmod you for that, but man I don't agree with that comment. They are fitting data to expectations, and worse you are telling them to conceal the fact that they do so, just so no one can argue with them. That is pretty much as far away as you can possibly get from true science. What they should do is put massive error bars on the numbers and say "We can't." We can't tell you the true number, it's not possible to calculate it.
  - MichaelSalib 16y ago
    They are fitting data to expectations Eh? This is a war zone that has seen very intense fighting. It is not unreasonable to assume that data from this one site is unreliable for any of a million different reasons, especially given just how far it was from the other sites. Do you generally insist that scientists must never eliminate any outlier data points, no matter how suspect? you are telling them to conceal the fact that they do so I'm not telling them to conceal anything. I think they did the correct thing: they published all their data, and they disclosed exactly what data points they removed and why they did so. My point is that because they published their data and disclosed what they did and why they did it, they faced criticisms that never would have seen the light of day otherwise. And to me that suggests that it is unreasonable to believe that deniers are going to become less prevalent in a world where more data is published openly. What they should do is put massive error bars on the numbers and say "We can't." We can't tell you the true number, it's not possible to calculate it. I don't understand this at all. "Massive" doesn't mean anything. They calculated appropriate confidence intervals and included them in their analysis. What exactly do you think they did wrong?
    - keefe 16y ago
      >>unreasonable to believe that deniers are going to become less prevalent in a world where more data is published this is almost a tautology. if data isn't published, there's nothing to deny. are you suggesting... everybody just shut the data up behind closed doors so people can say whatever they want and at least none of those pesky dissenters can stir up trouble? If the data is out there, more people will see the truth. You really can't argue with #s - if I include studies X,Y,Z the mean is M and if I exclude Z it's N. You may argue about Z but people should make those decisions for themselves.
keefe 16y ago
I think you underestimate people a little bit. Let's take your example in a world of data accessibility : Research groups do spot interviews for fatalities in the iraq war. This data is submitted to the central repository. The government analyzes it, removes outliers emits report A. The ACLU analyzes it with the outliers, emits report B. Some people throw a fit about A, some people throw a fit about B. If the data were available, independent third parties could analyze it and say - hey ok, Fallujah was actually a valid data point so we present report B or whatever. You can't argue with data, not and maintain the appearance of sanity. Here's the data. Here's the interpretation. Even laypeople can understand this.
- MichaelSalib 16y ago
  I think my example may have been unclear, so let me give an analogy I picked up from a demographer who analyzed the study in question: "Imagine that if you took 32 water samples in your child's school and found that arsenic samples were 60% higher than the national standards, and you were 98% sure that you were above the standard. Then you took one more sample that was ten times higher than the standard. The logic endorsed by Michael Fumento is that we do nothing about the water supply because this outlier broadened our statistical confidence interval and we are not sure that we are above the national standard." If the data were available, independent third parties could analyze it and say - hey ok, Fallujah was actually a valid data point so we present report B or whatever. So, in the example above, do you really think that anyone claiming "oh, there's no reason to believe the school's water supply is dangerous since the confidence interval has been pushed out far enough that there is a minute possibility that the water is safe" is acting in good faith? I am curious: do you believe that any professional can make such claims in good faith and in accordance with their professional obligations? If the data were available, independent third parties could analyze it and say - hey ok, Fallujah was actually a valid data point so we present report B or whatever.
  - keefe 16y ago
    >oh, there's no reason to believe the school's water supply is dangerous This is obviously false. I understand the idea of the confidence interval, I'd say the interpretation is : credible evidence for arsenic levels above standards, high variance in data requires further testing. Here is how I am saying it should work : All 33 samples are in a public database D. The R script that generates the statistical analysis is in that same database. Company C says because of results from R based on D I conclude X. Similarly for other companies. The bottom line is if the data is public and competent, unbiased observers exist (many of them do exist) then it becomes much harder to lie with stats.

MichaelSalib 16y ago

Downvoters: I'm curious, what is wrong with this comment?

keefe 16y ago
I often wonder this myself, something seems to make people more inclined to disagree than agree.
- ars 16y ago
  It's not people - it's person. There are a lot more readers than modders.
ars 16y ago
I wouldn't downmod you for that, but man I don't agree with that comment. They are fitting data to expectations, and worse you are telling them to conceal the fact that they do so, just so no one can argue with them. That is pretty much as far away as you can possibly get from true science. What they should do is put massive error bars on the numbers and say "We can't." We can't tell you the true number, it's not possible to calculate it.
- MichaelSalib 16y ago
  They are fitting data to expectations Eh? This is a war zone that has seen very intense fighting. It is not unreasonable to assume that data from this one site is unreliable for any of a million different reasons, especially given just how far it was from the other sites. Do you generally insist that scientists must never eliminate any outlier data points, no matter how suspect? you are telling them to conceal the fact that they do so I'm not telling them to conceal anything. I think they did the correct thing: they published all their data, and they disclosed exactly what data points they removed and why they did so. My point is that because they published their data and disclosed what they did and why they did it, they faced criticisms that never would have seen the light of day otherwise. And to me that suggests that it is unreasonable to believe that deniers are going to become less prevalent in a world where more data is published openly. What they should do is put massive error bars on the numbers and say "We can't." We can't tell you the true number, it's not possible to calculate it. I don't understand this at all. "Massive" doesn't mean anything. They calculated appropriate confidence intervals and included them in their analysis. What exactly do you think they did wrong?
  - keefe 16y ago
    >>unreasonable to believe that deniers are going to become less prevalent in a world where more data is published this is almost a tautology. if data isn't published, there's nothing to deny. are you suggesting... everybody just shut the data up behind closed doors so people can say whatever they want and at least none of those pesky dissenters can stir up trouble? If the data is out there, more people will see the truth. You really can't argue with #s - if I include studies X,Y,Z the mean is M and if I exclude Z it's N. You may argue about Z but people should make those decisions for themselves.

keefe 16y ago

I think you underestimate people a little bit. Let's take your example in a world of data accessibility : Research groups do spot interviews for fatalities in the iraq war. This data is submitted to the central repository. The government analyzes it, removes outliers emits report A. The ACLU analyzes it with the outliers, emits report B. Some people throw a fit about A, some people throw a fit about B. If the data were available, independent third parties could analyze it and say - hey ok, Fallujah was actually a valid data point so we present report B or whatever. You can't argue with data, not and maintain the appearance of sanity. Here's the data. Here's the interpretation. Even laypeople can understand this.

MichaelSalib 16y ago
I think my example may have been unclear, so let me give an analogy I picked up from a demographer who analyzed the study in question: "Imagine that if you took 32 water samples in your child's school and found that arsenic samples were 60% higher than the national standards, and you were 98% sure that you were above the standard. Then you took one more sample that was ten times higher than the standard. The logic endorsed by Michael Fumento is that we do nothing about the water supply because this outlier broadened our statistical confidence interval and we are not sure that we are above the national standard." If the data were available, independent third parties could analyze it and say - hey ok, Fallujah was actually a valid data point so we present report B or whatever. So, in the example above, do you really think that anyone claiming "oh, there's no reason to believe the school's water supply is dangerous since the confidence interval has been pushed out far enough that there is a minute possibility that the water is safe" is acting in good faith? I am curious: do you believe that any professional can make such claims in good faith and in accordance with their professional obligations? If the data were available, independent third parties could analyze it and say - hey ok, Fallujah was actually a valid data point so we present report B or whatever.
- keefe 16y ago
  >oh, there's no reason to believe the school's water supply is dangerous This is obviously false. I understand the idea of the confidence interval, I'd say the interpretation is : credible evidence for arsenic levels above standards, high variance in data requires further testing. Here is how I am saying it should work : All 33 samples are in a public database D. The R script that generates the statistical analysis is in that same database. Company C says because of results from R based on D I conclude X. Similarly for other companies. The bottom line is if the data is public and competent, unbiased observers exist (many of them do exist) then it becomes much harder to lie with stats.