3 ms·
It's not trash. A single-blind trial is much better than nothing. Any variables you can remove will improve the probability of the result being right.
by sp332 13y ago
It's not trash. A single-blind trial is much better than nothing. Any variables you can remove will improve the probability of the result being right.
- gnaritas 13y agoNot much better than nothing; results are useful after others have replicated them, until then one should assume the trial was flawed.
- dllthomas 13y ago"one should assume the trial was flawed." Only to the extent that similar, pre-replication studies are typically flawed. I don't actually know what extent that is.
- gnaritas 13y agoAgreed.
- lutusp 13y ago>> "one should assume the trial was flawed." > Only to the extent that similar, pre-replication studies are typically flawed. I don't actually know what extent that is. There's a logical error in this approach. If one study has some probability p to be systematically flawed, can an evaluation of that study's outcome be modified by the knowledge that ten other studies, each with the same probability of flaw, exist? Well, no, not really -- they must all be assumed to be flawed, regardless of how many studies exist. If this were not the case, we could cancel out the effect of the systematic flaws in each study by performing a lot of flawed studies. (When this idea is put in practice, often it results in the round-filing of all the studies that failed to meet the researchers' expectations.) Notice that I'm addressing the question of how we might evaluate the outcome of one study, given the existence of other similar studies. A meta-analysis of all the studies at once uses different rules.
- dllthomas 13y agoYou seem to be attacking the notion of replication here, rather than anything I said. I was explicitly not talking about studies looking at the same question in the same way (we were talking pre-replication vs. post-replication). By "similar studies" I mean studies that have similar attributes, in terms of the information available to us. Random preprint on arXiv.org is going to be different than a study from a reputable researcher that's been peer reviewed. It's not going to be 100% in any event, of course, or even as likely as the purported p-value would suggest (since that necessarily can't take into account odds that the study methodology is flawed). That doesn't mean "assume it's flawed" is correct advice.
- lutusp 13y ago> You seem to be attacking the notion of replication here, rather than anything I said. No, I was only pointing out that our evaluation of a given study cannot depend on how many other similar studies exist. I'm not saying that doesn't happen -- it does -- I'm only saying it contradicts scientific best practice. > Random preprint on arXiv.org is going to be different than a study from a reputable researcher that's been peer reviewed. Not in science. Peer review and the imprimatur of a "reputable researcher" cannot grant extra authority to a study, because there is no authority in science -- to the degree that a reliance on authority is a logical error. > That doesn't mean "assume it's flawed" is correct advice. In science? Of course it's correct advice. If studies are flawed, and if we don't make the default scientific assumption that they're broken and meaningless, we may not exercise the effort required to sift the few properly conducted studies from the chaff.
- dllthomas 13y ago"No, I was only pointing out that our evaluation of a given study cannot depend on how many other similar studies exist." 1) That's replication, and 2) that's not what I was suggesting. So I don't understand your "no" at all. To clarify a little: First, by "similar studies" I mean "studies drawn from the same population of studies", not "studies that are looking at the same thing in the same way". Second, I said that the study has a similar chance of being flawed as those other studies, not that more studies means less chance of being wrong (which would be true to that degree the chances of error are uncorrelated, but wasn't at all what I was saying). 'Not in science. Peer review and the imprimatur of a "reputable researcher" cannot grant extra authority to a study, because there is no authority in science -- to the degree that a reliance on authority is a logical error.' So wait... first, peer review is not a part of science? Second, a paper having passed peer review tells us nothing about whether it's flawed? Seriously? So if I pick two papers at random, one from "the set of papers that have not been peer reviewed" and one from "the set of papers that have been peer reviewed in a reputable journal and not found wanting", they are equally likely to be erroneous?
- pizza234 13y agoIt depends on the context. In the context of fields where junk studies are the norm, then it is trash, and actually worse than nothing, because it's a symptom of a widespread misguidance of the scientific approach. Homeopathy has probably plenty of such studies, and you know... it's homeopathy.
- lutusp 13y ago> It's not trash. A single-blind trial is much better than nothing. Let's test this idea with a thought experiment. Let's say I'm a doctor and I've created a revolutionary cure for the common cold. My cure is to shake a dried gourd over the cold sufferer until he gets better. The cure might take a week, but it always works. My method is repeatable and perfectly reliable, and I've published my cure in a refereed scientific journal (there are now any number of phony refereed scientific journals). And, because (in this thought experiment) science can get along without defining theories, I'm under no obligation to try to explain my cure, or consider alternative explanations for my breakthrough — I only have to describe it. Because I've cured the common cold, and because I've met all the requirements that non-scientists recognize for science, I deserve a Nobel Prize. Yes or no? The answer is obviously no, and to establish the reliability of dried gourds as a cure for the common cold, we would need a double-blind experiment -- both the experimenter and the subject would need to be blinded about which treatment is being applied.
- dllthomas 13y agoSingle blind doesn't mean there is no control group!
- lutusp 13y agoNever claimed otherwise. My example was intentionally as simple as possible. Also, absent a control group, one can have control experiments interleaved with research experiments -- this is a common practice where resources are limited.
- dllthomas 13y ago"Never claimed otherwise. My example was intentionally as simple as possible." This is disingenuous. The primary reason your example was flawed is the lack of a control group, and your phrasing in establishing that it is flawed more or less points that out. Using that to condemn single-blind studies is inane - if it were double blind but there was no control, it would give the same result.