3 ms·

>> "one should assume the trial was flawed." > Only to the extent that similar, pre-replication studies are typically flawed. I don't actually know what extent

by lutusp 13y ago

>> "one should assume the trial was flawed." > Only to the extent that similar, pre-replication studies are typically flawed. I don't actually know what extent that is. There's a logical error in this approach. If one study has some probability p to be systematically flawed, can an evaluation of that study's outcome be modified by the knowledge that ten other studies, each with the same probability of flaw, exist? Well, no, not really -- they must all be assumed to be flawed, regardless of how many studies exist. If this were not the case, we could cancel out the effect of the systematic flaws in each study by performing a lot of flawed studies. (When this idea is put in practice, often it results in the round-filing of all the studies that failed to meet the researchers' expectations.) Notice that I'm addressing the question of how we might evaluate the outcome of one study, given the existence of other similar studies. A meta-analysis of all the studies at once uses different rules.

dllthomas 13y ago
You seem to be attacking the notion of replication here, rather than anything I said. I was explicitly not talking about studies looking at the same question in the same way (we were talking pre-replication vs. post-replication). By "similar studies" I mean studies that have similar attributes, in terms of the information available to us. Random preprint on arXiv.org is going to be different than a study from a reputable researcher that's been peer reviewed. It's not going to be 100% in any event, of course, or even as likely as the purported p-value would suggest (since that necessarily can't take into account odds that the study methodology is flawed). That doesn't mean "assume it's flawed" is correct advice.
- lutusp 13y ago
  > You seem to be attacking the notion of replication here, rather than anything I said. No, I was only pointing out that our evaluation of a given study cannot depend on how many other similar studies exist. I'm not saying that doesn't happen -- it does -- I'm only saying it contradicts scientific best practice. > Random preprint on arXiv.org is going to be different than a study from a reputable researcher that's been peer reviewed. Not in science. Peer review and the imprimatur of a "reputable researcher" cannot grant extra authority to a study, because there is no authority in science -- to the degree that a reliance on authority is a logical error. > That doesn't mean "assume it's flawed" is correct advice. In science? Of course it's correct advice. If studies are flawed, and if we don't make the default scientific assumption that they're broken and meaningless, we may not exercise the effort required to sift the few properly conducted studies from the chaff.
  - dllthomas 13y ago
    "No, I was only pointing out that our evaluation of a given study cannot depend on how many other similar studies exist." 1) That's replication, and 2) that's not what I was suggesting. So I don't understand your "no" at all. To clarify a little: First, by "similar studies" I mean "studies drawn from the same population of studies", not "studies that are looking at the same thing in the same way". Second, I said that the study has a similar chance of being flawed as those other studies, not that more studies means less chance of being wrong (which would be true to that degree the chances of error are uncorrelated, but wasn't at all what I was saying). 'Not in science. Peer review and the imprimatur of a "reputable researcher" cannot grant extra authority to a study, because there is no authority in science -- to the degree that a reliance on authority is a logical error.' So wait... first, peer review is not a part of science? Second, a paper having passed peer review tells us nothing about whether it's flawed? Seriously? So if I pick two papers at random, one from "the set of papers that have not been peer reviewed" and one from "the set of papers that have been peer reviewed in a reputable journal and not found wanting", they are equally likely to be erroneous?
    - lutusp 13y ago
      >> 'Not in science. Peer review and the imprimatur of a "reputable researcher" cannot grant extra authority to a study, because there is no authority in science -- to the degree that a reliance on authority is a logical error.' > So wait... first, peer review is not a part of science? It's part of how science is practiced, but it is not part of science's fundamentals -- the latter honors only direct evidence -- see below. > Second, a paper having passed peer review tells us nothing about whether it's flawed? Well, since I never said that, I don't need to defend it. Instead, I will refer you to the motto of the Royal Society, science's oldest professional society: http://en.wikipedia.org/wiki/Nullius_in_verba http://en.wikipedia.org/wiki/Nullius_in_verba Quote: "Nullius in verba (Latin for "on the word of no one" or "Take nobody's word for it") is the motto of the Royal Society. John Evelyn and other Royal Society fellows chose the motto soon after the founding of the Society.[1] The current Royal Society website explains the motto thus: 'It is an expression of the determination of Fellows to withstand the domination of authority and to verify all statements by an appeal to facts determined by experiment.'[2]"

by lutusp 13y ago

dllthomas 13y ago

You seem to be attacking the notion of replication here, rather than anything I said. I was explicitly not talking about studies looking at the same question in the same way (we were talking pre-replication vs. post-replication). By "similar studies" I mean studies that have similar attributes, in terms of the information available to us. Random preprint on arXiv.org is going to be different than a study from a reputable researcher that's been peer reviewed. It's not going to be 100% in any event, of course, or even as likely as the purported p-value would suggest (since that necessarily can't take into account odds that the study methodology is flawed). That doesn't mean "assume it's flawed" is correct advice.

lutusp 13y ago
> You seem to be attacking the notion of replication here, rather than anything I said. No, I was only pointing out that our evaluation of a given study cannot depend on how many other similar studies exist. I'm not saying that doesn't happen -- it does -- I'm only saying it contradicts scientific best practice. > Random preprint on arXiv.org is going to be different than a study from a reputable researcher that's been peer reviewed. Not in science. Peer review and the imprimatur of a "reputable researcher" cannot grant extra authority to a study, because there is no authority in science -- to the degree that a reliance on authority is a logical error. > That doesn't mean "assume it's flawed" is correct advice. In science? Of course it's correct advice. If studies are flawed, and if we don't make the default scientific assumption that they're broken and meaningless, we may not exercise the effort required to sift the few properly conducted studies from the chaff.
- dllthomas 13y ago
  "No, I was only pointing out that our evaluation of a given study cannot depend on how many other similar studies exist." 1) That's replication, and 2) that's not what I was suggesting. So I don't understand your "no" at all. To clarify a little: First, by "similar studies" I mean "studies drawn from the same population of studies", not "studies that are looking at the same thing in the same way". Second, I said that the study has a similar chance of being flawed as those other studies, not that more studies means less chance of being wrong (which would be true to that degree the chances of error are uncorrelated, but wasn't at all what I was saying). 'Not in science. Peer review and the imprimatur of a "reputable researcher" cannot grant extra authority to a study, because there is no authority in science -- to the degree that a reliance on authority is a logical error.' So wait... first, peer review is not a part of science? Second, a paper having passed peer review tells us nothing about whether it's flawed? Seriously? So if I pick two papers at random, one from "the set of papers that have not been peer reviewed" and one from "the set of papers that have been peer reviewed in a reputable journal and not found wanting", they are equally likely to be erroneous?
  - lutusp 13y ago
    >> 'Not in science. Peer review and the imprimatur of a "reputable researcher" cannot grant extra authority to a study, because there is no authority in science -- to the degree that a reliance on authority is a logical error.' > So wait... first, peer review is not a part of science? It's part of how science is practiced, but it is not part of science's fundamentals -- the latter honors only direct evidence -- see below. > Second, a paper having passed peer review tells us nothing about whether it's flawed? Well, since I never said that, I don't need to defend it. Instead, I will refer you to the motto of the Royal Society, science's oldest professional society: http://en.wikipedia.org/wiki/Nullius_in_verba http://en.wikipedia.org/wiki/Nullius_in_verba Quote: "Nullius in verba (Latin for "on the word of no one" or "Take nobody's word for it") is the motto of the Royal Society. John Evelyn and other Royal Society fellows chose the motto soon after the founding of the Society.[1] The current Royal Society website explains the motto thus: 'It is an expression of the determination of Fellows to withstand the domination of authority and to verify all statements by an appeal to facts determined by experiment.'[2]"