10 ms·

This. If it's not a double-blind random crossover trial, it probably means nada. Thus, most nutritional, health, and economics 'science' is trash. Unfortunat

This. If it's not a double-blind random crossover trial, it probably means nada. Thus, most nutritional, health, and economics 'science' is trash. Unfortunately, it also means it's very hard to prove things are safe or unsafe. Phracking, pesticides, GMO, ... all get a free ride for too long, IMHO, because they can say 'well nobody has proved they're unsafe'.

sp332 13y ago
It's not trash. A single-blind trial is much better than nothing. Any variables you can remove will improve the probability of the result being right.
- gnaritas 13y ago
  Not much better than nothing; results are useful after others have replicated them, until then one should assume the trial was flawed.
  - dllthomas 13y ago
    "one should assume the trial was flawed." Only to the extent that similar, pre-replication studies are typically flawed. I don't actually know what extent that is.
    - gnaritas 13y ago
      Agreed.
    - lutusp 13y ago
      >> "one should assume the trial was flawed." > Only to the extent that similar, pre-replication studies are typically flawed. I don't actually know what extent that is. There's a logical error in this approach. If one study has some probability p to be systematically flawed, can an evaluation of that study's outcome be modified by the knowledge that ten other studies, each with the same probability of flaw, exist? Well, no, not really -- they must all be assumed to be flawed, regardless of how many studies exist. If this were not the case, we could cancel out the effect of the systematic flaws in each study by performing a lot of flawed studies. (When this idea is put in practice, often it results in the round-filing of all the studies that failed to meet the researchers' expectations.) Notice that I'm addressing the question of how we might evaluate the outcome of one study, given the existence of other similar studies. A meta-analysis of all the studies at once uses different rules.
      - dllthomas 13y ago
        You seem to be attacking the notion of replication here, rather than anything I said. I was explicitly not talking about studies looking at the same question in the same way (we were talking pre-replication vs. post-replication). By "similar studies" I mean studies that have similar attributes, in terms of the information available to us. Random preprint on arXiv.org is going to be different than a study from a reputable researcher that's been peer reviewed. It's not going to be 100% in any event, of course, or even as likely as the purported p-value would suggest (since that necessarily can't take into account odds that the study methodology is flawed). That doesn't mean "assume it's flawed" is correct advice.
        lutusp 13y ago
        > You seem to be attacking the notion of replication here, rather than anything I said. No, I was only pointing out that our evaluation of a given study cannot depend on how many other similar studies exist. I'm not saying that doesn't happen -- it does -- I'm only saying it contradicts scientific best practice. > Random preprint on arXiv.org is going to be different than a study from a reputable researcher that's been peer reviewed. Not in science. Peer review and the imprimatur of a "reputable researcher" cannot grant extra authority to a study, because there is no authority in science -- to the degree that a reliance on authority is a logical error. > That doesn't mean "assume it's flawed" is correct advice. In science? Of course it's correct advice. If studies are flawed, and if we don't make the default scientific assumption that they're broken and meaningless, we may not exercise the effort required to sift the few properly conducted studies from the chaff.
        dllthomas 13y ago
        "No, I was only pointing out that our evaluation of a given study cannot depend on how many other similar studies exist." 1) That's replication, and 2) that's not what I was suggesting. So I don't understand your "no" at all. To clarify a little: First, by "similar studies" I mean "studies drawn from the same population of studies", not "studies that are looking at the same thing in the same way". Second, I said that the study has a similar chance of being flawed as those other studies, not that more studies means less chance of being wrong (which would be true to that degree the chances of error are uncorrelated, but wasn't at all what I was saying). 'Not in science. Peer review and the imprimatur of a "reputable researcher" cannot grant extra authority to a study, because there is no authority in science -- to the degree that a reliance on authority is a logical error.' So wait... first, peer review is not a part of science? Second, a paper having passed peer review tells us nothing about whether it's flawed? Seriously? So if I pick two papers at random, one from "the set of papers that have not been peer reviewed" and one from "the set of papers that have been peer reviewed in a reputable journal and not found wanting", they are equally likely to be erroneous?
- pizza234 13y ago
  It depends on the context. In the context of fields where junk studies are the norm, then it is trash, and actually worse than nothing, because it's a symptom of a widespread misguidance of the scientific approach. Homeopathy has probably plenty of such studies, and you know... it's homeopathy.
- lutusp 13y ago
  > It's not trash. A single-blind trial is much better than nothing. Let's test this idea with a thought experiment. Let's say I'm a doctor and I've created a revolutionary cure for the common cold. My cure is to shake a dried gourd over the cold sufferer until he gets better. The cure might take a week, but it always works. My method is repeatable and perfectly reliable, and I've published my cure in a refereed scientific journal (there are now any number of phony refereed scientific journals). And, because (in this thought experiment) science can get along without defining theories, I'm under no obligation to try to explain my cure, or consider alternative explanations for my breakthrough — I only have to describe it. Because I've cured the common cold, and because I've met all the requirements that non-scientists recognize for science, I deserve a Nobel Prize. Yes or no? The answer is obviously no, and to establish the reliability of dried gourds as a cure for the common cold, we would need a double-blind experiment -- both the experimenter and the subject would need to be blinded about which treatment is being applied.
  - dllthomas 13y ago
    Single blind doesn't mean there is no control group!
    - lutusp 13y ago
      Never claimed otherwise. My example was intentionally as simple as possible. Also, absent a control group, one can have control experiments interleaved with research experiments -- this is a common practice where resources are limited.
      - dllthomas 13y ago
        "Never claimed otherwise. My example was intentionally as simple as possible." This is disingenuous. The primary reason your example was flawed is the lack of a control group, and your phrasing in establishing that it is flawed more or less points that out. Using that to condemn single-blind studies is inane - if it were double blind but there was no control, it would give the same result.
dalke 13y ago
The problem is in what to do about the "probably". As one paper humorously pointed out, there's no double-blind random crossover trial to see if parachutes are effective. More seriously, astronomy deals with a bunch of singular events. How do we do double-blind random crossover trials of a comet crashing into Jupiter (like Shoemaker–Levy 9)? Closer to home, and closer to the point, detecting rare but deadly drug side-effect (and especially drug combination effects) cannot be done blind, since the drugs are on the market and people know what they are taking.
- Fomite 13y ago
  Nor has there been, to my knowledge, a double-blind study of smoking and lung cancer.
- gizmo686 13y ago
  From the paper [1]: "As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute." [1] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/?utm_source=dlvr.it&utm_medium=twitter http://www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/?utm_sour...
abrichr 13y ago
Phracking: a portmanteau of phreaking (phone system manipulation) and hacking (computer system manipulation). Fracking: short for "hydraulic fracturing", the fracturing of rock via pressurized liquid.
DenisM 13y ago
>GMO, ... all get a free ride for too long, What do you base that opinion on? Have you actually looked for any conclusive studies on GMO, and found them wanting?
- penguindev 13y ago
  I base it off of the simple doctrine of first do no harm. If you view food as being equal to a medicine/drug in power, I think the hippocratic oath applies. There's also a curious study comparing ancient wheat with modern wheat (I realize that may be not technically GMO) in, you guessed it, a blind crossover trial. http://www.ncbi.nlm.nih.gov/pubmed/24521561?dopt=Citation http://www.ncbi.nlm.nih.gov/pubmed/24521561?dopt=Citation
  - barry-cotter 13y ago
    I too support not eating anything that's been domesticated. Probably best not to cook food either, that's unnatural.
  - DenisM 13y ago
    I'm just curious as to why you chose to ignore hundreds of GMO safety studies that were already published. I mean, it seems that you didn't even look for them, instead making up your mind that there aren't enough of them.
    - penguindev 13y ago
      How about you point me to a study that took a hundred years (to look at effect on future generations), controlled all other variables, had no dropouts, and didn't rely on self reported information. No? Ok, well when you're playing god, and I assume you're probably paid to work in this industry, just don't tell me that it's "safe", because that's damn hard (impossible?) and expensive to prove [because technically you'd have to do this on every new strain of every food], and that would conflict with short-term profits. PS GMO is also closely related to pesticide & herbicide use and biodiversity. How do you even begin to quantify those long term effects.

sp332 13y ago

It's not trash. A single-blind trial is much better than nothing. Any variables you can remove will improve the probability of the result being right.

gnaritas 13y ago
Not much better than nothing; results are useful after others have replicated them, until then one should assume the trial was flawed.
- dllthomas 13y ago
  "one should assume the trial was flawed." Only to the extent that similar, pre-replication studies are typically flawed. I don't actually know what extent that is.
  - gnaritas 13y ago
    Agreed.
  - lutusp 13y ago
    >> "one should assume the trial was flawed." > Only to the extent that similar, pre-replication studies are typically flawed. I don't actually know what extent that is. There's a logical error in this approach. If one study has some probability p to be systematically flawed, can an evaluation of that study's outcome be modified by the knowledge that ten other studies, each with the same probability of flaw, exist? Well, no, not really -- they must all be assumed to be flawed, regardless of how many studies exist. If this were not the case, we could cancel out the effect of the systematic flaws in each study by performing a lot of flawed studies. (When this idea is put in practice, often it results in the round-filing of all the studies that failed to meet the researchers' expectations.) Notice that I'm addressing the question of how we might evaluate the outcome of one study, given the existence of other similar studies. A meta-analysis of all the studies at once uses different rules.
    - dllthomas 13y ago
      You seem to be attacking the notion of replication here, rather than anything I said. I was explicitly not talking about studies looking at the same question in the same way (we were talking pre-replication vs. post-replication). By "similar studies" I mean studies that have similar attributes, in terms of the information available to us. Random preprint on arXiv.org is going to be different than a study from a reputable researcher that's been peer reviewed. It's not going to be 100% in any event, of course, or even as likely as the purported p-value would suggest (since that necessarily can't take into account odds that the study methodology is flawed). That doesn't mean "assume it's flawed" is correct advice.
      - lutusp 13y ago
        > You seem to be attacking the notion of replication here, rather than anything I said. No, I was only pointing out that our evaluation of a given study cannot depend on how many other similar studies exist. I'm not saying that doesn't happen -- it does -- I'm only saying it contradicts scientific best practice. > Random preprint on arXiv.org is going to be different than a study from a reputable researcher that's been peer reviewed. Not in science. Peer review and the imprimatur of a "reputable researcher" cannot grant extra authority to a study, because there is no authority in science -- to the degree that a reliance on authority is a logical error. > That doesn't mean "assume it's flawed" is correct advice. In science? Of course it's correct advice. If studies are flawed, and if we don't make the default scientific assumption that they're broken and meaningless, we may not exercise the effort required to sift the few properly conducted studies from the chaff.
        dllthomas 13y ago
        "No, I was only pointing out that our evaluation of a given study cannot depend on how many other similar studies exist." 1) That's replication, and 2) that's not what I was suggesting. So I don't understand your "no" at all. To clarify a little: First, by "similar studies" I mean "studies drawn from the same population of studies", not "studies that are looking at the same thing in the same way". Second, I said that the study has a similar chance of being flawed as those other studies, not that more studies means less chance of being wrong (which would be true to that degree the chances of error are uncorrelated, but wasn't at all what I was saying). 'Not in science. Peer review and the imprimatur of a "reputable researcher" cannot grant extra authority to a study, because there is no authority in science -- to the degree that a reliance on authority is a logical error.' So wait... first, peer review is not a part of science? Second, a paper having passed peer review tells us nothing about whether it's flawed? Seriously? So if I pick two papers at random, one from "the set of papers that have not been peer reviewed" and one from "the set of papers that have been peer reviewed in a reputable journal and not found wanting", they are equally likely to be erroneous?
pizza234 13y ago
It depends on the context. In the context of fields where junk studies are the norm, then it is trash, and actually worse than nothing, because it's a symptom of a widespread misguidance of the scientific approach. Homeopathy has probably plenty of such studies, and you know... it's homeopathy.
lutusp 13y ago
> It's not trash. A single-blind trial is much better than nothing. Let's test this idea with a thought experiment. Let's say I'm a doctor and I've created a revolutionary cure for the common cold. My cure is to shake a dried gourd over the cold sufferer until he gets better. The cure might take a week, but it always works. My method is repeatable and perfectly reliable, and I've published my cure in a refereed scientific journal (there are now any number of phony refereed scientific journals). And, because (in this thought experiment) science can get along without defining theories, I'm under no obligation to try to explain my cure, or consider alternative explanations for my breakthrough — I only have to describe it. Because I've cured the common cold, and because I've met all the requirements that non-scientists recognize for science, I deserve a Nobel Prize. Yes or no? The answer is obviously no, and to establish the reliability of dried gourds as a cure for the common cold, we would need a double-blind experiment -- both the experimenter and the subject would need to be blinded about which treatment is being applied.
- dllthomas 13y ago
  Single blind doesn't mean there is no control group!
  - lutusp 13y ago
    Never claimed otherwise. My example was intentionally as simple as possible. Also, absent a control group, one can have control experiments interleaved with research experiments -- this is a common practice where resources are limited.
    - dllthomas 13y ago
      "Never claimed otherwise. My example was intentionally as simple as possible." This is disingenuous. The primary reason your example was flawed is the lack of a control group, and your phrasing in establishing that it is flawed more or less points that out. Using that to condemn single-blind studies is inane - if it were double blind but there was no control, it would give the same result.

dalke 13y ago

The problem is in what to do about the "probably". As one paper humorously pointed out, there's no double-blind random crossover trial to see if parachutes are effective. More seriously, astronomy deals with a bunch of singular events. How do we do double-blind random crossover trials of a comet crashing into Jupiter (like Shoemaker–Levy 9)? Closer to home, and closer to the point, detecting rare but deadly drug side-effect (and especially drug combination effects) cannot be done blind, since the drugs are on the market and people know what they are taking.

Fomite 13y ago
Nor has there been, to my knowledge, a double-blind study of smoking and lung cancer.
gizmo686 13y ago
From the paper [1]: "As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute." [1] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/?utm_source=dlvr.it&utm_medium=twitter http://www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/?utm_sour...

abrichr 13y ago

Phracking: a portmanteau of phreaking (phone system manipulation) and hacking (computer system manipulation). Fracking: short for "hydraulic fracturing", the fracturing of rock via pressurized liquid.

DenisM 13y ago

>GMO, ... all get a free ride for too long, What do you base that opinion on? Have you actually looked for any conclusive studies on GMO, and found them wanting?

penguindev 13y ago
I base it off of the simple doctrine of first do no harm. If you view food as being equal to a medicine/drug in power, I think the hippocratic oath applies. There's also a curious study comparing ancient wheat with modern wheat (I realize that may be not technically GMO) in, you guessed it, a blind crossover trial. http://www.ncbi.nlm.nih.gov/pubmed/24521561?dopt=Citation http://www.ncbi.nlm.nih.gov/pubmed/24521561?dopt=Citation
- barry-cotter 13y ago
  I too support not eating anything that's been domesticated. Probably best not to cook food either, that's unnatural.
- DenisM 13y ago
  I'm just curious as to why you chose to ignore hundreds of GMO safety studies that were already published. I mean, it seems that you didn't even look for them, instead making up your mind that there aren't enough of them.
  - penguindev 13y ago
    How about you point me to a study that took a hundred years (to look at effect on future generations), controlled all other variables, had no dropouts, and didn't rely on self reported information. No? Ok, well when you're playing god, and I assume you're probably paid to work in this industry, just don't tell me that it's "safe", because that's damn hard (impossible?) and expensive to prove [because technically you'd have to do this on every new strain of every food], and that would conflict with short-term profits. PS GMO is also closely related to pesticide & herbicide use and biodiversity. How do you even begin to quantify those long term effects.