3 ms·

Roberts is collected one long detailed case study. Sample size of 1, no controls, no blinding and tiny experiments. Ghost hunters and anti-vaxxers have better

Roberts is collected one long detailed case study. Sample size of 1, no controls, no blinding and tiny experiments. Ghost hunters and anti-vaxxers have better science behind their pet theories. We expect this type of endeavour to be extraordinarily prone to influence from noise. The entire modern process of science has been built to control for external factors that Roberts is ignoring or outright embracing. And no one should be surprised when he finds links between brain function and dental amalgam, which is both preposterous and specific to an existing hypothesis common among other pseudosciences. Isn't this just more evidence that the unblinded nature of Roberts anomaly hunting is being skewed by bias? That doesn't mean that some of the things he has found aren't true but 100+ years of trial and error and deep thought on this single issue by an enormous number of intelligent people taking the most meticulous records humanity has ever produced all around the world have determined that his methods will produce more false correlations than true ones. The failure to produce previous controlled experiments is alarming. I'm just baffled as to why you think that removing controls and oversight and experimenter blinding and lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better?

beagle3 14y ago
> Roberts is collected one long detailed case study. Sample size of 1, no controls, no blinding and tiny experiments. You might want to actually read what Roberts collects, rather than what you think he collects, because it is apparent that you extrapolate from this post. It's a good way to stroke your ego, but not a good way to have a discussion. > Isn't this just more evidence that the unblinded nature of Roberts anomaly hunting is being skewed by bias? No, it isn't, and if you read what he publishes rather than have a knee-jerk reaction to one short summary, you might be able to appreciate that. > have determined that his methods will produce more false correlations than true ones. That's actually not true, as far as I know, and I would appreciate if you specified exactly what you are referring to. (saying "everyone knows" is not a good enough answer) What I do know is that this meticulous record keeping produces bogus results at an alarming rate because of the things it does not consider, like "publish or perish" pressure and the "negative reuslts go in the drawer" effect. This xkcd describes the mechanism http://xkcd.com/882/ http://xkcd.com/882/ , and these extremely well written, peer-reviewed papers, show that most published results (which are golden according to the school of thought you seem to belong to) is actually wrong: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ http://www.reuters.com/article/2012/03/28/us-science-cancer-idUSBRE82R12P20120328 http://www.reuters.com/article/2012/03/28/us-science-cancer-... (points to a paywalled nature article). In case you are unable to get a copy of the nature paper: 88% of celebrated cancer results published in the last 10 years CAN NOT BE REPLICATED (that is, are most probably WRONG), despite following the "golden rules of science". > The failure to produce previous controlled experiments is alarming. What exactly are you referring to? > I'm just baffled as to why you think that removing controls and oversight and experimenter blinding and lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better? Because I actually live in the real world and not in some fantasy world. I don't think removing controls and oversight is a good idea. However, Roberts has no funding for these experiments because there's no way to make money off them. So he does what he finds interesting, in the best way he can, and lets others try to replicate -- and then collects their results as well. He has stumbled on a diet that seems to be more effective than any other out there (in real world terms, that of keeping weight off years later). It only works for 80% of the people, but for those it works wonders. If you can describe a plausible placebo mechanism for this result, please do - I would love to hear one (and why it seems to skip every other popular diet out there) I do believe that a cheap, safe, easy to do experiment that is not blinded is infinitely more useful than an expensive one that will never be carried out. The first is not perfect, but yields results that are easy to later work with, and easy to verify whether they are right or wrong by doing the same test again or other test that would pinpoint the effect. It is a weak result, but it is a result of some kind. The second yields zero information, since it will never be done. > lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better? You don't have to agree with what Roberts is doing. But you will have to excuse me for pointing out that making comments based on what you infer he is doing or claiming from a ten line summary makes you look like an idiot, even if it makes you feel superior.
- freshhawk 14y ago
  > That's actually not true, as far as I know, and I would appreciate if you specified exactly what you are referring to. (saying "everyone knows" is not a good enough answer) Let's take one, specific, example: Blinding. So you think that science added the concept of blinding and double blinded experiments because it didn't result in an improvement in results and a decrease in bias? And a sample size of more than 1? Are you saying there is no evidence that a sample size larger than one leads to better results? What about my statement that having, rather than not having, basic scientific controls in place results in more trustworthy evidence is "actually not true" > What I do know is that this meticulous record keeping produces bogus results at an alarming rate because of the things it does not consider, like "publish or perish" pressure and the "negative reuslts go in the drawer" effect. This xkcd describes the mechanism http://xkcd.com/882/ http://xkcd.com/882/ , and these extremely well written, peer-reviewed papers, show that most published results (which are golden according to the school of thought you seem to belong to) is actually wrong: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ http://www.reuters.com/article/2012/03/28/us-science-cancer-.. http://www.reuters.com/article/2012/03/28/us-science-cancer-.... (points to a paywalled nature article). In case you are unable to get a copy of the nature paper: 88% of celebrated cancer results published in the last 10 years CAN NOT BE REPLICATED (that is, are most probably WRONG), despite following the "golden rules of science". > Because I actually live in the real world and not in some fantasy world. I don't think removing controls and oversight is a good idea. However, Roberts has no funding for these experiments because there's no way to make money off them. So he does what he finds interesting, in the best way he can, and lets others try to replicate -- and then collects their results as well. Which side are you arguing? So science is hard and there are a ton of shitty results accepted as valid. But you seem to be suggesting that the solution to this is good intentions and a scrappy can-do attitude. If the well funded experiments with large sample sizes, double blinding and excellent controls still screw it up most of the time because of some bias or error in procedure then why would I expect removing all controls would result in something even worth looking at? PS Just to clear something up, in science unreplicated experiements are weak evidence, most are expected to be wrong. replicated experiments are stronger. and theories with multiple lines of evidence from multiple experiments in multiple fields and have emerged as the consensus opinion among experts of how things work are strong. The way you pointed out that unreplicated papers are probably wrong seemed like you thought you had some kind of gotcha. Well the "golden rules of science" are pretty clear on the value to be placed on unreplicated papers in a field overrun by media attention.

beagle3 14y ago

> Roberts is collected one long detailed case study. Sample size of 1, no controls, no blinding and tiny experiments. You might want to actually read what Roberts collects, rather than what you think he collects, because it is apparent that you extrapolate from this post. It's a good way to stroke your ego, but not a good way to have a discussion. > Isn't this just more evidence that the unblinded nature of Roberts anomaly hunting is being skewed by bias? No, it isn't, and if you read what he publishes rather than have a knee-jerk reaction to one short summary, you might be able to appreciate that. > have determined that his methods will produce more false correlations than true ones. That's actually not true, as far as I know, and I would appreciate if you specified exactly what you are referring to. (saying "everyone knows" is not a good enough answer) What I do know is that this meticulous record keeping produces bogus results at an alarming rate because of the things it does not consider, like "publish or perish" pressure and the "negative reuslts go in the drawer" effect. This xkcd describes the mechanism http://xkcd.com/882/ http://xkcd.com/882/ , and these extremely well written, peer-reviewed papers, show that most published results (which are golden according to the school of thought you seem to belong to) is actually wrong: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ http://www.reuters.com/article/2012/03/28/us-science-cancer-idUSBRE82R12P20120328 http://www.reuters.com/article/2012/03/28/us-science-cancer-... (points to a paywalled nature article). In case you are unable to get a copy of the nature paper: 88% of celebrated cancer results published in the last 10 years CAN NOT BE REPLICATED (that is, are most probably WRONG), despite following the "golden rules of science". > The failure to produce previous controlled experiments is alarming. What exactly are you referring to? > I'm just baffled as to why you think that removing controls and oversight and experimenter blinding and lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better? Because I actually live in the real world and not in some fantasy world. I don't think removing controls and oversight is a good idea. However, Roberts has no funding for these experiments because there's no way to make money off them. So he does what he finds interesting, in the best way he can, and lets others try to replicate -- and then collects their results as well. He has stumbled on a diet that seems to be more effective than any other out there (in real world terms, that of keeping weight off years later). It only works for 80% of the people, but for those it works wonders. If you can describe a plausible placebo mechanism for this result, please do - I would love to hear one (and why it seems to skip every other popular diet out there) I do believe that a cheap, safe, easy to do experiment that is not blinded is infinitely more useful than an expensive one that will never be carried out. The first is not perfect, but yields results that are easy to later work with, and easy to verify whether they are right or wrong by doing the same test again or other test that would pinpoint the effect. It is a weak result, but it is a result of some kind. The second yields zero information, since it will never be done. > lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better? You don't have to agree with what Roberts is doing. But you will have to excuse me for pointing out that making comments based on what you infer he is doing or claiming from a ten line summary makes you look like an idiot, even if it makes you feel superior.

freshhawk 14y ago
> That's actually not true, as far as I know, and I would appreciate if you specified exactly what you are referring to. (saying "everyone knows" is not a good enough answer) Let's take one, specific, example: Blinding. So you think that science added the concept of blinding and double blinded experiments because it didn't result in an improvement in results and a decrease in bias? And a sample size of more than 1? Are you saying there is no evidence that a sample size larger than one leads to better results? What about my statement that having, rather than not having, basic scientific controls in place results in more trustworthy evidence is "actually not true" > What I do know is that this meticulous record keeping produces bogus results at an alarming rate because of the things it does not consider, like "publish or perish" pressure and the "negative reuslts go in the drawer" effect. This xkcd describes the mechanism http://xkcd.com/882/ http://xkcd.com/882/ , and these extremely well written, peer-reviewed papers, show that most published results (which are golden according to the school of thought you seem to belong to) is actually wrong: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ http://www.reuters.com/article/2012/03/28/us-science-cancer-.. http://www.reuters.com/article/2012/03/28/us-science-cancer-.... (points to a paywalled nature article). In case you are unable to get a copy of the nature paper: 88% of celebrated cancer results published in the last 10 years CAN NOT BE REPLICATED (that is, are most probably WRONG), despite following the "golden rules of science". > Because I actually live in the real world and not in some fantasy world. I don't think removing controls and oversight is a good idea. However, Roberts has no funding for these experiments because there's no way to make money off them. So he does what he finds interesting, in the best way he can, and lets others try to replicate -- and then collects their results as well. Which side are you arguing? So science is hard and there are a ton of shitty results accepted as valid. But you seem to be suggesting that the solution to this is good intentions and a scrappy can-do attitude. If the well funded experiments with large sample sizes, double blinding and excellent controls still screw it up most of the time because of some bias or error in procedure then why would I expect removing all controls would result in something even worth looking at? PS Just to clear something up, in science unreplicated experiements are weak evidence, most are expected to be wrong. replicated experiments are stronger. and theories with multiple lines of evidence from multiple experiments in multiple fields and have emerged as the consensus opinion among experts of how things work are strong. The way you pointed out that unreplicated papers are probably wrong seemed like you thought you had some kind of gotcha. Well the "golden rules of science" are pretty clear on the value to be placed on unreplicated papers in a field overrun by media attention.