4 ms·

This is what the first part of science is. He's getting as far as one piece of data that might suggest a hypothesis worth testing, although you'd probably want

This is what the first part of science is. He's getting as far as one piece of data that might suggest a hypothesis worth testing, although you'd probably want more than one piece of preliminary data or you'd be hunting shadows all the time. He then stops before getting to that whole "controlled experiment" part that really makes up the central part of doing science.

beagle3 14y ago
> "controlled experiment" part that really makes up the central part of doing science. In theory, that's how it is supposed to work. But in practice, it doesn't. See e.g. John Ionaddis work showing that a significant part of published research (including that "controlled experiment" setting you are thinking about) is wrong; or the recent nature paper about failing to reproduce 47 out of 53 celebrated ("controlled experiment") results from the last 10 years, and what some more research about those results came up with. Roberts is not doing conventional research. But what he does do is (a) cheap, (b) safe, (c) yields results quickly, and (d) individual testable. He also publishes negative results, and admits mistakes. I think it is actually more useful than 90% of "proper science" done out there. He is not under a "publish or perish" whereas most scientists in the world are, and it shows.
- freshhawk 14y ago
  Roberts is collected one long detailed case study. Sample size of 1, no controls, no blinding and tiny experiments. Ghost hunters and anti-vaxxers have better science behind their pet theories. We expect this type of endeavour to be extraordinarily prone to influence from noise. The entire modern process of science has been built to control for external factors that Roberts is ignoring or outright embracing. And no one should be surprised when he finds links between brain function and dental amalgam, which is both preposterous and specific to an existing hypothesis common among other pseudosciences. Isn't this just more evidence that the unblinded nature of Roberts anomaly hunting is being skewed by bias? That doesn't mean that some of the things he has found aren't true but 100+ years of trial and error and deep thought on this single issue by an enormous number of intelligent people taking the most meticulous records humanity has ever produced all around the world have determined that his methods will produce more false correlations than true ones. The failure to produce previous controlled experiments is alarming. I'm just baffled as to why you think that removing controls and oversight and experimenter blinding and lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better?
  - beagle3 14y ago
    > Roberts is collected one long detailed case study. Sample size of 1, no controls, no blinding and tiny experiments. You might want to actually read what Roberts collects, rather than what you think he collects, because it is apparent that you extrapolate from this post. It's a good way to stroke your ego, but not a good way to have a discussion. > Isn't this just more evidence that the unblinded nature of Roberts anomaly hunting is being skewed by bias? No, it isn't, and if you read what he publishes rather than have a knee-jerk reaction to one short summary, you might be able to appreciate that. > have determined that his methods will produce more false correlations than true ones. That's actually not true, as far as I know, and I would appreciate if you specified exactly what you are referring to. (saying "everyone knows" is not a good enough answer) What I do know is that this meticulous record keeping produces bogus results at an alarming rate because of the things it does not consider, like "publish or perish" pressure and the "negative reuslts go in the drawer" effect. This xkcd describes the mechanism http://xkcd.com/882/ http://xkcd.com/882/ , and these extremely well written, peer-reviewed papers, show that most published results (which are golden according to the school of thought you seem to belong to) is actually wrong: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ http://www.reuters.com/article/2012/03/28/us-science-cancer-idUSBRE82R12P20120328 http://www.reuters.com/article/2012/03/28/us-science-cancer-... (points to a paywalled nature article). In case you are unable to get a copy of the nature paper: 88% of celebrated cancer results published in the last 10 years CAN NOT BE REPLICATED (that is, are most probably WRONG), despite following the "golden rules of science". > The failure to produce previous controlled experiments is alarming. What exactly are you referring to? > I'm just baffled as to why you think that removing controls and oversight and experimenter blinding and lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better? Because I actually live in the real world and not in some fantasy world. I don't think removing controls and oversight is a good idea. However, Roberts has no funding for these experiments because there's no way to make money off them. So he does what he finds interesting, in the best way he can, and lets others try to replicate -- and then collects their results as well. He has stumbled on a diet that seems to be more effective than any other out there (in real world terms, that of keeping weight off years later). It only works for 80% of the people, but for those it works wonders. If you can describe a plausible placebo mechanism for this result, please do - I would love to hear one (and why it seems to skip every other popular diet out there) I do believe that a cheap, safe, easy to do experiment that is not blinded is infinitely more useful than an expensive one that will never be carried out. The first is not perfect, but yields results that are easy to later work with, and easy to verify whether they are right or wrong by doing the same test again or other test that would pinpoint the effect. It is a weak result, but it is a result of some kind. The second yields zero information, since it will never be done. > lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better? You don't have to agree with what Roberts is doing. But you will have to excuse me for pointing out that making comments based on what you infer he is doing or claiming from a ten line summary makes you look like an idiot, even if it makes you feel superior.

beagle3 14y ago

> "controlled experiment" part that really makes up the central part of doing science. In theory, that's how it is supposed to work. But in practice, it doesn't. See e.g. John Ionaddis work showing that a significant part of published research (including that "controlled experiment" setting you are thinking about) is wrong; or the recent nature paper about failing to reproduce 47 out of 53 celebrated ("controlled experiment") results from the last 10 years, and what some more research about those results came up with. Roberts is not doing conventional research. But what he does do is (a) cheap, (b) safe, (c) yields results quickly, and (d) individual testable. He also publishes negative results, and admits mistakes. I think it is actually more useful than 90% of "proper science" done out there. He is not under a "publish or perish" whereas most scientists in the world are, and it shows.

freshhawk 14y ago
Roberts is collected one long detailed case study. Sample size of 1, no controls, no blinding and tiny experiments. Ghost hunters and anti-vaxxers have better science behind their pet theories. We expect this type of endeavour to be extraordinarily prone to influence from noise. The entire modern process of science has been built to control for external factors that Roberts is ignoring or outright embracing. And no one should be surprised when he finds links between brain function and dental amalgam, which is both preposterous and specific to an existing hypothesis common among other pseudosciences. Isn't this just more evidence that the unblinded nature of Roberts anomaly hunting is being skewed by bias? That doesn't mean that some of the things he has found aren't true but 100+ years of trial and error and deep thought on this single issue by an enormous number of intelligent people taking the most meticulous records humanity has ever produced all around the world have determined that his methods will produce more false correlations than true ones. The failure to produce previous controlled experiments is alarming. I'm just baffled as to why you think that removing controls and oversight and experimenter blinding and lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better?
- beagle3 14y ago
  > Roberts is collected one long detailed case study. Sample size of 1, no controls, no blinding and tiny experiments. You might want to actually read what Roberts collects, rather than what you think he collects, because it is apparent that you extrapolate from this post. It's a good way to stroke your ego, but not a good way to have a discussion. > Isn't this just more evidence that the unblinded nature of Roberts anomaly hunting is being skewed by bias? No, it isn't, and if you read what he publishes rather than have a knee-jerk reaction to one short summary, you might be able to appreciate that. > have determined that his methods will produce more false correlations than true ones. That's actually not true, as far as I know, and I would appreciate if you specified exactly what you are referring to. (saying "everyone knows" is not a good enough answer) What I do know is that this meticulous record keeping produces bogus results at an alarming rate because of the things it does not consider, like "publish or perish" pressure and the "negative reuslts go in the drawer" effect. This xkcd describes the mechanism http://xkcd.com/882/ http://xkcd.com/882/ , and these extremely well written, peer-reviewed papers, show that most published results (which are golden according to the school of thought you seem to belong to) is actually wrong: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ http://www.reuters.com/article/2012/03/28/us-science-cancer-idUSBRE82R12P20120328 http://www.reuters.com/article/2012/03/28/us-science-cancer-... (points to a paywalled nature article). In case you are unable to get a copy of the nature paper: 88% of celebrated cancer results published in the last 10 years CAN NOT BE REPLICATED (that is, are most probably WRONG), despite following the "golden rules of science". > The failure to produce previous controlled experiments is alarming. What exactly are you referring to? > I'm just baffled as to why you think that removing controls and oversight and experimenter blinding and lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better? Because I actually live in the real world and not in some fantasy world. I don't think removing controls and oversight is a good idea. However, Roberts has no funding for these experiments because there's no way to make money off them. So he does what he finds interesting, in the best way he can, and lets others try to replicate -- and then collects their results as well. He has stumbled on a diet that seems to be more effective than any other out there (in real world terms, that of keeping weight off years later). It only works for 80% of the people, but for those it works wonders. If you can describe a plausible placebo mechanism for this result, please do - I would love to hear one (and why it seems to skip every other popular diet out there) I do believe that a cheap, safe, easy to do experiment that is not blinded is infinitely more useful than an expensive one that will never be carried out. The first is not perfect, but yields results that are easy to later work with, and easy to verify whether they are right or wrong by doing the same test again or other test that would pinpoint the effect. It is a weak result, but it is a result of some kind. The second yields zero information, since it will never be done. > lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better? You don't have to agree with what Roberts is doing. But you will have to excuse me for pointing out that making comments based on what you infer he is doing or claiming from a ten line summary makes you look like an idiot, even if it makes you feel superior.