6 ms·

"Professional scientists almost never use this method." That's because of confirmation bias. All of his examples seem to be: 1) Experience a 1 to 2-sigma vari

"Professional scientists almost never use this method." That's because of confirmation bias. All of his examples seem to be: 1) Experience a 1 to 2-sigma variation in normal performance. 2) Look for something different in what you did that day. 3) Keep doing that, and voila, a minor improvement. You'll find yourself doing some pretty goofy things if you follow his suggestion too regularly. Edit: misspelling/layout.

npsimons 14y ago
You'll find yourself doing some pretty goofy things if you follow his suggestion to regularly. Isn't that what science is? It may be biased, and it's very hard to isolate variables, but by the very nature of self-experimentation it's very difficult to do double blind tests. At least some people are trying to live the examined life and look for causes, rather than blaming it on genetics or luck.
- dbaupp 14y ago
  it's very difficult to do double blind tests Depending on the nature of the experiment, it's not that hard, e.g. http://www.gwern.net/Nootropics#adderall-blind-testing http://www.gwern.net/Nootropics#adderall-blind-testing
  - npsimons 14y ago
    Depending on the nature of the experiment, it's not that hard, e.g. http://www.gwern.net/Nootropics#adderall-blind-testing http://www.gwern.net/Nootropics#adderall-blind-testing :) I did almost type "impossible" instead of "very difficult". I'm one of those people who believes nothing is impossible, not even skiing through a revolving door (with short enough skis . . . ).
- kennywinker 14y ago
  He gets to the "hypothesis forming" stage, and draws a conclusion. That's fine if you're trying to figure out how to live your life best. The negative consequences of removing his mercury fillings are probably null, he can accept his hypothesis as true and move on. But this is bad science, without question. We don't know that his mental function has improved because of removing fillings... it's a potential hypothesis, sure, but to say "Mercury Damage Revealed by Brain Test" in a headline seems a bit bold for a sample size of n with no control and a bunch of other confounding variables at play.
  - npsimons 14y ago
    Yes, he does seem to pull out the "jump to conclusions" mat a bit much. That being said, I can't help but think that this would only justify more data. Log more variables; have someone review your methodology; etc. I've always wondered, how do you test your own intelligence? If you're writing and scoring the test, doesn't that invalidate scores you get? Ditto for taking the same test more than once. Maybe I'm just not smart or creative enough to think of ways around this (is that a test? :)
    - dminor 14y ago
      I'm guessing he has a program generating the test using random numbers.
    - sinak 14y ago
      Check out http://quantified-mind.com http://quantified-mind.com for accurate, repeatable cognitive testing. The UI isn't great, but I'm working on that at the moment.
      - nyan_sandwich 14y ago
        It looks like I have to sign up to play. Trivial inconvenience, I know, but now I'm bored and doing other things. Just FYI.
        sinak 14y ago
        Good to know - will work on that
    - beagle3 14y ago
      > I've always wondered, how do you test your own intelligence? He is not measuring intelligence; He's measuring reaction times (how long he takes to solve a very simple arithmetic problem; or how long he takes to see which letter in a 4 letter string appears twice) and fine balance/motor function (how long he can keep his balance on an unstable platform he has built) He went over these things in his blog in great detail over the years. He keeps trying new tests, keeps using those that are statistically stable in a "stable" environment (without changing nutrition / sleep / etc, and after a learning period) - and after he has a statistically significant baseline, he measures how e.g. eating more butter or flax oil changes the results of these tests. Yes, this is drawing conclusions for n=1, but for himself, these conclusions are very scientific - and us humans are not unique enough for them to have wider applicability. No, you cannot deduce from n=1 to the entire human race. Yes, his conclusions are plausible and most of them are verified by others. Read e.g. gwern.net on vitamin D.
- freshhawk 14y ago
  This is what the first part of science is. He's getting as far as one piece of data that might suggest a hypothesis worth testing, although you'd probably want more than one piece of preliminary data or you'd be hunting shadows all the time. He then stops before getting to that whole "controlled experiment" part that really makes up the central part of doing science.
  - beagle3 14y ago
    > "controlled experiment" part that really makes up the central part of doing science. In theory, that's how it is supposed to work. But in practice, it doesn't. See e.g. John Ionaddis work showing that a significant part of published research (including that "controlled experiment" setting you are thinking about) is wrong; or the recent nature paper about failing to reproduce 47 out of 53 celebrated ("controlled experiment") results from the last 10 years, and what some more research about those results came up with. Roberts is not doing conventional research. But what he does do is (a) cheap, (b) safe, (c) yields results quickly, and (d) individual testable. He also publishes negative results, and admits mistakes. I think it is actually more useful than 90% of "proper science" done out there. He is not under a "publish or perish" whereas most scientists in the world are, and it shows.
    - freshhawk 14y ago
      Roberts is collected one long detailed case study. Sample size of 1, no controls, no blinding and tiny experiments. Ghost hunters and anti-vaxxers have better science behind their pet theories. We expect this type of endeavour to be extraordinarily prone to influence from noise. The entire modern process of science has been built to control for external factors that Roberts is ignoring or outright embracing. And no one should be surprised when he finds links between brain function and dental amalgam, which is both preposterous and specific to an existing hypothesis common among other pseudosciences. Isn't this just more evidence that the unblinded nature of Roberts anomaly hunting is being skewed by bias? That doesn't mean that some of the things he has found aren't true but 100+ years of trial and error and deep thought on this single issue by an enormous number of intelligent people taking the most meticulous records humanity has ever produced all around the world have determined that his methods will produce more false correlations than true ones. The failure to produce previous controlled experiments is alarming. I'm just baffled as to why you think that removing controls and oversight and experimenter blinding and lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better?

npsimons 14y ago

You'll find yourself doing some pretty goofy things if you follow his suggestion to regularly. Isn't that what science is? It may be biased, and it's very hard to isolate variables, but by the very nature of self-experimentation it's very difficult to do double blind tests. At least some people are trying to live the examined life and look for causes, rather than blaming it on genetics or luck.

dbaupp 14y ago
it's very difficult to do double blind tests Depending on the nature of the experiment, it's not that hard, e.g. http://www.gwern.net/Nootropics#adderall-blind-testing http://www.gwern.net/Nootropics#adderall-blind-testing
- npsimons 14y ago
  Depending on the nature of the experiment, it's not that hard, e.g. http://www.gwern.net/Nootropics#adderall-blind-testing http://www.gwern.net/Nootropics#adderall-blind-testing :) I did almost type "impossible" instead of "very difficult". I'm one of those people who believes nothing is impossible, not even skiing through a revolving door (with short enough skis . . . ).
kennywinker 14y ago
He gets to the "hypothesis forming" stage, and draws a conclusion. That's fine if you're trying to figure out how to live your life best. The negative consequences of removing his mercury fillings are probably null, he can accept his hypothesis as true and move on. But this is bad science, without question. We don't know that his mental function has improved because of removing fillings... it's a potential hypothesis, sure, but to say "Mercury Damage Revealed by Brain Test" in a headline seems a bit bold for a sample size of n with no control and a bunch of other confounding variables at play.
- npsimons 14y ago
  Yes, he does seem to pull out the "jump to conclusions" mat a bit much. That being said, I can't help but think that this would only justify more data. Log more variables; have someone review your methodology; etc. I've always wondered, how do you test your own intelligence? If you're writing and scoring the test, doesn't that invalidate scores you get? Ditto for taking the same test more than once. Maybe I'm just not smart or creative enough to think of ways around this (is that a test? :)
  - dminor 14y ago
    I'm guessing he has a program generating the test using random numbers.
  - sinak 14y ago
    Check out http://quantified-mind.com http://quantified-mind.com for accurate, repeatable cognitive testing. The UI isn't great, but I'm working on that at the moment.
    - nyan_sandwich 14y ago
      It looks like I have to sign up to play. Trivial inconvenience, I know, but now I'm bored and doing other things. Just FYI.
      - sinak 14y ago
        Good to know - will work on that
  - beagle3 14y ago
    > I've always wondered, how do you test your own intelligence? He is not measuring intelligence; He's measuring reaction times (how long he takes to solve a very simple arithmetic problem; or how long he takes to see which letter in a 4 letter string appears twice) and fine balance/motor function (how long he can keep his balance on an unstable platform he has built) He went over these things in his blog in great detail over the years. He keeps trying new tests, keeps using those that are statistically stable in a "stable" environment (without changing nutrition / sleep / etc, and after a learning period) - and after he has a statistically significant baseline, he measures how e.g. eating more butter or flax oil changes the results of these tests. Yes, this is drawing conclusions for n=1, but for himself, these conclusions are very scientific - and us humans are not unique enough for them to have wider applicability. No, you cannot deduce from n=1 to the entire human race. Yes, his conclusions are plausible and most of them are verified by others. Read e.g. gwern.net on vitamin D.
freshhawk 14y ago
This is what the first part of science is. He's getting as far as one piece of data that might suggest a hypothesis worth testing, although you'd probably want more than one piece of preliminary data or you'd be hunting shadows all the time. He then stops before getting to that whole "controlled experiment" part that really makes up the central part of doing science.
- beagle3 14y ago
  > "controlled experiment" part that really makes up the central part of doing science. In theory, that's how it is supposed to work. But in practice, it doesn't. See e.g. John Ionaddis work showing that a significant part of published research (including that "controlled experiment" setting you are thinking about) is wrong; or the recent nature paper about failing to reproduce 47 out of 53 celebrated ("controlled experiment") results from the last 10 years, and what some more research about those results came up with. Roberts is not doing conventional research. But what he does do is (a) cheap, (b) safe, (c) yields results quickly, and (d) individual testable. He also publishes negative results, and admits mistakes. I think it is actually more useful than 90% of "proper science" done out there. He is not under a "publish or perish" whereas most scientists in the world are, and it shows.
  - freshhawk 14y ago
    Roberts is collected one long detailed case study. Sample size of 1, no controls, no blinding and tiny experiments. Ghost hunters and anti-vaxxers have better science behind their pet theories. We expect this type of endeavour to be extraordinarily prone to influence from noise. The entire modern process of science has been built to control for external factors that Roberts is ignoring or outright embracing. And no one should be surprised when he finds links between brain function and dental amalgam, which is both preposterous and specific to an existing hypothesis common among other pseudosciences. Isn't this just more evidence that the unblinded nature of Roberts anomaly hunting is being skewed by bias? That doesn't mean that some of the things he has found aren't true but 100+ years of trial and error and deep thought on this single issue by an enormous number of intelligent people taking the most meticulous records humanity has ever produced all around the world have determined that his methods will produce more false correlations than true ones. The failure to produce previous controlled experiments is alarming. I'm just baffled as to why you think that removing controls and oversight and experimenter blinding and lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better?