14 ms·

Why Self-Track? The Possibility of Hard-to-Explain Change

76 points by harscoat 14y ago

rubidium 14y ago
"Professional scientists almost never use this method." That's because of confirmation bias. All of his examples seem to be: 1) Experience a 1 to 2-sigma variation in normal performance. 2) Look for something different in what you did that day. 3) Keep doing that, and voila, a minor improvement. You'll find yourself doing some pretty goofy things if you follow his suggestion too regularly. Edit: misspelling/layout.
- npsimons 14y ago
  You'll find yourself doing some pretty goofy things if you follow his suggestion to regularly. Isn't that what science is? It may be biased, and it's very hard to isolate variables, but by the very nature of self-experimentation it's very difficult to do double blind tests. At least some people are trying to live the examined life and look for causes, rather than blaming it on genetics or luck.
  - dbaupp 14y ago
    it's very difficult to do double blind tests Depending on the nature of the experiment, it's not that hard, e.g. http://www.gwern.net/Nootropics#adderall-blind-testing http://www.gwern.net/Nootropics#adderall-blind-testing
    - npsimons 14y ago
      Depending on the nature of the experiment, it's not that hard, e.g. http://www.gwern.net/Nootropics#adderall-blind-testing http://www.gwern.net/Nootropics#adderall-blind-testing :) I did almost type "impossible" instead of "very difficult". I'm one of those people who believes nothing is impossible, not even skiing through a revolving door (with short enough skis . . . ).
  - kennywinker 14y ago
    He gets to the "hypothesis forming" stage, and draws a conclusion. That's fine if you're trying to figure out how to live your life best. The negative consequences of removing his mercury fillings are probably null, he can accept his hypothesis as true and move on. But this is bad science, without question. We don't know that his mental function has improved because of removing fillings... it's a potential hypothesis, sure, but to say "Mercury Damage Revealed by Brain Test" in a headline seems a bit bold for a sample size of n with no control and a bunch of other confounding variables at play.
    - npsimons 14y ago
      Yes, he does seem to pull out the "jump to conclusions" mat a bit much. That being said, I can't help but think that this would only justify more data. Log more variables; have someone review your methodology; etc. I've always wondered, how do you test your own intelligence? If you're writing and scoring the test, doesn't that invalidate scores you get? Ditto for taking the same test more than once. Maybe I'm just not smart or creative enough to think of ways around this (is that a test? :)
      - dminor 14y ago
        I'm guessing he has a program generating the test using random numbers.
      - sinak 14y ago
        Check out http://quantified-mind.com http://quantified-mind.com for accurate, repeatable cognitive testing. The UI isn't great, but I'm working on that at the moment.
        nyan_sandwich 14y ago
        It looks like I have to sign up to play. Trivial inconvenience, I know, but now I'm bored and doing other things. Just FYI.
        14y ago
  - freshhawk 14y ago
    This is what the first part of science is. He's getting as far as one piece of data that might suggest a hypothesis worth testing, although you'd probably want more than one piece of preliminary data or you'd be hunting shadows all the time. He then stops before getting to that whole "controlled experiment" part that really makes up the central part of doing science.
    - beagle3 14y ago
      > "controlled experiment" part that really makes up the central part of doing science. In theory, that's how it is supposed to work. But in practice, it doesn't. See e.g. John Ionaddis work showing that a significant part of published research (including that "controlled experiment" setting you are thinking about) is wrong; or the recent nature paper about failing to reproduce 47 out of 53 celebrated ("controlled experiment") results from the last 10 years, and what some more research about those results came up with. Roberts is not doing conventional research. But what he does do is (a) cheap, (b) safe, (c) yields results quickly, and (d) individual testable. He also publishes negative results, and admits mistakes. I think it is actually more useful than 90% of "proper science" done out there. He is not under a "publish or perish" whereas most scientists in the world are, and it shows.
      - freshhawk 14y ago
        Roberts is collected one long detailed case study. Sample size of 1, no controls, no blinding and tiny experiments. Ghost hunters and anti-vaxxers have better science behind their pet theories. We expect this type of endeavour to be extraordinarily prone to influence from noise. The entire modern process of science has been built to control for external factors that Roberts is ignoring or outright embracing. And no one should be surprised when he finds links between brain function and dental amalgam, which is both preposterous and specific to an existing hypothesis common among other pseudosciences. Isn't this just more evidence that the unblinded nature of Roberts anomaly hunting is being skewed by bias? That doesn't mean that some of the things he has found aren't true but 100+ years of trial and error and deep thought on this single issue by an enormous number of intelligent people taking the most meticulous records humanity has ever produced all around the world have determined that his methods will produce more false correlations than true ones. The failure to produce previous controlled experiments is alarming. I'm just baffled as to why you think that removing controls and oversight and experimenter blinding and lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better?
npsimons 14y ago
Most serious athletes already know about this (in fact, on that same site: http://quantifiedself.com/2011/03/personal-data-visualization/ http://quantifiedself.com/2011/03/personal-data-visualizatio...). Keeping a record of your resting heart rate (plus blood pressure, temperature, etc) every morning will tell you many things. For example, I managed to get my resting heart rate down to mid 40s when I was training regularly (and heavily). Now that I'm more sedentary, my resting HR is in the mid 50s to low 60s, and I definitely don't feel as good or make as good times. Spikes while in training were also usually indicative of overtraining. It's also mentioned in "The Hacker's Diet" that if you weigh every day, but pay attention to the moving average it can be much more encouraging.
cunninghamd 14y ago
But HOW do you track all that data? Today, with a smartphone, it seems easier, but even then I’d like something akin to a master spreadsheet that I can store it all in. Seth: what format are you tracking all that data in? Of course, now that I type that, I’m worried the answer is “pen and paper.”
- jurjenh 14y ago
  In one of the linked pages - http://quantifiedself.com/2012/07/nick-winter-a-lazy-mans-approach-to-cognitive-testing/ http://quantifiedself.com/2012/07/nick-winter-a-lazy-mans-ap... he mentions quantifed mind, but I can't find any site with that name. EDIT: Ooops, a little googling further reveals http://www.quantified-mind.com http://www.quantified-mind.com After a little link-chasing I did get to a guide / tools page that may be useful: http://quantifiedself.com/guide/ http://quantifiedself.com/guide/
- mahyarm 14y ago
  One of my someday projects to create a generic self tracking app that presents a dashboard for quick entry tracking and visualization and syncs with a website. It will also export to a spreadsheet. It will also allow you to visualize data with graphs and so on. There will be a bunch of presets for various special types of graphs, like with blood pressure and so on. You could also specify zones for numbers so you can show 'red' for a danger zone glucose level for example. Managing spreadsheets on a smartphone is very kludgey.
  - raelshark 14y ago
    That's basically a project I'm actively developing now - generic self-tracking (manual and via sensors and imported from other platforms), with presets and intuitive visualizations. This is a big gap in most tracking tools that are out there - letting you track anything you want, while also making sense of all the information in a useful way. My motivation is to create something that's effective for sufferers of obscure chronic illnesses, since most tracking tools out there focus on major well-known illnesses, or simply aren't flexible enough to track weird symptoms that are so obscure that the developers have never heard of them. But the side-product is that the design will therefore be flexible enough to be useful for anyone who wants to track any element of their health.
doktrin 14y ago
I have to admit I cringed at some of the examples provided, as well as the associated causal evidence. 1. Eating Fruit for Breakfast => Reduced Sleep Quality [in fact, "...I figured out that any breakfast, if eaten early, disturbed my sleep". 2. Removal of Mercury Fillings => Improved Brain Function. The evidence was surprisingly lacking : "The evidence for causality — removal of mercury amalgam fillings improved my arithmetic score — rests on three things: 1. Four other explanations made incorrect predictions. 2. The improvement, which lasted months, started within a few days of the removal. Long-term improvements (not due to practice) are rare — this is the only one I’ve noticed. 3. Mercury is known to harm neural function (“mad as a hatter”). As far as I’m concerned, that’s plenty." I am all for self-observation, but I can't help but think that some of the causal links the OP has established may not be particularly well founded.
- SwellJoe 14y ago
  But, but, we always rub the bald guy's head before the game, and we totally play better! I wrote it down, so it is now science.
- beagle3 14y ago
  If this was the only writeup Roberts ever produced, the cringing would be well deserved. However, he does have a long history of correctly quantifying changes in himself and identifying their source. Specifically, his (on its face, ridiculous) shangri-la diet works well for 80% of the people -- probably better than any other diet (Although it should be noted that about 20% of people who try report it does not work at all). Also, his flax seed, butter and vitamin D results have been replicated consistently by many (though again, not all). You can wait 15 years until these things get verified/shown wrong, or just try them yourself. And about "well founded" - it's easy to dismiss some guy on the internet (regardless of his track record and academic qualifications, as this particular writeup was not done in the context of these qualifications). But you should be aware of your confirmation bias - e.g., if you ever considered any of the examples I give here http://news.ycombinator.com/item?id=4427910 http://news.ycombinator.com/item?id=4427910 well founded.
mthoms 14y ago
Are we sure this isn't trolling? I mean come on: "Sleep and breakfast. I changed my breakfast from oatmeal to fruit because a student told me he had lost weight eating foods with high water content (such as fruit). I did not lose weight but my sleep suddenly got worse. I started waking up early every morning instead of half the time. From this I figured out that any breakfast, if eaten early, disturbed my sleep."
- beagle3 14y ago
  It isn't trolling. It's a short summary piece. If you are into these things, his blog and semi-formal papers ("what makes food fattening" and others) are a gold mine.
A1kmm 14y ago
This is a method for hypothesis generation, but is not a valid way to draw inferences. The reason why scientists don't draw conclusion from single data points (analogies) is that you can come up with flawed conclusions as a result. The changes could be purely random, or due to some other explanation that wasn't thought of. In order of reliability: 1. Do an experiment where an intervention is assigned at random, and scored by a method that is blind to the intervention (preferably on subjects who are blind to the intervention - but that is hard for some cases). 2. Observe historical data where there have been many instances of each of the variables under study, avoiding correlation between variables with things like time. 3. Use anecdotes.
asher_ 14y ago
I love the QS movement, but I sometimes worry for it. It is young, and there is a wide variety of methodologies being used to draw conclusions, some are solid, others are far from it. This can be expected from a group of people that have no requirements for joining, and this is a strength as well as a weakness. I have noticed that QS has attracted some "alternative [to] medicine" types who seem to be happy to have the validation, as well as some seriously interesting projects from hard core science types. I am hoping that as this movement evolves, it will mature in the right way. Having amateurs participating is incredibly worthwhile, but guidance and best practices will hopefully develop in the community so that when people want to draw conclusions about causality, they can do it legitimately.
- freshhawk 14y ago
  I feel the same as you, but the movement is evolving in the alt-med/pseudoscience direction really quickly. Amateurs are collecting tons of cool data (amazing!) and then doing their amateur analysis (Ancient Aliens level depressing!). Without a very strong internal pressure to get their shit together and act like smart people who understand that some problems require expertise the movement is screwed.

76 points by harscoat 14y ago

"Professional scientists almost never use this method." That's because of confirmation bias. All of his examples seem to be: 1) Experience a 1 to 2-sigma variation in normal performance. 2) Look for something different in what you did that day. 3) Keep doing that, and voila, a minor improvement. You'll find yourself doing some pretty goofy things if you follow his suggestion too regularly. Edit: misspelling/layout.

npsimons 14y ago
You'll find yourself doing some pretty goofy things if you follow his suggestion to regularly. Isn't that what science is? It may be biased, and it's very hard to isolate variables, but by the very nature of self-experimentation it's very difficult to do double blind tests. At least some people are trying to live the examined life and look for causes, rather than blaming it on genetics or luck.
- dbaupp 14y ago
  it's very difficult to do double blind tests Depending on the nature of the experiment, it's not that hard, e.g. http://www.gwern.net/Nootropics#adderall-blind-testing http://www.gwern.net/Nootropics#adderall-blind-testing
  - npsimons 14y ago
    Depending on the nature of the experiment, it's not that hard, e.g. http://www.gwern.net/Nootropics#adderall-blind-testing http://www.gwern.net/Nootropics#adderall-blind-testing :) I did almost type "impossible" instead of "very difficult". I'm one of those people who believes nothing is impossible, not even skiing through a revolving door (with short enough skis . . . ).
- kennywinker 14y ago
  He gets to the "hypothesis forming" stage, and draws a conclusion. That's fine if you're trying to figure out how to live your life best. The negative consequences of removing his mercury fillings are probably null, he can accept his hypothesis as true and move on. But this is bad science, without question. We don't know that his mental function has improved because of removing fillings... it's a potential hypothesis, sure, but to say "Mercury Damage Revealed by Brain Test" in a headline seems a bit bold for a sample size of n with no control and a bunch of other confounding variables at play.
  - npsimons 14y ago
    Yes, he does seem to pull out the "jump to conclusions" mat a bit much. That being said, I can't help but think that this would only justify more data. Log more variables; have someone review your methodology; etc. I've always wondered, how do you test your own intelligence? If you're writing and scoring the test, doesn't that invalidate scores you get? Ditto for taking the same test more than once. Maybe I'm just not smart or creative enough to think of ways around this (is that a test? :)
    - dminor 14y ago
      I'm guessing he has a program generating the test using random numbers.
    - sinak 14y ago
      Check out http://quantified-mind.com http://quantified-mind.com for accurate, repeatable cognitive testing. The UI isn't great, but I'm working on that at the moment.
      - nyan_sandwich 14y ago
        It looks like I have to sign up to play. Trivial inconvenience, I know, but now I'm bored and doing other things. Just FYI.
        14y ago
- freshhawk 14y ago
  This is what the first part of science is. He's getting as far as one piece of data that might suggest a hypothesis worth testing, although you'd probably want more than one piece of preliminary data or you'd be hunting shadows all the time. He then stops before getting to that whole "controlled experiment" part that really makes up the central part of doing science.
  - beagle3 14y ago
    > "controlled experiment" part that really makes up the central part of doing science. In theory, that's how it is supposed to work. But in practice, it doesn't. See e.g. John Ionaddis work showing that a significant part of published research (including that "controlled experiment" setting you are thinking about) is wrong; or the recent nature paper about failing to reproduce 47 out of 53 celebrated ("controlled experiment") results from the last 10 years, and what some more research about those results came up with. Roberts is not doing conventional research. But what he does do is (a) cheap, (b) safe, (c) yields results quickly, and (d) individual testable. He also publishes negative results, and admits mistakes. I think it is actually more useful than 90% of "proper science" done out there. He is not under a "publish or perish" whereas most scientists in the world are, and it shows.
    - freshhawk 14y ago
      Roberts is collected one long detailed case study. Sample size of 1, no controls, no blinding and tiny experiments. Ghost hunters and anti-vaxxers have better science behind their pet theories. We expect this type of endeavour to be extraordinarily prone to influence from noise. The entire modern process of science has been built to control for external factors that Roberts is ignoring or outright embracing. And no one should be surprised when he finds links between brain function and dental amalgam, which is both preposterous and specific to an existing hypothesis common among other pseudosciences. Isn't this just more evidence that the unblinded nature of Roberts anomaly hunting is being skewed by bias? That doesn't mean that some of the things he has found aren't true but 100+ years of trial and error and deep thought on this single issue by an enormous number of intelligent people taking the most meticulous records humanity has ever produced all around the world have determined that his methods will produce more false correlations than true ones. The failure to produce previous controlled experiments is alarming. I'm just baffled as to why you think that removing controls and oversight and experimenter blinding and lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better?

npsimons 14y ago

Most serious athletes already know about this (in fact, on that same site: http://quantifiedself.com/2011/03/personal-data-visualization/ http://quantifiedself.com/2011/03/personal-data-visualizatio...). Keeping a record of your resting heart rate (plus blood pressure, temperature, etc) every morning will tell you many things. For example, I managed to get my resting heart rate down to mid 40s when I was training regularly (and heavily). Now that I'm more sedentary, my resting HR is in the mid 50s to low 60s, and I definitely don't feel as good or make as good times. Spikes while in training were also usually indicative of overtraining. It's also mentioned in "The Hacker's Diet" that if you weigh every day, but pay attention to the moving average it can be much more encouraging.

cunninghamd 14y ago

But HOW do you track all that data? Today, with a smartphone, it seems easier, but even then I’d like something akin to a master spreadsheet that I can store it all in. Seth: what format are you tracking all that data in? Of course, now that I type that, I’m worried the answer is “pen and paper.”

jurjenh 14y ago
In one of the linked pages - http://quantifiedself.com/2012/07/nick-winter-a-lazy-mans-approach-to-cognitive-testing/ http://quantifiedself.com/2012/07/nick-winter-a-lazy-mans-ap... he mentions quantifed mind, but I can't find any site with that name. EDIT: Ooops, a little googling further reveals http://www.quantified-mind.com http://www.quantified-mind.com After a little link-chasing I did get to a guide / tools page that may be useful: http://quantifiedself.com/guide/ http://quantifiedself.com/guide/
mahyarm 14y ago
One of my someday projects to create a generic self tracking app that presents a dashboard for quick entry tracking and visualization and syncs with a website. It will also export to a spreadsheet. It will also allow you to visualize data with graphs and so on. There will be a bunch of presets for various special types of graphs, like with blood pressure and so on. You could also specify zones for numbers so you can show 'red' for a danger zone glucose level for example. Managing spreadsheets on a smartphone is very kludgey.
- raelshark 14y ago
  That's basically a project I'm actively developing now - generic self-tracking (manual and via sensors and imported from other platforms), with presets and intuitive visualizations. This is a big gap in most tracking tools that are out there - letting you track anything you want, while also making sense of all the information in a useful way. My motivation is to create something that's effective for sufferers of obscure chronic illnesses, since most tracking tools out there focus on major well-known illnesses, or simply aren't flexible enough to track weird symptoms that are so obscure that the developers have never heard of them. But the side-product is that the design will therefore be flexible enough to be useful for anyone who wants to track any element of their health.

doktrin 14y ago

I have to admit I cringed at some of the examples provided, as well as the associated causal evidence. 1. Eating Fruit for Breakfast => Reduced Sleep Quality [in fact, "...I figured out that any breakfast, if eaten early, disturbed my sleep". 2. Removal of Mercury Fillings => Improved Brain Function. The evidence was surprisingly lacking : "The evidence for causality — removal of mercury amalgam fillings improved my arithmetic score — rests on three things: 1. Four other explanations made incorrect predictions. 2. The improvement, which lasted months, started within a few days of the removal. Long-term improvements (not due to practice) are rare — this is the only one I’ve noticed. 3. Mercury is known to harm neural function (“mad as a hatter”). As far as I’m concerned, that’s plenty." I am all for self-observation, but I can't help but think that some of the causal links the OP has established may not be particularly well founded.

SwellJoe 14y ago
But, but, we always rub the bald guy's head before the game, and we totally play better! I wrote it down, so it is now science.
beagle3 14y ago
If this was the only writeup Roberts ever produced, the cringing would be well deserved. However, he does have a long history of correctly quantifying changes in himself and identifying their source. Specifically, his (on its face, ridiculous) shangri-la diet works well for 80% of the people -- probably better than any other diet (Although it should be noted that about 20% of people who try report it does not work at all). Also, his flax seed, butter and vitamin D results have been replicated consistently by many (though again, not all). You can wait 15 years until these things get verified/shown wrong, or just try them yourself. And about "well founded" - it's easy to dismiss some guy on the internet (regardless of his track record and academic qualifications, as this particular writeup was not done in the context of these qualifications). But you should be aware of your confirmation bias - e.g., if you ever considered any of the examples I give here http://news.ycombinator.com/item?id=4427910 http://news.ycombinator.com/item?id=4427910 well founded.

mthoms 14y ago

Are we sure this isn't trolling? I mean come on: "Sleep and breakfast. I changed my breakfast from oatmeal to fruit because a student told me he had lost weight eating foods with high water content (such as fruit). I did not lose weight but my sleep suddenly got worse. I started waking up early every morning instead of half the time. From this I figured out that any breakfast, if eaten early, disturbed my sleep."

beagle3 14y ago
It isn't trolling. It's a short summary piece. If you are into these things, his blog and semi-formal papers ("what makes food fattening" and others) are a gold mine.

A1kmm 14y ago

This is a method for hypothesis generation, but is not a valid way to draw inferences. The reason why scientists don't draw conclusion from single data points (analogies) is that you can come up with flawed conclusions as a result. The changes could be purely random, or due to some other explanation that wasn't thought of. In order of reliability: 1. Do an experiment where an intervention is assigned at random, and scored by a method that is blind to the intervention (preferably on subjects who are blind to the intervention - but that is hard for some cases). 2. Observe historical data where there have been many instances of each of the variables under study, avoiding correlation between variables with things like time. 3. Use anecdotes.

asher_ 14y ago

I love the QS movement, but I sometimes worry for it. It is young, and there is a wide variety of methodologies being used to draw conclusions, some are solid, others are far from it. This can be expected from a group of people that have no requirements for joining, and this is a strength as well as a weakness. I have noticed that QS has attracted some "alternative [to] medicine" types who seem to be happy to have the validation, as well as some seriously interesting projects from hard core science types. I am hoping that as this movement evolves, it will mature in the right way. Having amateurs participating is incredibly worthwhile, but guidance and best practices will hopefully develop in the community so that when people want to draw conclusions about causality, they can do it legitimately.

freshhawk 14y ago
I feel the same as you, but the movement is evolving in the alt-med/pseudoscience direction really quickly. Amateurs are collecting tons of cool data (amazing!) and then doing their amateur analysis (Ancient Aliens level depressing!). Without a very strong internal pressure to get their shit together and act like smart people who understand that some problems require expertise the movement is screwed.