4 ms·

Yes, he does seem to pull out the "jump to conclusions" mat a bit much. That being said, I can't help but think that this would only justify more data. Log more

Yes, he does seem to pull out the "jump to conclusions" mat a bit much. That being said, I can't help but think that this would only justify more data. Log more variables; have someone review your methodology; etc. I've always wondered, how do you test your own intelligence? If you're writing and scoring the test, doesn't that invalidate scores you get? Ditto for taking the same test more than once. Maybe I'm just not smart or creative enough to think of ways around this (is that a test? :)

dminor 14y ago
I'm guessing he has a program generating the test using random numbers.
sinak 14y ago
Check out http://quantified-mind.com http://quantified-mind.com for accurate, repeatable cognitive testing. The UI isn't great, but I'm working on that at the moment.
- nyan_sandwich 14y ago
  It looks like I have to sign up to play. Trivial inconvenience, I know, but now I'm bored and doing other things. Just FYI.
  - sinak 14y ago
    Good to know - will work on that
beagle3 14y ago
> I've always wondered, how do you test your own intelligence? He is not measuring intelligence; He's measuring reaction times (how long he takes to solve a very simple arithmetic problem; or how long he takes to see which letter in a 4 letter string appears twice) and fine balance/motor function (how long he can keep his balance on an unstable platform he has built) He went over these things in his blog in great detail over the years. He keeps trying new tests, keeps using those that are statistically stable in a "stable" environment (without changing nutrition / sleep / etc, and after a learning period) - and after he has a statistically significant baseline, he measures how e.g. eating more butter or flax oil changes the results of these tests. Yes, this is drawing conclusions for n=1, but for himself, these conclusions are very scientific - and us humans are not unique enough for them to have wider applicability. No, you cannot deduce from n=1 to the entire human race. Yes, his conclusions are plausible and most of them are verified by others. Read e.g. gwern.net on vitamin D.
- drostie 14y ago
  I'm sorry, but science is neither "we measure the things" nor is it "we look for the cause." Yes, those are things which scientists do, but no, that is not the essence of science. There is no attempt being made to come to any wholesome understanding. That is, a scientist is trying to model nature, and this consists in two parts: (a) develop a model, (b) test it against nature. Seth Roberts has perhaps hit upon a way to develop models, but he doesn't seem to test them. That is, cum hoc, ergo propter hoc -- "with this, therefore because of this" -- is not actually a test for causation, but merely a guess for causation. To do a proper test, you need variable isolation, and the stability of test results is no guarantee that it is an isolated variable. A good example of an isolated variable is a simple light switch. A terrible example of an isolated variable would be your time of waking up, because that time cannot be isolated from your own thoughts and beliefs -- that is, many people, especially if they're not sleep-deprived, can wake up earlier simply by telling themselves "I'm going to wake up early tomorrow." (I myself usually wake up before my alarm clock goes off.) Humans in general are not light switches. As has been documented repeatedly in medicine, dual effects of placebo and hypochondria plague us; things we expect to be medicinal often placate us even when they have no medicinal content and you can suddenly feel the symptoms of a malady soon after reading about it on Wikipedia. So even if you want to say "for himself, these conclusions are very scientific," you're going to have to account for the problem that he is probably going to confirm whatever he already expects. That is, any "follow up" tests after the first guess are already tainted by the fact that Seth knows what's being tested.
  - beagle3 14y ago
    > There is no attempt being made to come to any wholesome understanding Do you actually follow Roberts, or is your understand based on this one short summary? Because, e.g. "What Makes Food Fattening" (77-page pdf here http://media.sethroberts.net/about/whatmakesfoodfattening.pdf http://media.sethroberts.net/about/whatmakesfoodfattening.pd... ) is very much an attempt to come to wholesome understanding, develop a model, test it against nature. (The test is "putting it out in the wild", and the result is "works beautifully for 80% of people who try, not at all for the remaining 20%". He's not funded to test this, nor is he making any money of it - I don't think there's a better route for him to take). > To do a proper test, you need variable isolation, and the stability of test results is no guarantee that it is an isolated variable Yep. Except, in the real world, NO published result related to nutrition, and almost no published result regarding medicine, is a proper test with isolated variables. Including almost every guideline your doctor works by. > that is, many people, especially if they're not sleep-deprived, can wake up earlier simply by telling themselves "I'm going to wake up early tomorrow." (I myself usually wake up before my alarm clock goes off.) That is true. And yet, a lot of people want diets to work, and they don't. A lot of insomniacs want a placebo to work, and it doesn't. Placebo is powerful, but is not all-powerful. It is stronger with some people, weaker with others. Discarding results just because they were not the result of a double-blind placebo-controlled test is not rational. Neither is accepting results just because the author thinks that they are double-blind placebo-controlled: e.g., almost all placebos today are sugar pills; If the tested-against-material has a side effect, such as causing flushing or a dry tongue (though does not produce the wanted outcome -- which is very often the case), this is not in face a double-blind; the patient knows that they did not get a placebo, and the whole test is useless. Yet, this is the gold standard. Furthermore, if you read the NEJM / BMJ / Nature / Science medical papers, you'd notice that their result are tested on (and are therefore only valid for) a small ethnic group, a small age group, etc. But then, very unscientifically, it is assumed (by others -- usually not by the authors of said paper) to apply much more widely. > That is, any "follow up" tests after the first guess are already tainted by the fact that Seth knows what's being tested. Yes, that does not make the results any less true or useful: Butter makes seth faster at arithmetics, in a consistent and statistically significant way. That may not be true for others. And may be entirely a placebo. Nevertheless, if Seth wants to be faster at arithmetics, he can do that by eating butter, regardless of the cause. Is that not science?

dminor 14y ago

I'm guessing he has a program generating the test using random numbers.

sinak 14y ago

Check out http://quantified-mind.com http://quantified-mind.com for accurate, repeatable cognitive testing. The UI isn't great, but I'm working on that at the moment.

nyan_sandwich 14y ago
It looks like I have to sign up to play. Trivial inconvenience, I know, but now I'm bored and doing other things. Just FYI.
- sinak 14y ago
  Good to know - will work on that

beagle3 14y ago

> I've always wondered, how do you test your own intelligence? He is not measuring intelligence; He's measuring reaction times (how long he takes to solve a very simple arithmetic problem; or how long he takes to see which letter in a 4 letter string appears twice) and fine balance/motor function (how long he can keep his balance on an unstable platform he has built) He went over these things in his blog in great detail over the years. He keeps trying new tests, keeps using those that are statistically stable in a "stable" environment (without changing nutrition / sleep / etc, and after a learning period) - and after he has a statistically significant baseline, he measures how e.g. eating more butter or flax oil changes the results of these tests. Yes, this is drawing conclusions for n=1, but for himself, these conclusions are very scientific - and us humans are not unique enough for them to have wider applicability. No, you cannot deduce from n=1 to the entire human race. Yes, his conclusions are plausible and most of them are verified by others. Read e.g. gwern.net on vitamin D.

drostie 14y ago
I'm sorry, but science is neither "we measure the things" nor is it "we look for the cause." Yes, those are things which scientists do, but no, that is not the essence of science. There is no attempt being made to come to any wholesome understanding. That is, a scientist is trying to model nature, and this consists in two parts: (a) develop a model, (b) test it against nature. Seth Roberts has perhaps hit upon a way to develop models, but he doesn't seem to test them. That is, cum hoc, ergo propter hoc -- "with this, therefore because of this" -- is not actually a test for causation, but merely a guess for causation. To do a proper test, you need variable isolation, and the stability of test results is no guarantee that it is an isolated variable. A good example of an isolated variable is a simple light switch. A terrible example of an isolated variable would be your time of waking up, because that time cannot be isolated from your own thoughts and beliefs -- that is, many people, especially if they're not sleep-deprived, can wake up earlier simply by telling themselves "I'm going to wake up early tomorrow." (I myself usually wake up before my alarm clock goes off.) Humans in general are not light switches. As has been documented repeatedly in medicine, dual effects of placebo and hypochondria plague us; things we expect to be medicinal often placate us even when they have no medicinal content and you can suddenly feel the symptoms of a malady soon after reading about it on Wikipedia. So even if you want to say "for himself, these conclusions are very scientific," you're going to have to account for the problem that he is probably going to confirm whatever he already expects. That is, any "follow up" tests after the first guess are already tainted by the fact that Seth knows what's being tested.
- beagle3 14y ago
  > There is no attempt being made to come to any wholesome understanding Do you actually follow Roberts, or is your understand based on this one short summary? Because, e.g. "What Makes Food Fattening" (77-page pdf here http://media.sethroberts.net/about/whatmakesfoodfattening.pdf http://media.sethroberts.net/about/whatmakesfoodfattening.pd... ) is very much an attempt to come to wholesome understanding, develop a model, test it against nature. (The test is "putting it out in the wild", and the result is "works beautifully for 80% of people who try, not at all for the remaining 20%". He's not funded to test this, nor is he making any money of it - I don't think there's a better route for him to take). > To do a proper test, you need variable isolation, and the stability of test results is no guarantee that it is an isolated variable Yep. Except, in the real world, NO published result related to nutrition, and almost no published result regarding medicine, is a proper test with isolated variables. Including almost every guideline your doctor works by. > that is, many people, especially if they're not sleep-deprived, can wake up earlier simply by telling themselves "I'm going to wake up early tomorrow." (I myself usually wake up before my alarm clock goes off.) That is true. And yet, a lot of people want diets to work, and they don't. A lot of insomniacs want a placebo to work, and it doesn't. Placebo is powerful, but is not all-powerful. It is stronger with some people, weaker with others. Discarding results just because they were not the result of a double-blind placebo-controlled test is not rational. Neither is accepting results just because the author thinks that they are double-blind placebo-controlled: e.g., almost all placebos today are sugar pills; If the tested-against-material has a side effect, such as causing flushing or a dry tongue (though does not produce the wanted outcome -- which is very often the case), this is not in face a double-blind; the patient knows that they did not get a placebo, and the whole test is useless. Yet, this is the gold standard. Furthermore, if you read the NEJM / BMJ / Nature / Science medical papers, you'd notice that their result are tested on (and are therefore only valid for) a small ethnic group, a small age group, etc. But then, very unscientifically, it is assumed (by others -- usually not by the authors of said paper) to apply much more widely. > That is, any "follow up" tests after the first guess are already tainted by the fact that Seth knows what's being tested. Yes, that does not make the results any less true or useful: Butter makes seth faster at arithmetics, in a consistent and statistically significant way. That may not be true for others. And may be entirely a placebo. Nevertheless, if Seth wants to be faster at arithmetics, he can do that by eating butter, regardless of the cause. Is that not science?