6 ms·

Why 50% of MIT students get the bat and ball problem wrong

elephant_burger 4y ago
These are the three questions of the Cognitive Reflection Test: (1) A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? __ cents (2) If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets? __ minutes (3) In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake? __ days
- NtochkaNzvanova 4y ago
  Here's another question for the Cognitive Reflection Test: In a study of 3,428 university students, 50% of participants across Harvard, MIT and Princeton got a question wrong on a Cognitive Reflection Test. What percentage of MIT students got the question wrong?
  - nobrains 4y ago
    What is the answer?
    - AwaAwa 4y ago
      Think its meant to be tongue in cheek. Read the OP link, and you'll see where it came from.
    - mytailorisrich 4y ago
      You cannot answer from the data in the question ;)
- enjoiful 4y ago
  Here's ChatGPT's responses. It's fascinating that it provides the same incorrect answers as people... I'm not sure how to interpret this from the robot. (1) "The ball costs $0.10. The bat costs $1.00 more than the ball, so it costs $1.00 + $0.10 = $1.10 in total." (2) "If 5 machines can make 5 widgets in 5 minutes, then each machine can make 1 widget in 5 minutes / 5 machines = 1 minute. Therefore, 100 machines can make 100 widgets in 100 minutes." (3) "If it takes 48 days for the patch of lily pads to cover the entire lake and the patch doubles in size every day, it would take half that time, or 24 days, for the patch to cover half of the lake."
stareatgoats 4y ago
This post title is heavily editorialized. The article title is "Cognitive Reflection Test - Examples of Reacting vs Checking", and doesn't really attempt to give a canonical answer the question of why many people fail the CRT. It discusses some possible reasons (mainly Kahnemans two modes of thinking) and proposes a 3-step strategy to to avoid falling into the trap of going with ones initial, gut feeling. The actual reason for the high proportion of failures doesn't seem that complicated, personally I'd label it as "falling for trick questions" and is something everyone is likely to do if such questions come out of the blue, especially if under some time constraint. Our minds tend to be as economical as possible.
- twelve40 4y ago
  yes exactly, i "failed" that test myself but then i thought, why didn't i do better? Because the cost of failing in this case is absolutely nothing, and the brain automatically optimizes for speed with the first random answer.
- mytailorisrich 4y ago
  The 3 questions quoted in another comment are not trick questions and they obviously did not come out of the blue for anyone taking the test. They are simple reasoning questions. IMHO, people tend to "fall for them" not because they are trick questions but because people don't really reason carefully. The standard advice I have always received is to read the actual question carefully, the whole question and nothing but the question. This might catch a random member of the public but you would think university students are at their peak of "reading test questions skills". Here I think what the questions do is that they may cause our brain to very quickly come up with a shortcut and wrong answer, which should immediately be discarded if you were indeed carefully reading the question, i.e. paying attention (so maybe indeed something like a conflict between 'fast and slow thinking').
t-3 4y ago
Why is there no sales tax? If you're going to do story questions, make them realistic, or else they're just confusing garbage.
- eesmith 4y ago
  For those prices it was clearly something like a garage sale purchase, with no need to pay taxes. Cheapest bat&ball at walmart.com is $10.
  - t-3 4y ago
    Could be a clearance sale, used sporting goods store, in the past, in another country, etc. It's just not a good question for anything but tricking or frustrating people.
    - eesmith 4y ago
      While true, my scenario is not unrealistic. Also, if you include "in another country" then other countries include taxes in the item price.
      - t-3 4y ago
        Yes, that's my point, there's no way to know without making assumptions. Making assumptions about important details is sloppy and irresponsible, not what should be promoted by putting it on a test. They could have easily used a the same setup with weight or speed or anything else but money and it would have been a fine question.
        eesmith 4y ago
        Life requires making assumptions. Understanding the specific tax requirements for this exchange of goods is not an important detail. Your overthinking the problem. As posed, the question doesn't even get to the point where the bat and ball have been purchased, which is where tax needs to be paid. If I'm shopping and someone asks me "I'm sorry, I left my reading glasses at home. Could you tell me the price of the pimentos", then I might say "$2.99 for the jar." I won't say "$2.99 for the jar, plus applicable sales tax", and neither would the vast majority of people.
readthenotes1 4y ago
Regarding fast and slow: "It is likely that Kahneman’s book, or at least some of his chapters, would be very different from the actual book, if it had been written just a few years later. However, in 2011 most psychologists believed that most published results in their journals can be trusted." https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-on-thinking-fast-and-slow/ https://replicationindex.com/2020/12/30/a-meta-scientific-pe... I wonder if the 50% test has been replicated...
phendrenad2 4y ago
I really tried to analyze my thought process before/during/after seeing the problem, trying to solve it, and then seeing the solution. It seems to me that a lot of our thought process isn't under fully conscious control. We're pattern-matching machines, and math and logic don't come naturally to us. To apply math or logic, you have to train your brain to look at a problem without any shortcuts. Even then, you sometimes need to look at a problem from another angle. For instance, I think that people who read this problem out loud, slowly, may spot the trick more easily than those who just try to reason about it silently in their minds.
- smitty1e 4y ago
  The problem for me was the difference between a constant and a variable. "Costs a dollar more" is not saying that the bat cost $1 as a constant. This was a variable, relative price compared to the ball. Failure to understand the mutability--or lack thereof--of the symbols in the system is a source of deep woe.
- mytailorisrich 4y ago
  IMHO, the point is you need to carefully read and analyse the question. Don't let your "fast brain" trick you.
enjoiful 4y ago
I asked ChatGPT this question. Here's its first response: Q: "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?" A: "The ball costs $0.10. The bat costs $1.00 more than the ball, so it costs $1.00 + $0.10 = $1.10 in total." Fascinating that it gets it wrong exactly the same way that a human gets it wrong. I kept trying a few times, and eventually could get a correct answer out of ChatGPT. But most of the time it gives the wrong answer.
hnbear 4y ago
What about other students? Specifying MIT et al. makes it look like it’s a specific flaw in stereotypically ‘smart’ people, rather than the general population. Curious if this is more generally tested and known, or what biases are in place in specific populations. Eg. Bias to immediately trust first thought/gut, vs being taught to be more thoughtful and cautious.

These are the three questions of the Cognitive Reflection Test: (1) A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? __ cents (2) If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets? __ minutes (3) In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake? __ days

NtochkaNzvanova 4y ago
Here's another question for the Cognitive Reflection Test: In a study of 3,428 university students, 50% of participants across Harvard, MIT and Princeton got a question wrong on a Cognitive Reflection Test. What percentage of MIT students got the question wrong?
- nobrains 4y ago
  What is the answer?
  - AwaAwa 4y ago
    Think its meant to be tongue in cheek. Read the OP link, and you'll see where it came from.
  - mytailorisrich 4y ago
    You cannot answer from the data in the question ;)
enjoiful 4y ago
Here's ChatGPT's responses. It's fascinating that it provides the same incorrect answers as people... I'm not sure how to interpret this from the robot. (1) "The ball costs $0.10. The bat costs $1.00 more than the ball, so it costs $1.00 + $0.10 = $1.10 in total." (2) "If 5 machines can make 5 widgets in 5 minutes, then each machine can make 1 widget in 5 minutes / 5 machines = 1 minute. Therefore, 100 machines can make 100 widgets in 100 minutes." (3) "If it takes 48 days for the patch of lily pads to cover the entire lake and the patch doubles in size every day, it would take half that time, or 24 days, for the patch to cover half of the lake."

stareatgoats 4y ago

This post title is heavily editorialized. The article title is "Cognitive Reflection Test - Examples of Reacting vs Checking", and doesn't really attempt to give a canonical answer the question of why many people fail the CRT. It discusses some possible reasons (mainly Kahnemans two modes of thinking) and proposes a 3-step strategy to to avoid falling into the trap of going with ones initial, gut feeling. The actual reason for the high proportion of failures doesn't seem that complicated, personally I'd label it as "falling for trick questions" and is something everyone is likely to do if such questions come out of the blue, especially if under some time constraint. Our minds tend to be as economical as possible.

twelve40 4y ago
yes exactly, i "failed" that test myself but then i thought, why didn't i do better? Because the cost of failing in this case is absolutely nothing, and the brain automatically optimizes for speed with the first random answer.
mytailorisrich 4y ago
The 3 questions quoted in another comment are not trick questions and they obviously did not come out of the blue for anyone taking the test. They are simple reasoning questions. IMHO, people tend to "fall for them" not because they are trick questions but because people don't really reason carefully. The standard advice I have always received is to read the actual question carefully, the whole question and nothing but the question. This might catch a random member of the public but you would think university students are at their peak of "reading test questions skills". Here I think what the questions do is that they may cause our brain to very quickly come up with a shortcut and wrong answer, which should immediately be discarded if you were indeed carefully reading the question, i.e. paying attention (so maybe indeed something like a conflict between 'fast and slow thinking').

t-3 4y ago

Why is there no sales tax? If you're going to do story questions, make them realistic, or else they're just confusing garbage.

eesmith 4y ago
For those prices it was clearly something like a garage sale purchase, with no need to pay taxes. Cheapest bat&ball at walmart.com is $10.
- t-3 4y ago
  Could be a clearance sale, used sporting goods store, in the past, in another country, etc. It's just not a good question for anything but tricking or frustrating people.
  - eesmith 4y ago
    While true, my scenario is not unrealistic. Also, if you include "in another country" then other countries include taxes in the item price.
    - t-3 4y ago
      Yes, that's my point, there's no way to know without making assumptions. Making assumptions about important details is sloppy and irresponsible, not what should be promoted by putting it on a test. They could have easily used a the same setup with weight or speed or anything else but money and it would have been a fine question.
      - eesmith 4y ago
        Life requires making assumptions. Understanding the specific tax requirements for this exchange of goods is not an important detail. Your overthinking the problem. As posed, the question doesn't even get to the point where the bat and ball have been purchased, which is where tax needs to be paid. If I'm shopping and someone asks me "I'm sorry, I left my reading glasses at home. Could you tell me the price of the pimentos", then I might say "$2.99 for the jar." I won't say "$2.99 for the jar, plus applicable sales tax", and neither would the vast majority of people.

readthenotes1 4y ago

Regarding fast and slow: "It is likely that Kahneman’s book, or at least some of his chapters, would be very different from the actual book, if it had been written just a few years later. However, in 2011 most psychologists believed that most published results in their journals can be trusted." https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-on-thinking-fast-and-slow/ https://replicationindex.com/2020/12/30/a-meta-scientific-pe... I wonder if the 50% test has been replicated...

phendrenad2 4y ago

I really tried to analyze my thought process before/during/after seeing the problem, trying to solve it, and then seeing the solution. It seems to me that a lot of our thought process isn't under fully conscious control. We're pattern-matching machines, and math and logic don't come naturally to us. To apply math or logic, you have to train your brain to look at a problem without any shortcuts. Even then, you sometimes need to look at a problem from another angle. For instance, I think that people who read this problem out loud, slowly, may spot the trick more easily than those who just try to reason about it silently in their minds.

smitty1e 4y ago
The problem for me was the difference between a constant and a variable. "Costs a dollar more" is not saying that the bat cost $1 as a constant. This was a variable, relative price compared to the ball. Failure to understand the mutability--or lack thereof--of the symbols in the system is a source of deep woe.
mytailorisrich 4y ago
IMHO, the point is you need to carefully read and analyse the question. Don't let your "fast brain" trick you.

enjoiful 4y ago

I asked ChatGPT this question. Here's its first response: Q: "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?" A: "The ball costs $0.10. The bat costs $1.00 more than the ball, so it costs $1.00 + $0.10 = $1.10 in total." Fascinating that it gets it wrong exactly the same way that a human gets it wrong. I kept trying a few times, and eventually could get a correct answer out of ChatGPT. But most of the time it gives the wrong answer.

hnbear 4y ago

What about other students? Specifying MIT et al. makes it look like it’s a specific flaw in stereotypically ‘smart’ people, rather than the general population. Curious if this is more generally tested and known, or what biases are in place in specific populations. Eg. Bias to immediately trust first thought/gut, vs being taught to be more thoughtful and cautious.