5 ms·
The bat-ball and lily-pad questions are 2 of the 3 questions on a short test called the 'Cognitive Reflection Test' (or 'CRT') meant to measure whether people m
by gojomo 14y ago
The bat-ball and lily-pad questions are 2 of the 3 questions on a short test called the 'Cognitive Reflection Test' (or 'CRT') meant to measure whether people make the effort to think beyond the obvious (but wrong) answer.
By using those examples, after its headline, this article seems to imply smarter people do worse on these CRT questions. But that is not what I've read elsewhere -- which is that the CRT is positively correlated with other quantitative measures of intelligence (including IQ scores, SATs, and high-school/collegiate grades). 'Smart' people (by those measures) do tend to do better on the CRT.
And if you read this article carefully, you see that while it uses these two CRT questions as examples of tricky questions, when it discusses the results about awareness-of-bias not helping alleviate bias, it isn't necessarily saying smart people do worse on those two CRT questions. It's a bit muddled in what it's saying, and reviewing the linked abstract doesn't help much either. The paper is evaluating some very specific things under the umbrella term 'cognitive sophistication', which might not map to what we usually call 'smart' or even 'test-smart'.
BTW, I personally think the CRT may be especially useful for evaluating software/systems proficiency. The bat-ball question probes understanding of algebra; the lily-pad question probes understanding of geometric growth (and someone accustomed to powers-of-2 will find it easier); the third question probes understanding of parallelism and projected-rates-of-work.
That third question happens to be:
"If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?"
(A software person might also think of it as: "If it takes 5 cores to compress 5 GB in 5 minutes, how long would it take 100 cores to compress 100 GB?")
- dougabug 14y agoObvious seems to depend on conditioning. For me the correct answer to the lily pad question seemed completely obvious. On the bat and ball problem, I experienced a cognitive dissonance, the knee jerk answer appeared unbidden, yet something also seemed wrong about it so I stopped to actually think about it before answering correctly. Unconscious algebra would appear to be unreliable for most of us.
- gojomo 14y agoMy guess would be familiarity with binary numbers and powers-of-two in a computer context (bytes, ints, longs, shift-operations, etc.) helps make the lily-pad answer 'obvious' to some people. Experience in a particular domain can hone a different reflexive intuition from the norm.
- irishcoffee 14y agoThis was my experience as well. I was very surprised by both: my inability to accept my knee-jerk (because I knew it was a "trick" question) and b.) the 2 minutes it took me to puzzle out why it was 1.05. Which surprises the hell out of me, as I've had more than my fair share of high level math courses.
- kwekly 14y agoThis article has been cobbled together from a few threads here and there taken from Kahneman's own recent book popularising his research (Fast and Slow Thinking, 2011). The book explores several interesting ideas, but I think I can safely say that "why smart people are stupid" is almost certainly NOT one of the more important themes in the book. Which I think is why, as you've noted, the results from the CRT testing don't line up with the conclusions in the article. Actually I'll go a bit further and say that in simplifying a fairly nuanced and complex concept down to an attention grabbing headline, this article is ironically appeals to the intuitive bias trap that Kahneman describes in his book. "Why smart people are stupid" gives us an easy bypass to answer a complex question and saves us the mental effort of actually coming to grips with the problem. The explanation is satisfying, but it's also flat out WRONG. No, the deeper theme is something of an inconvenient truth for both smart and not so smart alike. Best for you to read the book yourself if you're really interested, but I think it's not too big a stretch to say that PG was approaching the same idea in his wisdom (intuition, fast thinking) vs. intelligence (slow, deliberate thinking) essay: "And while wisdom yields calmness, intelligence much of the time leads to discontentment. That's particularly worth remembering. A physicist friend recently told me half his department was on Prozac. Perhaps if we acknowledge that some amount of frustration is inevitable in certain kinds of work, we can mitigate its effects."
- cbsmith 14y agoThanks for this. I kept feeling like the article was just completely lacking in depth. Now I know the research isn't. ;-)
- archgoon 14y ago>>(A software person might also think of it as: "If it takes 5 cores to compress 5 GB in 5 minutes, how long would it take 100 cores to compress 100 GB?") I don't think that most compression algorithms are Embarrassingly Parallel (http://en.wikipedia.org/wiki/Amdahl%27s_law http://en.wikipedia.org/wiki/Amdahl%27s_law), so it's not clear to me the software scenario is equivalent, or are you saying that the jobs come in 5 gigabyte chunks?
- T-hawk 14y agoMake your own chunks, and the algorithm easily becomes embarrassingly parallel. Give each of the 100 cores 1 GB, let them run independently, and don't sweat the few bits of compression efficiency you might lose. If Amdahl's Law comes into play, it's because you're constrained by memory or some other I/O system rather than the CPU core. Decompression may not be embarrassingly parallellizable for an existing algorithm, if it relies on state that persists through every bit in the data set. But a codec algorithm can be designed to be embarrassingly parallel for both legs.
- gojomo 14y agoYou could raise the same objections to the original 'machines'/'widgets' formulation. (There are always economies or diseconomies of scale.) Typically such questions imply the unstated assumption, "answer to the same level of precision/abstraction as the question itself, and assume you have all the info needed to give an answer". With that assumption both questions should be answerable. Compression tends to be pretty parallelizable, if by nothing else than choosing to break the input into separate chunks. You might lose a little bit of efficiency in output size – more restarts, each compressor has less global information – but those don't mean slowdowns (and in a few contrived situations might even mean speedups at the cost of size). See for example 'pigz' and 'pbzip2'. If I were asking this question, I'd accept the 'rough, assuming perfect parallelization' answer as correct-enough in the spirit and level-of-precision implied by the question. If the answerer brought up the difficulties in assuming perfect parallelizability or specific to compression algorithms or choice of inputs, that'd be worth some extra credit, and would trigger followups along the lines of, "how would those factors affect the size?" and "what bottlenecks might you expect?" and "could it ever be faster when split among more machines?"
- gwern 14y ago> And if you read this article carefully, you see that while it uses these two CRT questions as examples of tricky questions, when it discusses the results about awareness-of-bias not helping alleviate bias, it isn't necessarily saying smart people do worse on those two CRT questions. It's a bit muddled in what it's saying, and reviewing the linked abstract doesn't help much either. The paper is evaluating some very specific things under the umbrella term 'cognitive sophistication', which might not map to what we usually call 'smart' or even 'test-smart'. Fulltext: http://dl.dropbox.com/u/5317066/2012-west.pdf http://dl.dropbox.com/u/5317066/2012-west.pdf No, the smart people essentially never perform worse on the CRT or bias questions than their stupider confreres. What the headline is boasting about is that when asked to guess whether they will do better on the biases than their cohort, the smarter people tend to think they will do much better than their cohort, but actually do only a tiny bit better or identical. But as the authors admit at the very end, this may be a reasonable generalization from their lives, in which they observe smart people outperforming stupid people at just about everything and these biases were selected from the big heuristics & biases literature for being resistance to raw smarts...
- cheatercheater 14y agoBear in mind the parent comment isn't saying that the two CRT questions are bad because they probe understanding of mathematics. Understanding of mathematics was presupposed in the objects under test. Everyone had it in them to answer the question correctly, but a large part didn't, and that's the actual kicker of the article.
- hef19898 14y agoSo they are testing if you are too lazy to think much about obvious puzzle questions without any purpose beyond being puzzle questions or not. Yes, you could call that a way of intelligence testing... :-)
- jakobe 14y ago5 minutes!