9 ms·
I'm curious to see more about the distribution of questions and answers people had, and how the HN population may differ from the NYT's. There will certainly b
by ddlatham 11y ago
I'm curious to see more about the distribution of questions and answers people had, and how the HN population may differ from the NYT's. There will certainly be self selection bias here, but if you're willing to share how you did with others, please enter it here:
https://docs.google.com/forms/d/17e5BIL0lH8OHsGj89Zdtdl8GeCVLC8TbMxsPApGcSGg/viewform https://docs.google.com/forms/d/17e5BIL0lH8OHsGj89Zdtdl8GeCV...
The result summary is visible here:
https://docs.google.com/forms/d/17e5BIL0lH8OHsGj89Zdtdl8GeCVLC8TbMxsPApGcSGg/viewanalytics https://docs.google.com/forms/d/17e5BIL0lH8OHsGj89Zdtdl8GeCV...
The raw answers are visible here:
https://docs.google.com/spreadsheets/d/1ZxR2_eOUtNLXwgKfLO1JWhiXx-UQOxsYlKaBFe7f3m0/ https://docs.google.com/spreadsheets/d/1ZxR2_eOUtNLXwgKfLO1J...
- fenomas 11y agoCool idea, thanks! Personally, I spent most of my answers playing with the inputs. The form was happy to report that "1e1, 15, 0x10" was a valid sequence. :D
- veli_joza 11y agoPeople familiar with unit testing and test driven development will feel at home with this kind of puzzle. That doesn't mean that they will be less biased in social/political decisions, it just means that this test will fail to prove a point.
- ttctciyf 11y agoHah :) Seriously, though, it seems a bit of leap from the existence of confirmation bias to explaining away the public outpourings of US Politicians about their financial crises and foreign policy disasters - in the absence of better data as to just why the given statements were made ascribing this to confirmation bias seems itself open to accusations of confirmation bias! :)
- diminishedprime 11y agoThat's exactly what I was thinking. I (sometimes) follow TDD, and I applied it to this problem. I made sure to include negatives, 0, positives, and include primes here or there to help avoid issues with multiplication/exponentiation. After a few of these, I felt pretty confident that the rule was simple.
- ubernostrum 11y agoMy process was: [3 5 7] [7 5 3] [8 4 2] [5 7 3] [1 2 3] [1 1 1] [0 1 2] At that point I could've done some more to be really certain, but felt confident enough and guessed (correctly).
- jonkiddy 11y ago[16 32 64] [0 0 0] [1 2 3] [3 2 1] [-1 0 1] [3 5 7] [-2 -1 0] [111 54231 9999999999999999999999] At that point, I correctly answered the question.
- jws 11y agoSimilar to mine, but we both actually failed. Real numbers are accepted and we did not test that.
- WildUtah 11y agoIt accepts floating point numbers. To approximately double precision. But it accepts zero percent of real numbers.
- eru 11y agoI can nitpick better than that! To talk about a certain fraction of real numbers you have to have a distribution over them. In general we take the uniform distribution if no distribution is explicitly given. That doesn't work for real numbers (it doesn't even work for natural numbers). (See https://math.stackexchange.com/questions/14777/why-isnt-there-a-uniform-probability-distribution-over-the-positive-real-number https://math.stackexchange.com/questions/14777/why-isnt-ther...) If there's no implicit default distribution, we have to pick on. I can pick one where they cover an arbitrary high percentage of real numbers..
- elnion 11y agoDown the rabbit hole of pedantry: we don't need a distribution, just a measure, if we want to talk about how many reals it accepts. The Lebesgue measure is implied on the Reals if none is given, and the computable reals have measure zero. We can't reasonably talk about a percent coverage, since the Lebesgue measure of the reals is infinite, but as a non-technical description, 'zero percent' is morally equivalent to saying it only covers a measure-zero set.
- benihana 11y agoI was just writing this out! I believe practice in unit testing is what let me get this question right. I almost submitted n^1, n^2, n^3 before I realized I didn't get any wrong answers and something didn't feel right. I credit this 'instinct' to having written thousands of tests, and trying to make them fail to make sure my tests weren't lying to me.
- LoSboccacc 11y agogot tricked even if I tested 1, 1, 1
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- tjohns 11y ago1, 1, 1 was actually the test case that clued me in that n^1, n^2, n^3 couldn't be the right solution.
- tikhonj 11y agoI mean, it also illustrates how training and systematic reasoning can improve these things. Whether explicitly or implicitly, I picked up certain skills and procedures for problem solving (from programming and math contests) that I now use by default. Trying a bunch of examples, coming up with a hypothesis, trying to disprove it, testing edge cases… This doesn't mean I always use these—at the very least, I have to explicitly jump into "problem solving" mode—but it means they can be useful. It's still a meaningful difference, and could very well apply to lots of things beyond this kind of puzzle.
- aptwebapps 11y agoExactly. I'm just as 'No' averse as the next person but the 'No' I'm averse to is the one where you make wrong assumptions and it comes back to haunt you afterwards.
- maxerickson 11y agoI wanted to answer "Probably" to the last question about having seen a similar question before. I don't specifically recall seeing one, but it is likely that I have.
- elevensies 11y agoThis is a good idea. I submitted the survey but here are my tests: http://imgur.com/hYKudyl http://imgur.com/hYKudyl FWIW, I've seen this kind of game before and I was expecting it to be something simple.
- wzdd 11y agoYour last question is "have you seen a test about confirmation bias before?" But the text of this test says that it's about "why no-one likes to be wrong", which means it's pretty obviously about confirmation bias (and therefore that the test-taker should be wary about just confirming their first intuition).
- pitchups 11y agoIntersting that the split of correct/wrong answers from the HN crowd is 78%/22% - the exact opposite of the general population : 22% / 78%! The HN community does think different :)
- snowwrestler 11y agoOr at least, the portion of the HN community who reported their results, reported that they think different. :-)
- JFoss117 11y agoYep, I'd guess this is a massive bias. The people who get it right are much more apt to report and show how smart they are
- philh 11y agoIf you're taking that from the article, I think you misread it. > Remarkably, 78 percent of people who have played this game so far have guessed the answer without first hearing a single no. Some of those 78% probably got it right, and some of the remainder would have got it wrong.
- jessaustin 11y agoSome of those 78% probably got it right... In what sense? How did they know that e.g. (1, 1, 1) didn't work, unless they tried it?
- philh 11y agoIn the sense that they guessed the rule correctly. The correct answer was the dominant hypothesis in my mind before I tried any sequences, and others here report the same.
- teahat 11y agoThe article doesn't actually specify that 78% guessed correctly or incorrectly, just that 78% guessed without ever entering an invalid sequence.
- nzealand 11y agoThis crowd needs one more question: How many "I know the answer, but can I find a flaw in their code" questions did you ask?
- jerf 11y agoThis is now the fourth time I've seen this exact rule. Another example: https://www.youtube.com/watch?v=vKA4w2O61Xo https://www.youtube.com/watch?v=vKA4w2O61Xo It's probably not "fair" to say I got it in zero... but I did. :) Now that the NYT has done it, this puzzle has probably attained enough popularity now that you really ought to change it up a bit now if you're going to run it yourself. Granted, the space of hypotheses as simple as "increasing/decreasing" is pretty small, but your ability to fool people with the first sample run is almost unbounded, so that helps.
- Verdex 11y agoSimilar story here. I got it in zero because this problem shows up at the early part of HPMOR. I suspect that the basic idea behind it is about right (people who insist on failures before committing to a theory will probably do "better"). But it seems to me that this test will be best at selecting people who've seen it before and can pretend they didn't (or even remember to ask negative questions when someone asks you to guess three numbers to get the job).
- MichaelDickens 11y agoYeah, I'd like to see other sorts of questions that you can only get right by looking for disconfirming evidence, but that don't have anything to do with choosing a sequence of numbers.
- stillsut 11y agoto be pedantic: "monotonically increasing" is incorrect. It should be "strictly increasing"
- deleted 11y ago[deleted]
- hammock 11y agoGuessed after just one "No" 3 9 27 yes (is it exponential series?) 4 16 64 yes (is it only odd numbers?) 5 7 9 yes (is it any numbers of the same parity?) 6 7 8 yes (is it any set of increasing numbers?) 6 7 6 no (just to confirm that it's x<y<z, and not something like x<=y<=z)
- Jugurtha 11y agoMy answer was: "The sequence is of an increasing real variable, where each subsequent value is greater than the preceding value. It's monotonically varying." When I clicked "I think I know it", nothing happened. I don't want to click their "I don't want to play; just tell me the answer". But it seems like the right answer. I can't answer your form question if it is the right anwer, since I haven't clicked on their link and don't know for a fact whether it is or not. Although I used the wrong term, it's strictly increasing.
- abandonliberty 11y agoThere's a selection bias - those of us who got it right are more likely to fill that out (: As a result we can't really rely on overall accuracy, but we can break it out by yes/no to account for the selection bias to get a profile for how a HN correct and incorrect differ.
- larkspur 11y agoIf you look at the raw answers, a lot of people who guessed the wrong answer according to: "What answer did you give?" said they got it right: Right? What answer did you give? Yes, double the previous number Yes, Each number much larger than the previous Yes, sequence must always increase by 1 Yes, Powers of 2 Yes, h = 2n, i = 2(n+1), j = 2(n+2) . n is an integer Yes, Powers of 2
- Myrmornis 11y agoIt would be relevant to include a histogram of #yes_answers - #no_answers in the summary, to test whether people are biased towards positive rather than negative tests. I think the raw data suggests that it does although I totally failed to create a histogram in google spreadsheets within 5 minutes.
- beatbrokedown 11y agoMy fuzzing went: 8 4 2 (no) 1 2 3 (yes) 1 1 1 (no) 1 100 123 (yes) 1.0 1.1 1.2 (yes) answer: incrementing numbers
- scuba_man_spiff 11y ago[ 456 , 42 , 48 , No ] [ 1 , 2 , 3 , Yes ] [ 3 , 2 , 1 , No ] [ 45 , 46 , 47 , Yes ] [ -1 , 999 ,1238978 , Yes ] [ 1+1 , 1+2 , 1+3 , Fail ] [ 7 , 654 , 653 , No ] [ 7 , 654 , 655 , Yes ] [ 1.1 , 1.2 , 1.3 , Yes ] [ 1 , e , pi , Fail ] [ 1 , e , 42 , Fail ] [ -1.1 , -1.0 , -0.9 , Yes ] [ 2i , 3i , 4i , Fail ] [ 1 , e , pi , Fail ] [ 0 , 1 , 2 , Yes ] [ -1 , 0 , 1 , Yes ] If anyone's tallying
- scuba_man_spiff 11y ago[ 456 , 42 , 48 , No ] [ 1 , 2 , 3 , Yes ] [ 3 , 2 , 1 , No ] [ 45 , 46 , 47 , Yes ] [ -1 , 999 ,1238978 , Yes ] [ 1+1 , 1+2 , 1+3 , Fail ] [ 7 , 654 , 653 , No ] [ 7 , 654 , 655 , Yes ] [ 1.1 , 1.2 , 1.3 , Yes ] [ 1 , e , pi , Fail ] [ 1 , e , 42 , Fail ] [ -1.1 , -1.0 , -0.9 , Yes ] [ 2i , 3i , 4i , Fail ] [ 1 , e , pi , Fail ] [ 0 , 1 , 2 , Yes ] [ -1 , 0 , 1 , Yes ] If anyone's tallying
- eru 11y agoThanks for setting up the survey! Could you perhaps move to a bar chart instead of pie charts?
- ddlatham 11y agoDon't think that's an option for Google docs - but feel free to take the data and share one.