20 ms·
We think this cool study we found is flawed. Help us reproduce it
- Klais 5y agoI see how the need for this profession is just increasing year after year, thus I see bright potential in this direction. When I was required to do my academic assignment, I looked for online buy assigmnents UK https://www.essaywritinglab.co.uk/buy-assignments-online/ https://www.essaywritinglab.co.uk/buy-assignments-online/ because my goal is to demonstrate that I am knowledgeable about the subject and that I will be an excellent nurse. The firm did an excellent job, and I gained valuable experience working with actual professionals. My teacher praised the quality of my work, and I was able to keep my position of power.
- coward123 5y agoVery early on in their explanation of the experiment, I found myself asking "What is random?" I'm sure one of you smart mathematicians has a learned perspective on this, but to my thinking, a whole lot of hairless apes sitting behind their keyboards randomly beating patterns for entertainment is pretty random. So to say that one hairless ape is more random than another in their keyboard pounding is pretty subjective. I guess I'd further posit that a person over 60 might not really give a shit much so their keyboard pounding might be less enthusiastic compared to the 20-something. Is that really measuring randomness or something more like physical dexterity in mouse movement and keyboard pounding?
- jesprenj 5y agoWell, the buttons didn't do anything on my phone. Okay, it opened in Fennec. A great example!
- devin 5y agoI’m over 60 apparently.
- deleted 5y ago[deleted]
- Closi 5y ago> As for our initial idea to make an age-guessing game, we have guessed right 51% of the time. Pretty much what we had expected . They guessed I was 'Under 60', which I am, but over 50% of people fit into the category of under 60... so the fact they can guess this with only 50% of accuracy doesn't really feel right?
- floodle 5y agoI think that's their point. Since there isn't an actual correlation, their guess is basically 50/50.
- Closi 5y agoBut if their guess is 50/50, that is from a subset that is mostly under-60's, which presumably means that they have evidence for the opposite? (i.e. there is no statement that the 51% figure is age adjusted) I'm probably just being silly here...
- deleted 5y ago[deleted]
- Freeboots 5y agoI think they're saying over or under 60 is completely arbitrary when judging only from 'random' answers. It could be over or under a month, or 1000 years, the result would still be 50/50 because the ability to create a sequence that appears random has no link to age.
- runnerup 5y agoGP is saying it's even more than 50/50. Less than half of people are >60. A more extreme example would be if it guessed: "You are younger than 99 years old."
- Closi 5y agoYeah, 'We tried to work out the impact of age on randomness, and guessed that people were younger than 99 years old 50% of the time!"
- rasse 5y agoI fail to grasp the idea of "complexity" or "randomness" in random sequences, if we're not talking about distribution tests.
- mungoman2 5y agoHm I'm curious about the complexity measurement for the coin tosses. How does it work? There is suspicious clustering in the plot, at complexity slightly lower and higher than -1. Complexity -1 is also more sparse than seems reasonable from the shape/density of the cloud.
- floodle 5y agoI'm also curious about this. Randomness doesn't actually look very random. There's a surprising number of repeated characters.
- deleted 5y ago[deleted]
- phorkyas82 5y agoI think, that's a common misconception about randomness: It doesn't mean the number has to change with every roll of dice. On the contrary the longer the sequence large clusters of the same number will occur. It's maybe not so counter-intuitive as the Birthday paradox, but maybe already showing how bad our intuition is at grasping randomness. Initially I had hoped about research on that, which is probably very hard. - Or not? Couldn't you look at the n-gram distributions of the data and look how much that is "random", i.e. are people avoiding clusters harder than they should?
- andi999 5y agoIf you cannot see the trend with looking at a diagram, and only algorithm then dont trust the algorithm.
- cormacrelf 5y agoCan anyone describe in a nutshell how measures of randomness work? Every sequence generated by independent random choices is equally likely, right?
- leksak 5y agoThey are equally likely, yes. But, the questions here pertain to whether or not an other person would perceive your sequence as random. It wasn't too long ago there was an article on HN (can't find it now) describing what feels random to people. Essentially, a sequence feels more random the harder it is to explain. So, HHHHHHHH doesn't feel random because it's summarized as 8 H's and HTHTHTHT doesn't feel random either because it is HT repeating. Strings that are only really communicable by repeating the string verbatim feel the most random.
- rmetzler 5y agoBut if this is about perceived randomness shouldn’t people also guess which sequence was created by a human vs sequences created by some algorithm? I understand that to score randomness you probably would create n-grams of the characters and look if these would be equally distributed. But for such short sequences it feels hard to do. Maybe a statistician can explain this? For me, using my right thumb on a smartphone seems enough to skew the randomness. Just by doing it again with my index finger (after writing most of this comment), I raised it from being “more random” than 18% to 84%.
- cormacrelf 5y ago'Perceived as random' seems like a pretty junk measure of other humans' efforts at producing randomness. Garbage in, garbage out. Surely analysing this tells you absolutely nothing about anything? I would understand if the actual measurement is not perceptual but mechanised, i.e. "how small can we compress this stream of random choices using our best known compression methods" or something. (But then a stream of 10 symbols is surely not enough to show you the humans.)
- 5y ago
- searchableguy 5y ago> Do you think the questionable response looks genuinely random and satisfies the instructions? We don’t. Could someone explain why putting sequences of same choice is invalid? Sure, a probability of having many tails continously is low but how is it not random?
- jat850 5y agoThis is a key point in the article. The researchers (original) did not want to throw out the data that doesn't "seem random", this group argues in favour of doing so.
- searchableguy 5y agoYeah, I read that part and I can't make sense of it. How are they deciding what is reasonably random?
- jat850 5y agoI felt the same way, for what it's worth.
- AnnikaL 5y ago> so that if another person is shown your sequence of digits from 1 to 6, he/she should not be able to tell whether these numbers were produced by a real die or just "made up" by somebody This explanation leads me to think that the decision of what is random in the study is based on human perceptions of randomness, not actual statistical randomness. Although any sequence is equally (un)likely to be rolled, 1111111111 would stand out from the other sequences much more than 3156263441.
- jonas21 5y agoNo, they're just saying that no person who was following the instructions would produce a sequence of all heads or all tails as a "random" sequence, so they're throwing out those two specific sequences.
- adastra22 5y ago
- IshKebab 5y agoVery cool website. Kind of "someone is wrong on the internet" crazy levels of effort! Really I think you can see that there's no trend just by looking at the graph. Always be suspicious of graphs that look like they have no trend by eye but have a solid trend line superimposed. Would be nice if they defined "complexity" somewhere. I think the sequence lengths are too short to distinguish true random number generators from poor random number generators. In other words what kind of graph would you see if you used a real coin? Although I guess if you sample over enough people then it doesn't matter.
- jonplackett 5y agoMy idea of what random actually looks like has been affected a lot by generating random numbers with a computer. They just don’t actually look that random. I read an anecdote about the iPod shuffle (hey kids - it was a music player with no screen so you could not choose songs directly) - they initially set it to be genuinely random in the way it chose the next song - people didn’t like it. It didn’t _feel_ random to pick a song you only just listened to again. So they had to make an algorithm that was sort-of-random but with some constraints to make it feel how we expect randomness to be.
- tomsmeding 5y agoI mean, that makes sense. What I want when putting a music player to "shuffle" is not "give me something unpredictable" -- that's fundamentally what randomness means. What I want is "give me something new". Something new is not something random, it's something _different_ from before, if reasonably possible.
- sjmm1989 5y agoWhich to a machine may as well be the same thing in either phrasing. You want something different from what you just listened to. To it, anything not 'that song' is different and 'new' potentially if also not 'just listened to' within a certain set amount of songs. Even without that certain set of songs being logged and considered; any picking of a different song from the last is verifiably random. Think of it all like a deck of cards. Shuffle is apt in that sense. You don't expect to see double aces each time you pick through the shuffled deck of cards, but sometimes you do. Sometimes, you also find double jacks, queens and kings; in a row. Sometimes you don't. That deck could be shuffled by the worlds best trick shufflers. Still gonna get doubles now and then. True Randomness is not really technically possible. At least, not with our current technologies available; and we have a lot of aces up our sleeves. The best we can manage for randomness right now, is creating random strings of numbers to serve as the seed for new randomness. At least, if I understand correctly. If I do, then this is why cryptography is so damn important for us in the computational side of things. Network Security requires randomness.
- tomsmeding 5y agohttps://xkcd.com/1725/ https://xkcd.com/1725/ . Those regression lines are absolutely laughable. Point cloud has line going through it, and beforehand I have no clue where the line will be (other than that it will go through the center, presumably). Even if there is a statistically significant effect, its meaning will be a rounding error.
- thombles 5y agoI understood my task as convincing another _human_ that it was randomly generated. Since N was low for each of the tasks, I was deliberate about sometimes having repeated values, and not ensuring that every option was picked an equal number of times, since that looks suspiciously algorithmic. Apparently I'm over 60.
- afro88 5y agoSame reasoning here, and same result.
- guywhocodes 5y agoMy age was guessed as over 60 as a result of this, which I guess is in line with their assumptions but maybe not for the right reason
- thomasguide 5y agoSame here.
- telesilla 5y agoI got the same answer as you, over 60, the first time as I was also very deliberate then went back and did it again like a 3 year old, jabbed anywhere and got a higher random result. Maybe there is something to the study?
- CipherThrowaway 5y agoPicking evenly seems to consistently produce higher "randomness" scores than picking unevenly or using an RNG. I wonder how this algorithm would rank random sequences vs shuffled linear sequences. The fundamental issue with a randomness metric for sequences is that an idealized independent generator will under-perform vs a constrained generator that excludes low scoring sequences.
- indecisive_user 5y agoAs I understood it, their entire point is that younger people are not better at generating random sequences than older people, so guessing someone's age based on their complexity (randomness) score is completely unreliable. Towards the bottom of the page they said they've only guessed age correctly 51% of the time, which lines up with there being no correlation between age and ability to generate random sequences
- strstr 5y agoUsed dice, and got very low "randomness" both times (8% and 25%). Hand-rolling got me 70%. Something feels off.
- quanto 5y agoBeautifully crafted web page. For any experimental science, the integrity of experiment (thus reliability of data) is important. For experiments with human subjects, the question is whether the subjects answered the questions in good faith. A sequence like 'HHHHHHHHHH' for the coin toss experiment looks like an answer in bad faith; it is mechanically easy to keep pressing the same button, and a subject is unlikely to think that such a sequence is a likely random sequence. Therefore, the replicating authors are fully justified to eliminate those poor faith answers. The original paper authors' claim that 'HHHHHHHHHH' is as equally probabilistic as any other sequence is irrelevant. Edited for clarity.
- foerbert 5y agoThis is not a sound approach. You're declaring what humans think random is first, and then throwing out any data that doesn't match your declaration. There is no way to learn anything from this. I also think 'HHHHHHHHHH' is unlikely to be a good faith response, but if the goal is to actually learn anything instead of merely reinforcing my prior beliefs, it doesn't matter. You need to find a way to design the experiment that discourages bad faith answers or let's you judge them objectively. Alternatively if you have some outside knowledge about the 'shape' of bad faith answers for your kind of experiment, you may be able to use that to properly adjust your data. But 'nah I don't think so' isn't an acceptable reason to throw out data. It's especially egregious to do so when the data is answers that are, at a bare minimum, technically correct.
- lapsis_beeftech 5y agoIt seems to me that if we reject a subset of experimental samples because they look like bad data (e.g. extreme outlier caused by sensor malfunction) we are still keeping all the bad data we are unable to recognize as such (e.g. sensor malfunctions producing less extreme data), which introduces a bias.
- alex_smart 5y agoYou don't have to view it as "throwing out the data". You can just think of it as an alternative explanation for the data. Original hypothesis: Old people are worse at giving random responses. Alternative hypothesis: Old people are more likely give bad faith responses. This review is suggesting the AH is equally good at explaining the data as the OH.
- sjmm1989 5y ago> As for our initial idea to make an age-guessing game, we have guessed right 51% of the time. Pretty much what we had expected . Yeah... you thought I was 60. Seems from the comments this is a common thing. You might want to check your algorithms. But then again, you do say in the end of the results that you need more 60+ year olds to help make this more accurate. Also, a bone to pick. You claim that people get less random as they get older over 25, with 25 being the peak. I would wonder if maybe that has some correlation with the brain finally fully developing from adolescents into true/full adulthood. (Remember folks, we do call people 'young adults' for a while in their 20's.) Also, while you make that claim about people as they get older, I still managed to get a coin flip result that was 13% more random than others at my age group of 33. Or something like that. I forget how it was worded exactly off hand this moment without going and checking again in my history. My point here is this. If randomness declines with age past 25, but my score at age 33 is 13% more random than others in the same age range; then is it truly declining for everyone equally or is it just some people more rapidly than others? I think this maybe correlates potentially with the findings of the trend disappearing once the non-random data is removed. (The all heads/all tails results.) Anyways. With all this said, I do agree you need some more participants above the age of 60. Have you considered using facebook at all?
- Aethylia 5y agoDid we read the same article? They aren't claiming those things at all. Those are the claims of the original study, which are being disputed by this attempt at reproduction. The writers suspect those claims to be false due to the choice of the original study to not remove likely intentionally non-random data. I believe the 13% stat you saw is that your score had a higher random score than than 13% of other participants, so not very random.
- Freeboots 5y agoI think you've misread the article.
- bumblebritches5 5y ago
- isitmadeofglass 5y ago> As for our initial idea to make an age-guessing game, we have guessed right 51% of the time. Pretty much what we had expected . I don’t know what the “guess” was for others. But for me it guessed “are you under 60?” If that’s what it’s doing guessing above or below 60, then I think it’s amazing they, only getting 51% right. I would expect that a strategy of ignoring data completely and always guessing under 60 would be significantly better.
- bandushrew 5y agoOver 60 according to something, but I’m not. Hypothesis: the older a person is, the more likely they are to understand that HHHHHHHHH is a statistically likely outcome from true randomness.
- seqizz 5y agoHeh, I also had the same result. Remembered the comic: https://www.incibe-cert.es/sites/default/files/blog/comprobando-aleatoriedad/dilbert.png https://www.incibe-cert.es/sites/default/files/blog/comproba...
- sacrosancty 5y agoI suspected it was the older a person is, the less of an eff they give about psychology researchers' questions :P
- stingraycharles 5y agoVery interesting, and I think this part sums up the crux: “The researchers believe that you can only analyze the raw responses because, statistically, any sequence is equally likely to occur, so where do you draw the line?” I’d say that, as it is a psychological study, making claims about a human behavior, treating humans as pure random number generators without considering _intent_ is a mistake. It is entirely possible that older people fill in more “questionable” responses because they can’t be bothered with the study, and that this causes the “decline” in ability as people age. But we don’t know for sure, because it was never investigated. Thus, the biggest problem is the original study not even bringing this into light, even though it appears the original authors were aware of it.
- Hendrikto 5y agoBut how can you go from a random sequence to the intent of a subject? I don’t think you really can.
- Shaanie 5y agoOne thing many surveys/studies do is to include "trap" questions (I'm sure there's a real name for 'em) which disqualifies any participants that answer them incorrectly.
- stingraycharles 5y agoI think they’re called “control questions”, as their purpose isn’t related to the study, but rather to control for BS answers.
- shusaku 5y agoI think they are being overly kind. The conclusion itself shouldn’t be sensitive to just removing the all H if all T answers. Since their trend disappears from removing just those answers, were only left with the far more mundane “older people are more likely to write all Hs or Ts”. The true conclusion was hidden by the averaging that goes on when you make a best fit line.
- Freeboots 5y agoDoes that mean that there is a link between age and participants who "misunderstood the instructions, or intentionally subverted the experiment"?
- ad404b8a372f2b9 5y agoThe more surprising part to me was that quote about the 2015 study that tried to reproduce 100 studies and managed to replicate 39. That's honestly way better than I expected for psychology. Here is the paper: https://www.nature.com/articles/nature.2015.18248 https://www.nature.com/articles/nature.2015.18248
- hemreldop 5y ago
- lcabral 5y agoI suspect they now have my credit card number, expiration and even the pin number
- plorntus 5y agoI feel like a lot of the comments here are written after only taking the test and many are not reading the rest of the article. The authors of the website are stating that they believe the study is wrong. The below/above 60 answer is showing you it’s incorrect half of the time along with data backing up the claim.
- gryson 5y agoYes, hilarious comments in this thread. Please at least skim the article.
- Brybry 5y agoThe end of the article was hilarious. > we decided to reduce our experiment to three tasks because of attention spans (not yours, it is exceptional if you are reading this).
- Closi 5y agoBut their data doesn't make sense to be personally... Only 5% of their dataset is above the age of 60, making their claim that they are getting 50% of their guesses wrong seem like they are calculating it wrong. Surely their cut-off should be at the 95th percentile of the data? They shouldn't be guessing 'under 60' the same proportion of times as 'over 60', because their population is mostly under 60.
- prolyx 5y agoYeah, they would be far better off just guessing under 60 every time...
- sohdas 5y agoAgain though, they are arguing that there is no correlation between randomness and age. This was just a demonstration that when they use randomness to predict age, the results are wrong 50% of the time-- which is precisely in accordance with their hypothesis
- doodlebugging 5y agoNice study. It picked me as under 60 though I am actually over. Does it give any more granular age guesses or is it stuck on over 25, under 60, over 60? I may retake this in the morning to see if anything changes for me or for the test.
- keithnz 5y agoI think the way the test is setup tends to create bias as it relies on mouse clicking and hence how people click on things is going to be a factor.
- peter_retief 5y agoI am over 60 and it was correct in my case. Makes me feel really old now :)
- staindk 5y agoI'm not even halfway there and it thought I was over 60. Even scientific studies are joking about my age now!
- peter_retief 5y agoEventually every single combination will be true in a random set, the joke is on them not you!
- kzrdude 5y agoI was trying to find a mental way to do a true coin toss. Anyone has ideas for how to truly dig into some randomness? Maybe most would think it's impossible, but aren't we at least better placed to do this better than deterministic machines (or maybe we are not - the true free will debate :)?
- pyk 5y agoIf you have two people, you can have person A ask person B to “pick a random number”, and use heads for odd, tails for even. Don’t tell person B why you’re asking and my guess is you’re relatively random heads/tails. No studies that I am aware of to back this up, so people could be biased towards odd/even, but a bias correction could correct that too.
- blackboxlogic 5y agoLook around the room for objects in sight. For each object, take its common name and count how many of the letters are "odd" letters acegikmoqsuwy, then mod 2. "Window" -> "wiow" -> 4 -> 0. Each word yields a single bit of very slow, pretty good entropy. Don't do this in the same room twice.
- ryukoposting 5y agoI wonder how effective it would be simply to count vowels mod 2. Much faster to calculate, at least for me.
- 5y ago
- mikewarot 5y agoIt seems to me if you correct/don't correct for people who didn't follow the directions, what you've actually come up with is a graph of how well people follow directions, which peaks at age 30. I don't feel bad for being 57% random at age 58. 8) I guess my ability to follow directions is about to nosedive.
- bumblebritches5 5y ago
- altvali 5y agoThe reason the trend line drops over 60 is that there are fewer people over 60 taking the test and thus, the questionable responses have a bigger influence. If you sample less from that cohort, it will be less likely that your sample is representative.
- hobofan 5y agoYes, while I like the article, it kind of distorts the evaluation in the original paper, where they don't only show the trend line but also it's 95% confidence interval (which very obviously significantly widens with age).
- mootzville 5y agoI'd imagine it's hard to prove randomness. Yes, HHHHHHHHHH can be chosen randomly as equally as HHTHTHTTTH. I don't remember seeing anything in the article speaking to the statistical significance of the data...though, I'm not a scientist, so maybe it's inferred from the discussion of the results.
- samuelfekete 5y agoSince this is a study about the predictability of humans, perhaps “less random” should be defined by how similar or predictable a sequence is based on the dataset of human inputs.
- EdwardDiego 5y agoConfounding factor - I'm using a trackpad, so I'm more likely to click things next to the things I just clicked. My randomness is mitigated by my laziness.
- icambron 5y agoI agree with the article that the study is flawed in its unwillingness to exclude the all-H and all-T answers. But I’ll go further: the original study is just silly. “Make a sequence that looks random” is sort of a nonsensical ask. Looks random to whom? To our algorithm, is what they meant. There’s no such thing as a randomness test that can look at a sequence and decide “is it random?”, so this algo measures something else and uses it as a proxy for “looking” random to…the study authors, I guess? There’s no ground truth here; it’s chasing a ghost. I don’t know what information we could even hypothetically gain from knowing older people score lower according to this algo—-perhaps that the paper’s authors are younger than 60 and thus picked a different “randomness-looking” algo than they would if they were older? At best that older people have an equally incorrect but qualitatively different idea of “random looking”? Of course we did not learn that; we only learned that older people pick all-H or all-T more. But my point is that there wasn’t really anything interesting at stake anyway. (Edit: expanding a bit)
- bombcar 5y agoI suspect many of the “bad” responses are smart aleks saying “11111111” is just as likely a “62536164”.
- op00to 5y agoIs 11111111 less likely than another sequence?
- bombcar 5y agoIt’s just as likely (or unlikely) as any particular sequence.
- prolyx 5y agoPrecisely. A difference only arises when comparing sets of strings. In our example, the number of elements in the set uniform strings aaaa...a of length L is equal to the number of distinct symbols N, while the number of district strings of length L is N^L. So if you ask "will the string be uniform?" (which is the only reason you thought up aaaa..a in the first place) you find the probability is exponentially small for long strings: N^(1-L). Really if you tend to choose a string for any particular reason other than chance, it can be exploited as long as the reason can be guessed (e.g. the dice rolls 31415 write out the digits of pi.).
- kuhewa 5y agoI haven't looked at the details of the CTM method for calculating complexity (because the math symbols didn't render on mobile) but from what I can tell in the two state coin flip case more state switches in the sequence is more complexity? That's not 'more random'.
- neals 5y agoWell thanks, now I feel old..
- denton-scratch 5y ago> so that if another person is shown your sequence of digits from 1 to 6, he/she should not be able to tell whether these numbers were produced by a real die or just “made up” by somebody. That instruction is a flaw in the experiment. It's always impossible to tell, for any given sequence, whether it was produced by a fair die. There's nothing an experimental subject can do to make the impossible more impossible. > the kind of sequence you’d get if you really rolled a die Well, there is no such sequence. The instruction is incoherent. You could take it as meaning "Construct a sequence that you think will convince others that it came from a random source". That would be coherent. And then it would be legitimate to eliminate responses that were all-heads ("clearly didn't even try"). But then what are you measuring? The comparative understanding of older and youger people concerning random sources, or the Gambler's Paradox? Their comparative expertise in human psychology? Their comparative willingness to move the mouse-pointer over the screen?
- simias 5y ago>It's always impossible to tell, for any given sequence, whether it was produced by a fair die. If the sequence is long enough you can model how likely it is to have been produced by a fair die. Are all numbers equally distributed? Are some numbers more likely to follow or not follow other numbers? Are some patterns repeating? Of course any sequence can be produced by a fair die, but you can still create some objective metric that will tell you how truly random a sequence is, and the longer it is the more accurate it will be. It's what tests like Diehard are about after all. Can you roll a fair die a thousand times and only get 6s? Well yes of course. It'll never happen though.
- deleted 5y ago[deleted]
- marcosdumay 5y agoAny single sequence of numbers has the exact same probability of being produced by a fair die (that's how the definition of "fair" goes). The probability of getting all 6 is the same of any other one you get.
- masterjay 5y ago
- Werewolf255 5y agoThe way I read "so that another human could not tell" it was not random is that the question is truly asking about luck and probability. I have studied gambling enough to know that runs are the norm and wins are not evenly distributed across the space, I.e the gambler's fallacy of thinking one is 'owed' a certain outcome after it doesn't occur for some time period. I understand that randomness is not uniform distribution and feel like people who are in similar situations are always going to skew results in some way.
- ahwvd37js 5y agoFrom the original paper [1]. > Paradoxically, participants with a scientific background may perform worse at producing random sequences, thanks to a common belief among them that the occurrence of any string is as statistically likely as any other (a bias deriving from their knowledge of classical probability theory), which further justifies controlling for Field of education, simplified as humanities v. science. [1] https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005408#pcbi.1005408.ref052 https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...
- natly 5y agoHow in the world did this get through peer review
- edflsafoiewq 5y ago> Finally, the variable Paranormal Belief was included as it has been related to RIG performance in previous studies.
- graderjs 5y agoWon't it to distort the results to be front loading people with all of this information before doing the test like they're doing on the website?
- nindalf 5y agoI’m devastated to find out I’m older than my parents. I am frantically calling my mother to find out how she kept this from me. But seriously, I love any initiative to replicate research. Good on pudding.cool to take the effort.
- elif 5y agoI believe this is conflating distribution with randomness. Having played too many games and rolled too many real and pseudorandom dice, i know streaks are the rule not the exception, and that completely missed rolls likewise expectable. Using this model of randomness, i tried to create sequences that matched. The result is it says i have 60 year old brain. I did the prompt again, but simply /ensured there were no missed rolls/ and i have an under 60 brain? In reality, i believe the odds of rolling all 6 sides using 10 dice rolls are relatively low... but in order to be "random" enough that seems like a prerequisite??
- want2takearide 5y agoI had the same issue. I generated random numbers and still got over 60 years old. I had to dive deep into it, and it turns out that the original paper is shit. They ask participants to make sequences that are as random as possible but they use Kolmogorov complexity as a measure, which (surprise) doesn't actually measure randomness but complexity. Random distributions tend to have lower complexity than what humans generate; see [1] Fig. 4. Here are the complexity scores for different sequences if you want to lose more sleep over this: [2]. At least the authors were nice enough to be open about their data, so props I guess. [1]: https://link.springer.com/content/pdf/10.3758/s13428-015-0574-3.pdf https://link.springer.com/content/pdf/10.3758/s13428-015-057... [2]: https://github.com/algorithmicnaturelab/HumanBehavioralComplexity/blob/master/lifespan.csv https://github.com/algorithmicnaturelab/HumanBehavioralCompl...
- m3047 5y agoMaybe the null hypothesis needs to be that older people have more experience of "random" and therefore expect or tolerate more variance before rejecting "chance". But they say that the effect is relatively small, or that their age guessing is "not as good as we had expected". I'd be prone to misinterpreting the question. If somebody asked me if the sequence "HTHTHTHT" is more or less likely to occur than “HTHHTHTT” I'd be confused. Ok, are they rejecting that they're equally likely out of hand, and if so why? At least subconsciously. If forced to suggest one or the other I'd offer the latter even though I know it's not mathematically correct. Additionally most of the problems I deal with have to do with probability within a continuous run and in a run of 1000 heads/tails and some quick and dirty Monte Carlo I get: m3047@sophia:~/temp> ./sequences.py | grep -E 'HTHTHTHT|HTHHTHTT' 11 HTHTHTHT 6 HTHHTHTT m3047@sophia:~/temp> ./sequences.py | grep -E 'HTHTHTHT|HTHHTHTT' 8 HTHTHTHT 3 HTHHTHTT m3047@sophia:~/temp> ./sequences.py | grep -E 'HTHTHTHT|HTHHTHTT' 5 HTHTHTHT 4 HTHHTHTT So I'd be wrong in practice at least with whatever PRNG is being utilized or maybe I'm just lucky. Always good to test and ask questions.
- somehnacct3757 5y agoIf you're forced to pick random numbers between 1 and X in your head, pick instead from a wider range of numbers and then modulo X. Your brain will legitimately have no idea what number you're picking. e.g. for a range 1-6, pick from 100-250 instead and modulo 6 plus 1. There are of course brand new biases at play (is your new range cleanly divisible by X?) But it's enough to tamp down the original biases you're worried about
- photochemsyn 5y agoI think you'd be better off taking a small pinch of sand, salt, pepper etc., throwing that on a smooth surface, then counting all the grains and then modulo it (just have the number of grains be >> than the range as in your example). This would reduce a lot of inherent biases, although perhaps introduce others. Reminds me of the Buttered Toast Ig Noble Prize: https://gizmodo.com/an-experiment-that-solves-the-worlds-most-important-que-5867322 https://gizmodo.com/an-experiment-that-solves-the-worlds-mos...
- johnday 5y agoI'm not sure that counts as "in your head", though. If such things were allowed, just look at your watch and mod the seconds by 6 (or whatever).
- helloworld8 5y agoAt what point do we draw the line and say that these methods are sufficiently random? The task at hand is to come up with a sequence you perceive as random based on the numbers themselves, so adding layers like this seems to go against the concept of the experiment entirely. How is this different from opening up my JavaScript console and doing Math.random() several times?
- photochemsyn 5y agoI'm certainly no expert in randomness generators but the notion has been floating around for awhile that quantum indeterminancy is the best option. Here's a paper on it: https://www.nature.com/articles/s41598-021-95388-7 https://www.nature.com/articles/s41598-021-95388-7 > "The presented above examples clearly show that classic random number generators may be exposed to various attacks, or may have the so-called backdoors. This justifies the need to develop alternative technologies that could replace the classic generators on a large scale. The most promising, because they have a fundamental justification for the randomness in the formalism of quantum mechanics, are quantum random number generators."
- camel-cdr 5y agoGeorge Marsaglia suggested a very simple multiply with carry PRNG you can use in your head to generate better quality random numbers. [1] 1. Select an initial random number between 1 and 58. (This is accomplished by mentally collecting entropy from the environment, e.g., counting a group of objects of unknown quantity) 2. Multiply the least significant digit by 6, add the most significant digit, and use the result as the next state. 3. The second digit of the state is your generated pseudorandom number. 4. Goto 2. Sequence generated by 42: 42|16|37|45|34|27|44|28|50|05|03|18|49|58|53|23|20|02|12|13|19|55|35|33|21 2| 6| 7| 5| 4| 7| 4| 8| 0| 5| 3| 8| 9| 8| 3| 3| 0| 2| 2| 3| 9| 5| 5| 3| 1 I really like the idea of using this when playing for example rock paper scissors (I'd take the last digit mod 3 and reject any 0's, so the distribution isn't as biased). [1] https://groups.google.com/g/sci.math/c/6BIYd0cafQo/m/Ucipn_5T_TMJ https://groups.google.com/g/sci.math/c/6BIYd0cafQo/m/Ucipn_5...
- helloworld8 5y agoNot a scientist but perhaps one can help me understand, wouldn’t posting this to a forum with many people interested in software add potential for selection bias. I’m guessing that people like this are more inclined to understand what a “random” sequence might look like and therefore skew the results. If not, why not?
- Vivtek 5y agoIt appears that what falls off with age is a tolerance for hoop-jumping. Or perhaps the ability to follow instructions.
- jmclnx 5y agoInteresting and fun. I did the test myself and then I wrote a program on OpenBSD to read /dev/urandom and used that in the test. I got similar results to what I picked manually. So I do not know what they think about the test, I would have expected the utility I wrote to come up with different results than I did.
- apwheele 5y agoNot about randomness - but about curve fitting. It is actually very difficult to verify non-linear effects -- or maybe I should say the opposite. The statistical tools we use to identify non-linearities are prone to be very noisy, so even in the subset of data including the no-variation responses I am quite skeptical that downward increase is real, or just due to variance in the tails of the data. So a common social science finding is `* graded effects` where you might not by default expect them, and that is the main headline of the paper. I think plateau effects are reasonable in many situations, https://blogs.sas.com/content/iml/2020/12/14/segmented-regression-sas.html https://blogs.sas.com/content/iml/2020/12/14/segmented-regre..., but the noisy data itself often won't be able to clearly differentiate between different curve shapes.
- d33k4y 5y agoI'm surprised to see this comment not echoed by more people. Besides the inadequate sample size of older participants, my immediate response to the variance in the data was that I doubt the slope could be distinguished from zero.
- deleted 5y ago[deleted]
- bogwog 5y agoWouldn't a better prompt be something like > Click the button to roll the die > (plausible dice roll animation is played) > Try to guess the result > (buttons labeled 1-6) That way there's no confusion about how to interpret the instructions, it's just people trying to predict the results of an event they perceive to be random and fair. When it asked me to create a "good" outcome, I didn't really know what it was expecting. If you roll a die 1000 times, the results are probably going to be evenly distributed, but if you only roll it ~12 times, the results could really be anything. Anyone who has played a board game can probably remember a time when it felt like the dice where loaded. I felt like making the random numbers too perfect would make it obvious that they were picked by a human, so I purposefully picked a lot of duplicate numbers, and at the end it guessed that I was over 60 years old (I'm less than half of that...)
- ajmurmann 5y agoI'm not sure if guessing the result would be any better. Knowing anything about a fair dice, you might as well just stick to the same result for all tosses. However, I approached the exercise in the same way as you did, as an adversarial game. I tried to generate numbers that would trick a human who is tasked with filtering out the series as random. No results repeating stuck me as a tell that the series was done by a human, so I included a fair number of repetitions of the same result that's still statistically plausible. Not sure how to work around that in the instructions. Edit: I think the study might say more about what the player thinks about other's expectation of randomness than about the player's own understanding of randomness.
- efreak 5y agoSpeaking about fair dice, I have a vague memory of reading somewhere that as dice get used, dirt builds up in the pips, causing them to be slightly off-balance. I cannot find anything about this online at all, so it might well be complete nonsense made up by whoever told me. I have, however, found that dice are not random, and that 1 is a more likely result than anything else: https://www.insidescience.org/news/dice-rolls-are-not-completely-random https://www.insidescience.org/news/dice-rolls-are-not-comple...
- awinter-py 5y ago> We believe that there are obvious responses that are candidates for removal that make the data more true. The researchers believe that you can only analyze the raw responses ... filtering any data would be tampering with the results interestingly millikan was accused of removing observations from his oil drop study to reduce the error in the electron charge. (but from like 2% to 0.5%, both far better than previous uncertainties)
- JoeAltmaier 5y agoAll sequences are equally probably. But that's not the thing being asked to distinguish. It's whether a human being produced the sequence. Humans choose patterns. Distinguishing between 'random chance' and 'patterned response' is much, much easier statistically.
- eterevsky 5y agoI think you are conflating a little bit randomness and complexity. Scott Aaronson in "Quantum Computing since Democritus" cites a study that shows that when people are asked to generate a random sequence the sequence that they write looks _more complex_ than a truly random sequence. For example, when generating a sequence of coin tosses people would try to avoid long sequences of heads of tails, making the probability of those sequences lower than in a truly random sequence. I decided to try and test this using your game. I generated the results for all three tests using random.org, and ended up somewhere between 10th and 20th percentile of complexity.
- mproud 5y agoThe thing I’ve learned early on is, with truly random sets, is they are not evenly distributed. You will at times have clumps of repeating numbers.
- asn1parse 5y agostupid
- furyofantares 5y agoA mathematical model sometimes used for shuffling cards is that after the cut, as you joke N cards in one hand and M in the other, the next card to fall has N/(N+M) chance of calling from the hand with N, and M/(N+M) chance from the band with M. This is used to come up with results like 8 riffle shuffles being needed to randomize a deck of cards. I sometimes thing about how this model gives an extremely small but non-zero chance that you drop all the cards from the left hand before the right and end up with no shuffle at all. Or just one or two cards fall before the other hand. And I'm sure this has actually happened to people! But if you're counting your shuffles trying to get to 8 to call it randomized, nobody would count that shuffle. They'd redo it. I don't think anyone submitting HHHHHHHHHH or TTTTTTTTTT does so thinking they've generated a random sequence.
- OOPMan 5y agoThis reminded me of a video I watched recently about the most common form of cheating in Magic: The Gathering: stacking the deck. The video, by NuxTaku, did a great job of explaining how this method of randomizing the deck actually leads to a distribution which is uniform rather than random and which in turn is effectively cheating. I haven't played MTG since the early 2000s but the video struck a chord with me because I remembered using this method and not even thinking about it (in my defence, I was a teenager) as being a problem. So yeah, reading this article I was instantly like "er....yeah...don't ask people to make random sequences because they often have an incorrect idea of what that actually means".
- SemanticStrengh 5y agoPeople should test it with nootropics such as CDP-choline or noopept or even magnesium-threonate.
- chkas 5y agoThe random number generator of my computer is also over 60 years old - just tested it.
- gnicholas 5y ago> As for our initial idea to make an age-guessing game, we have guessed right 48% of the time. For me, the guess was just "under 60". This seems pretty coarse (I'm 40, so it was correct). Were they any more granular with other people? Or were they only 48% of the time, even with this very coarse prediction?
- gnicholas 5y agoThe data that this study will gather depends hugely on what internet forums it is spread around. I would guess that HNers would have very different results than a randomly-selected sample of the population. I don't see how a study that gathers data via internet virality can possibly credit or discredit a theory like this.
- deleted 5y ago[deleted]
- kingkawn 5y agoWhen our concept of random is so precise as to permit the exclusion of all results that do not conform with the image of randomness, is there anything random about it at all?
- the_af 5y agoI sort of knew before going in to the experiment the "trick" that most humans perceive randomness wrongly. So I knew coin flip sequence "hhhhttthhht" is more "random" than "hthhtththhth", which most people would choose. I guess my knowledge kind of ruins the experiment. Note I chose the first sequence because I know, but I have to battle with my intuition which urges me to pick the second (wrong) option.
- runlevel1 5y agoAfter doing it for real, I tried it a few more times with a cryptographically random shuffle. Interestingly, the computer got a worse score on average. Method: I started `irb`, and picked like so: require 'securerandom' # Coin flip 12.times { p [:heads, :tails].shuffle(random: SecureRandom)[0] } # Dice roll 10.times { p (1..6).to_a.shuffle(random: SecureRandom)[0] } # 10 dots 10.times { p (1..9).to_a.shuffle(random: SecureRandom)[0] } I disconnected from the internet before the scores were submitted so as not to taint the survey. (Ranking calculation happens offline based on a CSV of scores loaded into the browser early on.) Results: For: "Your answer got a higher random score than X% of people in the study." Me: 46%, 71%, 59%, 72% Computer: 32%, 20%, 47%, 31% My sample size is obviously too small for anything conclusive. However, I'll admit it makes me a little suspicious that something else is amiss.
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- m3kw9 5y agoA sequence of all heads is as random as any other sequences taken as a whole independently
- Groxx 5y agoI decided to chase "how are they measuring randomness?" down, and: - the study mentions using the "acss" R package in the Methods section: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005408 https://journals.plos.org/ploscompbiol/article?id=10.1371/jo... - acss's documentation directs you to another site for describing the metric: https://www.rdocumentation.org/packages/acss/versions/0.2-5/topics/acss https://www.rdocumentation.org/packages/acss/versions/0.2-5/... - that site links to some other articles in its bibliography at the bottom, and mentions using "algorithmic probability" to approximate complexity: https://complexitycalculator.com/methodology.html https://complexitycalculator.com/methodology.html - the first of which is this, which describes algorithmic probability over halting 2-symbol 4-state Turing machines by exhaustive execution: https://www.sciencedirect.com/science/article/abs/pii/S0096300311012367 https://www.sciencedirect.com/science/article/abs/pii/S00963... So assuming I followed all that correctly: Your strings are more complex if there are fewer Turing machines that produce it, which is then normalized to the average of all strings to become "randomness". Too few or too many ways to create a string means less random, within this category of simple and very small Turing machines (otherwise it'd be impractical to compute). I have no idea how broadly applicable that metric is though. It seems fairly niche to my complete amateur reading... but AFAICT all Kolmogorov complexity measures are extremely niche, as it's extremely sensitive to what the execution environment is. But this was still an interesting rabbit hole. Figured I'd share:)
- karxxm 5y agoBy posting this study here on hacker news, you will somehow get a bias in your study. Here, we are usually very technical people. We work with randomness, we studied it, implemented it or proved something about it. Many have studied it in university and have a different understanding of randomness than non-technical people. Therefore I would be carful when evaluating this study when a lot of the participants of this study came from here.