8 ms·

I agree with the article that the study is flawed in its unwillingness to exclude the all-H and all-T answers. But I’ll go further: the original study is just s

I agree with the article that the study is flawed in its unwillingness to exclude the all-H and all-T answers. But I’ll go further: the original study is just silly. “Make a sequence that looks random” is sort of a nonsensical ask. Looks random to whom? To our algorithm, is what they meant. There’s no such thing as a randomness test that can look at a sequence and decide “is it random?”, so this algo measures something else and uses it as a proxy for “looking” random to…the study authors, I guess? There’s no ground truth here; it’s chasing a ghost. I don’t know what information we could even hypothetically gain from knowing older people score lower according to this algo—-perhaps that the paper’s authors are younger than 60 and thus picked a different “randomness-looking” algo than they would if they were older? At best that older people have an equally incorrect but qualitatively different idea of “random looking”? Of course we did not learn that; we only learned that older people pick all-H or all-T more. But my point is that there wasn’t really anything interesting at stake anyway. (Edit: expanding a bit)

bombcar 4y ago
I suspect many of the “bad” responses are smart aleks saying “11111111” is just as likely a “62536164”.
- op00to 4y ago
  Is 11111111 less likely than another sequence?
  - bombcar 4y ago
    It’s just as likely (or unlikely) as any particular sequence.
    - prolyx 4y ago
      Precisely. A difference only arises when comparing sets of strings. In our example, the number of elements in the set uniform strings aaaa...a of length L is equal to the number of distinct symbols N, while the number of district strings of length L is N^L. So if you ask "will the string be uniform?" (which is the only reason you thought up aaaa..a in the first place) you find the probability is exponentially small for long strings: N^(1-L). Really if you tend to choose a string for any particular reason other than chance, it can be exploited as long as the reason can be guessed (e.g. the dice rolls 31415 write out the digits of pi.).
  - ricardobeat 4y ago
    The fact it’s equally likely as any other sequence means it should be very unlikely to appear in the study’s 3429 samples, and extremely unlikely to show up more than once.
  - moconnor 4y ago
    You can estimate the probability distribution that generates this sequence as 0: 0, 1: 1. This is as far from 0: 0.5, 1: 0.5 (a fair coin toss) as you can get. Comparing mean and std dev can be used to estimate the distance between two distributions. See also, statistical testing.
  - phire 4y ago
    Depends. If you know it has been generated by a valid random number generator, then no. But if you know there is a chance it came from something other than a valid random number generator, then you would have to classify sequences like 11111111 in a "highly suspicious" category.
  - johnday 4y ago
    This is the wrong question. Think instead about whether a sequence like 1111111 is more likely to be produced by a fair die or a loaded one.
  - photochemsyn 4y ago
    Helps to think of it in binary, 11111111 (bin) == 255 (dec). Also helps to define the space of possible outcomes, 00000000 -> 11111111. Then ask, is this a discrete event or a sequence of discrete events? I.e. if we have a roulette wheel with 256 slots, and throw a ball in, then the chance of the ball falling in any slot is 1/256. But what if we say, we're going to generate that binary sequence by 8 successive flips of a coin, and we are aiming for 11111111 specifically? Then we have to multiply eight times. (0.5)^8 == .00390625 == 1/256 Where it gets a bit tricky is if we ask people to place bets after each successive flip of the coin. For example, starting with no flips, ask players to bet on the likelihood of 8 heads in a row. Next round, bet on the likelihood of 7 heads in a row, knowing the first was a head, etc. What minimal odds should the house give after each flip in order to reliably turn a profit on this game? Does it matter how many players are at the table when it comes to calculating those odds?
  - Siira 4y ago
    It’s about the reference class. The class of allsame sequences has only a few members. The class of … random numbers all over ala 71833791 has a lot of members. So the probability of seeing the former class is tiny. Think of it in terms of coin tosses. Two heads is less likely than a head and a tail, because there are two sequences, 10, and 01, that map to the class of having one tail and one head. But there is only a single sequence of 00 in the class of all heads.
- prolyx 4y ago
  111111 is just as likely as any other number. However, in practice, humans are far more likely to think of 111111 than other numbers, so we exploit the difference between the probability a human guesses a number vs. the probability of rolling a number on a fair dice. "Did I just roll a sequence X or am I lying?" vs. "What is the probability I roll sequence X next?" are quite different questions, if by "lying" you mean you are not drawing the sequence from a fair, random source.
  - elif 4y ago
    111111 is just as likely as any other *SEQUENCE* of numbers. This is a little confusing because the "sequential" requirement is somewhat masked by the repeating sequence used by the example. however, the odds of rolling six 1's is 0.00002143347. considering the non-sequential set of rolls 625631 you have the odds of exactly two 6's at 0.201 and also one 2 5 3 and 1 each at 0.402 0.402^4 * 0.201 = 0.00524928641, or ~244 x more likely.
    - prolyx 4y ago
      There are many ways of slicing and dicing things so that you group different sequences, e.g. group permutations as the same as you have done. But by "number" I was referring to a sequence of symbols drawn from a uniform distribution, so order matters. (On the website, you also choose options sequentially, anyways.)
  - nnoitra 4y ago
    No, it is not. On average you expect equal number of 1s and 0s. All 1s won't be "equally as likely" .
    - pessimizer 4y ago
      All 1s isn't equally as likely as the entire class of outcomes that are not all 1s, but you can also say that about every other outcome (as long as we're talking about distinguishable dice or a sequence.)
moconnor 4y ago
“Sequence looks random” is not nonsensical. The authors say your sequence should be indistinguishable from e.g. 12 coin tosses. This would have a uniform distribution. One approach would be estimate the probability distribution from the input sequence and calculate the KL-divergence [1] of that to the uniform distribution. This gives one objective measure of randomness. There are many others! TL;DR: There are definitions of randomness that can be tested against. [1] https://en.m.wikipedia.org/wiki/Kullback–Leibler_divergence https://en.m.wikipedia.org/wiki/Kullback–Leibler_divergence
- moasda 4y ago
  Exactly, it's impotant to chose a definition of randomness first.
  - moconnor 4y ago
    The test specifies the random distribution very clearly in each case, though.
- icambron 4y ago
  You can define terms however you want, and thus devise any measure you want. But that’s not a definition of randomness I recognize. Perhaps entropy. Randomness is not a property of the result; it’s a property of the process used to generate it. Let’s imagine applying your measure to a whole bunch of sequences, each of length N, with the elements of each actually drawn from a uniform distribution. You measure the randomness of each one. You’ll get a range of DLK results, distributed from, in your interpretation, “very random” to “not that random”. All-H or mostly-H will come up sometimes, the distribution estimator will return a skewed result, and it will get a divergent “score”. But everything came from the same distribution. So we’re now measuring the output of an actually random process and saying “we’ll, it’s usually random but not quite always” In contrast, let’s try your method on a different population of sequences, where instead of pulling the sequence elements from a distribution, every sequence is a hardcoded copy of HTHTHT… That gives “perfectly random”, even though it was very far from a random process. That’s close to what the study authors are doing here, except with a different definition of “random looking”. It’s measuring a property of a sequence, but it isn’t whether it was randomly generated. We could debate about whether this is a good measure of “random looking” and there could be lots of alternatives with no objectively best. But that is my point: if I ask “make me something random-looking”, I am only asking “how closely does your measure of post-facto randomness match mine?” An actual random sequence would be, well, random.
oneoff786 4y ago
I agree with this. I think 1 1 2 3 4 looks more real than 1 2 3 4 5 In that a draw of 5 numbers is likely to include at least one number twice and one number zero times. But is that the goal? Is it communicable to the audience
kevinwang 4y ago
Why doesn't entropy or kolmigorov complexity qualify as a valid metric here? A random process will produce a highly entropic result or something with high kolmogorov complexity, whereas something biased will produce something with less (like HHHHHHHHH). Edit: saw this other comment ( https://news.ycombinator.com/item?id=31224738 https://news.ycombinator.com/item?id=31224738 ) about how kolmogorov complexity is a bad metric and am now second guessing myself
capitainenemo 4y ago
Agreed, I deliberately used the linux RNG, and it generated T T T T T T T H H H H H Probably why I got an "age" of 60.
- zeroonetwothree 4y ago
  The age predictor seems broken, my sequence was scored as very random (90th percentile) and it still picked 60 for me. Ironically based on eyeballing the chart it actually did predict my age within 5 years
baobabKoodaa 4y ago
> “Make a sequence that looks random” is sort of a nonsensical ask. Looks random to whom? To other people. They specified this in the study instructions. (Not defending the study, I think it's flawed too.)
- icambron 4y ago
  But they didn’t measure that. They’d have to measure that by showing a bunch of people each sequence and asking “is this random?” Instead, they threw a complexity formula at it, which measures some specific thing, but not “other people” Though perhaps there is a body of existing literature showing that their complexity estimator matches people’s assessment of “randomness”? If so, does it include people over 60?
  - baobabKoodaa 4y ago
    > But they didn’t measure that. That's correct. > They’d have to measure that by showing a bunch of people each sequence and asking “is this random?” Agreed! > Instead, they threw a complexity formula at it, which measures some specific thing, but not “other people” Yep, they botched the study. Nonetheless, the "ask" that you criticized was fine. It explained that the subject should craft a pattern which would appear random to other people, and imo there's nothing wrong with that ask. The mistakes came afterwards.
    - icambron 4y ago
      Alright, fair enough
JuettnerDistrib 4y ago
> we only learned that older people pick all-H or all-T more. I think we learn that older people are less willing to put up with silly instructions.
- icambron 4y ago
  That’s my interpretation too, fwiw. But I stuck with the narrower facts
rahimiali 4y ago
Agreed. This ends up being a test of one of a few things: 1. Whether people of different age groups agree that random sequences are more complex as defined by the study’s notion of complexity. 2. Whether people of groups have the physical ability to generate those sequences in a web browser. 3. Whether people of groups have the mental ability to generate those sequences. Of these, the study purports to measure 3, but it’s actually conflating all of these.

bombcar 4y ago

I suspect many of the “bad” responses are smart aleks saying “11111111” is just as likely a “62536164”.

op00to 4y ago
Is 11111111 less likely than another sequence?
- bombcar 4y ago
  It’s just as likely (or unlikely) as any particular sequence.
  - prolyx 4y ago
    Precisely. A difference only arises when comparing sets of strings. In our example, the number of elements in the set uniform strings aaaa...a of length L is equal to the number of distinct symbols N, while the number of district strings of length L is N^L. So if you ask "will the string be uniform?" (which is the only reason you thought up aaaa..a in the first place) you find the probability is exponentially small for long strings: N^(1-L). Really if you tend to choose a string for any particular reason other than chance, it can be exploited as long as the reason can be guessed (e.g. the dice rolls 31415 write out the digits of pi.).
- ricardobeat 4y ago
  The fact it’s equally likely as any other sequence means it should be very unlikely to appear in the study’s 3429 samples, and extremely unlikely to show up more than once.
- moconnor 4y ago
  You can estimate the probability distribution that generates this sequence as 0: 0, 1: 1. This is as far from 0: 0.5, 1: 0.5 (a fair coin toss) as you can get. Comparing mean and std dev can be used to estimate the distance between two distributions. See also, statistical testing.
- phire 4y ago
  Depends. If you know it has been generated by a valid random number generator, then no. But if you know there is a chance it came from something other than a valid random number generator, then you would have to classify sequences like 11111111 in a "highly suspicious" category.
- johnday 4y ago
  This is the wrong question. Think instead about whether a sequence like 1111111 is more likely to be produced by a fair die or a loaded one.
- photochemsyn 4y ago
  Helps to think of it in binary, 11111111 (bin) == 255 (dec). Also helps to define the space of possible outcomes, 00000000 -> 11111111. Then ask, is this a discrete event or a sequence of discrete events? I.e. if we have a roulette wheel with 256 slots, and throw a ball in, then the chance of the ball falling in any slot is 1/256. But what if we say, we're going to generate that binary sequence by 8 successive flips of a coin, and we are aiming for 11111111 specifically? Then we have to multiply eight times. (0.5)^8 == .00390625 == 1/256 Where it gets a bit tricky is if we ask people to place bets after each successive flip of the coin. For example, starting with no flips, ask players to bet on the likelihood of 8 heads in a row. Next round, bet on the likelihood of 7 heads in a row, knowing the first was a head, etc. What minimal odds should the house give after each flip in order to reliably turn a profit on this game? Does it matter how many players are at the table when it comes to calculating those odds?
- Siira 4y ago
  It’s about the reference class. The class of allsame sequences has only a few members. The class of … random numbers all over ala 71833791 has a lot of members. So the probability of seeing the former class is tiny. Think of it in terms of coin tosses. Two heads is less likely than a head and a tail, because there are two sequences, 10, and 01, that map to the class of having one tail and one head. But there is only a single sequence of 00 in the class of all heads.
prolyx 4y ago
111111 is just as likely as any other number. However, in practice, humans are far more likely to think of 111111 than other numbers, so we exploit the difference between the probability a human guesses a number vs. the probability of rolling a number on a fair dice. "Did I just roll a sequence X or am I lying?" vs. "What is the probability I roll sequence X next?" are quite different questions, if by "lying" you mean you are not drawing the sequence from a fair, random source.
- elif 4y ago
  111111 is just as likely as any other *SEQUENCE* of numbers. This is a little confusing because the "sequential" requirement is somewhat masked by the repeating sequence used by the example. however, the odds of rolling six 1's is 0.00002143347. considering the non-sequential set of rolls 625631 you have the odds of exactly two 6's at 0.201 and also one 2 5 3 and 1 each at 0.402 0.402^4 * 0.201 = 0.00524928641, or ~244 x more likely.
  - prolyx 4y ago
    There are many ways of slicing and dicing things so that you group different sequences, e.g. group permutations as the same as you have done. But by "number" I was referring to a sequence of symbols drawn from a uniform distribution, so order matters. (On the website, you also choose options sequentially, anyways.)
- nnoitra 4y ago
  No, it is not. On average you expect equal number of 1s and 0s. All 1s won't be "equally as likely" .
  - pessimizer 4y ago
    All 1s isn't equally as likely as the entire class of outcomes that are not all 1s, but you can also say that about every other outcome (as long as we're talking about distinguishable dice or a sequence.)

moconnor 4y ago

“Sequence looks random” is not nonsensical. The authors say your sequence should be indistinguishable from e.g. 12 coin tosses. This would have a uniform distribution. One approach would be estimate the probability distribution from the input sequence and calculate the KL-divergence [1] of that to the uniform distribution. This gives one objective measure of randomness. There are many others! TL;DR: There are definitions of randomness that can be tested against. [1] https://en.m.wikipedia.org/wiki/Kullback–Leibler_divergence https://en.m.wikipedia.org/wiki/Kullback–Leibler_divergence

moasda 4y ago
Exactly, it's impotant to chose a definition of randomness first.
- moconnor 4y ago
  The test specifies the random distribution very clearly in each case, though.
icambron 4y ago
You can define terms however you want, and thus devise any measure you want. But that’s not a definition of randomness I recognize. Perhaps entropy. Randomness is not a property of the result; it’s a property of the process used to generate it. Let’s imagine applying your measure to a whole bunch of sequences, each of length N, with the elements of each actually drawn from a uniform distribution. You measure the randomness of each one. You’ll get a range of DLK results, distributed from, in your interpretation, “very random” to “not that random”. All-H or mostly-H will come up sometimes, the distribution estimator will return a skewed result, and it will get a divergent “score”. But everything came from the same distribution. So we’re now measuring the output of an actually random process and saying “we’ll, it’s usually random but not quite always” In contrast, let’s try your method on a different population of sequences, where instead of pulling the sequence elements from a distribution, every sequence is a hardcoded copy of HTHTHT… That gives “perfectly random”, even though it was very far from a random process. That’s close to what the study authors are doing here, except with a different definition of “random looking”. It’s measuring a property of a sequence, but it isn’t whether it was randomly generated. We could debate about whether this is a good measure of “random looking” and there could be lots of alternatives with no objectively best. But that is my point: if I ask “make me something random-looking”, I am only asking “how closely does your measure of post-facto randomness match mine?” An actual random sequence would be, well, random.

oneoff786 4y ago

I agree with this. I think 1 1 2 3 4 looks more real than 1 2 3 4 5 In that a draw of 5 numbers is likely to include at least one number twice and one number zero times. But is that the goal? Is it communicable to the audience

kevinwang 4y ago

Why doesn't entropy or kolmigorov complexity qualify as a valid metric here? A random process will produce a highly entropic result or something with high kolmogorov complexity, whereas something biased will produce something with less (like HHHHHHHHH). Edit: saw this other comment ( https://news.ycombinator.com/item?id=31224738 https://news.ycombinator.com/item?id=31224738 ) about how kolmogorov complexity is a bad metric and am now second guessing myself

capitainenemo 4y ago

Agreed, I deliberately used the linux RNG, and it generated T T T T T T T H H H H H Probably why I got an "age" of 60.

zeroonetwothree 4y ago
The age predictor seems broken, my sequence was scored as very random (90th percentile) and it still picked 60 for me. Ironically based on eyeballing the chart it actually did predict my age within 5 years

baobabKoodaa 4y ago

> “Make a sequence that looks random” is sort of a nonsensical ask. Looks random to whom? To other people. They specified this in the study instructions. (Not defending the study, I think it's flawed too.)

icambron 4y ago
But they didn’t measure that. They’d have to measure that by showing a bunch of people each sequence and asking “is this random?” Instead, they threw a complexity formula at it, which measures some specific thing, but not “other people” Though perhaps there is a body of existing literature showing that their complexity estimator matches people’s assessment of “randomness”? If so, does it include people over 60?
- baobabKoodaa 4y ago
  > But they didn’t measure that. That's correct. > They’d have to measure that by showing a bunch of people each sequence and asking “is this random?” Agreed! > Instead, they threw a complexity formula at it, which measures some specific thing, but not “other people” Yep, they botched the study. Nonetheless, the "ask" that you criticized was fine. It explained that the subject should craft a pattern which would appear random to other people, and imo there's nothing wrong with that ask. The mistakes came afterwards.
  - icambron 4y ago
    Alright, fair enough

JuettnerDistrib 4y ago

> we only learned that older people pick all-H or all-T more. I think we learn that older people are less willing to put up with silly instructions.

icambron 4y ago
That’s my interpretation too, fwiw. But I stuck with the narrower facts

rahimiali 4y ago

Agreed. This ends up being a test of one of a few things: 1. Whether people of different age groups agree that random sequences are more complex as defined by the study’s notion of complexity. 2. Whether people of groups have the physical ability to generate those sequences in a web browser. 3. Whether people of groups have the mental ability to generate those sequences. Of these, the study purports to measure 3, but it’s actually conflating all of these.