4 ms·

> For example, recently psychologists discovered a simple way to test for creativity. The simple test is to come up with a list of 10 words as different as poss

> For example, recently psychologists discovered a simple way to test for creativity. The simple test is to come up with a list of 10 words as different as possible from each other. The simple test is to come up with a list of 10 words as different as possible from each other. After posting my word combinations and their scores people were replying with theirs, and then of course eventually GPT-3’s results were posted. According to the test, GPT-3 is far more creative than the average person, scoring in the 85-95th percentile. If you like, take that test and see if you can beat the machine. So could a line of Python. random.choices(words_in_english, k = 10) This isn't really testing creativity, but more the size of the list of concepts you can keep in your head at once and then select from. I can see how that correlates with creativity in humans, but for a machine, it is just leveraging its ability to store and process far more data. EDIT: I tested this. Maybe there is some confounding factor in my implementation, but random.choice seems to consistently beat most people too, around 90-95%. Site used: https://www.datcreativity.com/task https://www.datcreativity.com/task? Repo of code: https://github.com/MattGaiser/CreativeWordChoice/tree/master/ https://github.com/MattGaiser/CreativeWordChoice/tree/master... I just grabbed a random list of nouns, narrowed it by the params specified, and fed them in as required. You may need more than 10 as it rejects some, but in that case I just re-ran it and put in the first nouns to come up.

inglor_cz 5y ago
My naive take is that if you can keep 2N concepts in your head at once, the number of possible connections among them is much more than twice as big than with N concepts, so you have a much wider graph to analyze and possibly fish out interesting ideas from. (Unless N is 0, of course.)
- MattGaiser 5y ago
  I agree, but that is a two stage process. Sure the machine can beat us on concepts, but can it make meaningful connections?
pddpro 5y ago
Memory is much more correlated with creativity than one would think.
PeterisP 5y ago
Arguably that line of Python wouldn't do very good - you'd want choices that maximize the difference as opposed to be merely something random, this is harder than it appears; and the distribution of words is also uneven enough (both in relative frequency of word usage, and also in lexeme "density" w.r.t how many terms there are for some domains and how few for others) that pure random sampling would be not a good option.
- MattGaiser 5y ago
  It does pretty well on this (https://www.datcreativity.com/task https://www.datcreativity.com/task?), at least with the below code. Consistently getting 80-90%. https://github.com/MattGaiser/CreativeWordChoice/tree/master/ https://github.com/MattGaiser/CreativeWordChoice/tree/master... I just grabbed a random list of nouns, narrowed it by the params specified, and fed them in as required. You may need more than 10 as it rejects some, but in that case I just re-ran it and put in the first nouns to come up.
- wongarsu 5y ago
  Word frequencies follow a power law, so nearly all words are used very infrequently. So a randomly chosen word is likely to be very uncommon. Also there are many more specific concepts than general ones (because each general concept encompasses many specific ones). So a randomly chosen word is more likely to be specific than general. Random sampling is likely to give you unusual, highly specific words. I don't see how that wouldn't do very good. Sure, you could do better by mapping each word to a concept, then selecting ten random concepts and selecting a word for each concept. But why do that when you get 99% of the way there with just random choice. To show this experientially, I let python choose 10 random words from the first English word list I found. I got "epilithic, codirects, serologic, capsian, digestibility, yasna, samsara, supergun, hepatoptosis, kyurin". Sure, capsian and kyurin are somewhat related (names of places/people), as are serologic and hepatoptosis (both medical terms), but it's a pretty good spread.
  - rich_sasha 5y ago
    Not poster but I think the point is you could do better, not that this doesn’t work. You can imagine having some discrete optimisation maximising the dissimilarity of words, though that’s hard to do, in particular in a discrete space.
the_jeremy 5y ago
Also the test I took[0] scores this very poorly, by determining how often those words are in the same section of a text as each other, meaning choosing uncommon words (crenellations, gravedigger) will show much more creativity than common words (castle, grave) that are similar conceptually. [0]: https://www.datcreativity.com/task https://www.datcreativity.com/task?
quimplo 5y ago
> recently psychologists discovered a simple way to test for creativity Jesus. That "discovery" is https://www.datcreativity.com/faq#simple https://www.datcreativity.com/faq#simple It's not a "discovery", it's not an objective way to test for creativity, and throwing GPT-3 at a test designed to surface relative differences among humans is like saying a pair of dice are better than humans at supplying random numbers. Actually, what is the methodology for applying a language prediction model to a "come up with 10 'unrelated' words" prompt? Especially when the "test" relies on a very specific and game-able definition of "unrelated" that needs to be hidden from test-takers to provide even a semblance of "valid" results? The amount of BS being juggled among those who should presumably know better is astounding and depressing.
advael 5y ago
I got a dual-major in psychology in undergrad, and while it's not been an exceptionally useful degree, it did bequeath me with an enormous dose of skepticism toward psychometrics like this. The replication crisis solidified that skepticism Being able to choose less "related" words doesn't sound like it has much meaningful basis at all in what we care about when we say "creativity". This metric, as many others like it, was likely chosen and popularized because it is easy to measure
- MattGaiser 5y ago
  I can see it being strongly correlated, as that human has more items to draw from for figuring out solutions, but I don't see how it can be used to test computer creativity. Python's random surely isn't creative.
  - advael 5y ago
    That's the problem with metrics though. You can make anything sound plausible. Once a model has been presented to you, you can come up with an endless stream of just-so stories for why it measures what you were told it measures. Nonetheless, some metrics simply aren't that useful, and get adopted anyway because they produce a bunch of data. Psychometrics like this have an awful track record for standing up to more meaningful tests like "Can we drive any outcome we care about with this"
bryanrasmussen 5y ago
> For example, recently psychologists discovered a simple way to test for creativity. The simple test is to come up with a list of 10 words as different as possible from each other. When Robert Frost was the poet laureate of the U.S some news magazine (I think NewsWeek, although it might have been Time) did a creativity test on him, using inkblots iirc, and he scored particularly low. Frost was irate afterwards and remarked something to the affect that to be creative required that you not see things in the inkblot, that is to say to not try to see things in distractions, that he had spent years training himself not to see things in meaningless shapes. I think that there is quite a lot of truth in this, also if someone wanted to give me a simple test to come up with a list of 10 words as different as possible from each other I would probably get verbally abusive - at least call them an idiot.
jimbokun 5y ago
I would just like to say, the insight to try just a random choice of words and see how well that satisfies the criteria demonstrates a kind of creativity that isn't captured by having people generate a list of "unrelated" words from memory.
ashtonkem 5y ago
That’s a good way to demonstrate where we are with AI; it can beat a human on any given narrow definition, but still fall far short on any task that requires some ineffable creativity, synthesis, or understanding of the word. We can make a Python script more “creative” than 95% of humans (by a meaninglessly narrow definition of creativity), but all our effort as a society has still not produced a car that can drive better than your average teenager.
andrepd 5y ago
> For example, recently psychologists discovered a simple way to test for creativity. The simple test is to come up with a list of 10 words as different as possible from each other. According to the test, GPT-3 is far more creative than the average person, scoring in the 85-95th percentile. This is the dumbest thing I read all week. My god, how can somebody be so obtuse and so... narrow-minded? An excellent argument in favour of including some humanities course in tech degrees at least.
- wheelinsupial 5y ago
  > An excellent argument in favour of including some humanities course in tech degrees at least. What are you seeing that I am not? It looks to me like the author of the article went to a liberal arts school (Hampshire College) for a BA in cognitive science and philosophy before studying a PhD in neuroscience, and is a published fiction writer (both short-stories and a novel). I suppose what a tech degree is needs to be defined, but in Canada a computer science degree would involve "breadth electives" that cover humanities and the social sciences. It is at least two of each depending on if one is studying for a BSc or a BA (a BA would require more). Even a degree in computer engineering in Canada would require at least one course in the humanities. I think this applies in the US as well. I believe computer science and computer engineering are the stereotypical tech degrees people picture when the argument is made that tech is lacking in an understanding of the humanities and ethics. It seems that the author of the article is about as far from a typical tech degree as I've outlined above, but you may be seeing something else that I've missed?
  - quinnjh 5y ago
    Church turing paradox but in comments ~ ;p - to elaborate One take is that it’s bollocks- a mechanistic measure for a mechanism to “win” at has nothing to do with human creativity. One would argue that the arts should be taught with stem to help engineers interface with humanity- especially the indefinables. Another take is to ask what it is the machine isnt quite doing. As i believe its been said before “tell me exactly what the program can’t do and i will make it do exactly that”. But where does it end? To satisfy the argument we will need to define creativity, humanity, arts, not to mention bicker on the semantics of building the necessary shared ontology! The synthesist in me thinks the next Shakespeare surely could benefit from n+1 “monkeys and typewriters”
frankenst1 5y ago
The tasks asked to name 10 nouns which are as "different" from each other as possible. I chose 10 common words which all belonged to different concepts, I scored 75%. The results then stated that "different" refers to the probability of two words appearing closely to each other in known texts. With that in mind I retook the test and chose 10 words which are rare or vulgar. Scored 93%.

inglor_cz 5y ago

My naive take is that if you can keep 2N concepts in your head at once, the number of possible connections among them is much more than twice as big than with N concepts, so you have a much wider graph to analyze and possibly fish out interesting ideas from. (Unless N is 0, of course.)

MattGaiser 5y ago
I agree, but that is a two stage process. Sure the machine can beat us on concepts, but can it make meaningful connections?

pddpro 5y ago

Memory is much more correlated with creativity than one would think.

PeterisP 5y ago

Arguably that line of Python wouldn't do very good - you'd want choices that maximize the difference as opposed to be merely something random, this is harder than it appears; and the distribution of words is also uneven enough (both in relative frequency of word usage, and also in lexeme "density" w.r.t how many terms there are for some domains and how few for others) that pure random sampling would be not a good option.

MattGaiser 5y ago
It does pretty well on this (https://www.datcreativity.com/task https://www.datcreativity.com/task?), at least with the below code. Consistently getting 80-90%. https://github.com/MattGaiser/CreativeWordChoice/tree/master/ https://github.com/MattGaiser/CreativeWordChoice/tree/master... I just grabbed a random list of nouns, narrowed it by the params specified, and fed them in as required. You may need more than 10 as it rejects some, but in that case I just re-ran it and put in the first nouns to come up.
wongarsu 5y ago
Word frequencies follow a power law, so nearly all words are used very infrequently. So a randomly chosen word is likely to be very uncommon. Also there are many more specific concepts than general ones (because each general concept encompasses many specific ones). So a randomly chosen word is more likely to be specific than general. Random sampling is likely to give you unusual, highly specific words. I don't see how that wouldn't do very good. Sure, you could do better by mapping each word to a concept, then selecting ten random concepts and selecting a word for each concept. But why do that when you get 99% of the way there with just random choice. To show this experientially, I let python choose 10 random words from the first English word list I found. I got "epilithic, codirects, serologic, capsian, digestibility, yasna, samsara, supergun, hepatoptosis, kyurin". Sure, capsian and kyurin are somewhat related (names of places/people), as are serologic and hepatoptosis (both medical terms), but it's a pretty good spread.
- rich_sasha 5y ago
  Not poster but I think the point is you could do better, not that this doesn’t work. You can imagine having some discrete optimisation maximising the dissimilarity of words, though that’s hard to do, in particular in a discrete space.

the_jeremy 5y ago

Also the test I took[0] scores this very poorly, by determining how often those words are in the same section of a text as each other, meaning choosing uncommon words (crenellations, gravedigger) will show much more creativity than common words (castle, grave) that are similar conceptually. [0]: https://www.datcreativity.com/task https://www.datcreativity.com/task?

quimplo 5y ago

> recently psychologists discovered a simple way to test for creativity Jesus. That "discovery" is https://www.datcreativity.com/faq#simple https://www.datcreativity.com/faq#simple It's not a "discovery", it's not an objective way to test for creativity, and throwing GPT-3 at a test designed to surface relative differences among humans is like saying a pair of dice are better than humans at supplying random numbers. Actually, what is the methodology for applying a language prediction model to a "come up with 10 'unrelated' words" prompt? Especially when the "test" relies on a very specific and game-able definition of "unrelated" that needs to be hidden from test-takers to provide even a semblance of "valid" results? The amount of BS being juggled among those who should presumably know better is astounding and depressing.

advael 5y ago

I got a dual-major in psychology in undergrad, and while it's not been an exceptionally useful degree, it did bequeath me with an enormous dose of skepticism toward psychometrics like this. The replication crisis solidified that skepticism Being able to choose less "related" words doesn't sound like it has much meaningful basis at all in what we care about when we say "creativity". This metric, as many others like it, was likely chosen and popularized because it is easy to measure

MattGaiser 5y ago
I can see it being strongly correlated, as that human has more items to draw from for figuring out solutions, but I don't see how it can be used to test computer creativity. Python's random surely isn't creative.
- advael 5y ago
  That's the problem with metrics though. You can make anything sound plausible. Once a model has been presented to you, you can come up with an endless stream of just-so stories for why it measures what you were told it measures. Nonetheless, some metrics simply aren't that useful, and get adopted anyway because they produce a bunch of data. Psychometrics like this have an awful track record for standing up to more meaningful tests like "Can we drive any outcome we care about with this"

bryanrasmussen 5y ago

> For example, recently psychologists discovered a simple way to test for creativity. The simple test is to come up with a list of 10 words as different as possible from each other. When Robert Frost was the poet laureate of the U.S some news magazine (I think NewsWeek, although it might have been Time) did a creativity test on him, using inkblots iirc, and he scored particularly low. Frost was irate afterwards and remarked something to the affect that to be creative required that you not see things in the inkblot, that is to say to not try to see things in distractions, that he had spent years training himself not to see things in meaningless shapes. I think that there is quite a lot of truth in this, also if someone wanted to give me a simple test to come up with a list of 10 words as different as possible from each other I would probably get verbally abusive - at least call them an idiot.

jimbokun 5y ago

I would just like to say, the insight to try just a random choice of words and see how well that satisfies the criteria demonstrates a kind of creativity that isn't captured by having people generate a list of "unrelated" words from memory.

ashtonkem 5y ago

That’s a good way to demonstrate where we are with AI; it can beat a human on any given narrow definition, but still fall far short on any task that requires some ineffable creativity, synthesis, or understanding of the word. We can make a Python script more “creative” than 95% of humans (by a meaninglessly narrow definition of creativity), but all our effort as a society has still not produced a car that can drive better than your average teenager.

andrepd 5y ago

> For example, recently psychologists discovered a simple way to test for creativity. The simple test is to come up with a list of 10 words as different as possible from each other. According to the test, GPT-3 is far more creative than the average person, scoring in the 85-95th percentile. This is the dumbest thing I read all week. My god, how can somebody be so obtuse and so... narrow-minded? An excellent argument in favour of including some humanities course in tech degrees at least.

wheelinsupial 5y ago
> An excellent argument in favour of including some humanities course in tech degrees at least. What are you seeing that I am not? It looks to me like the author of the article went to a liberal arts school (Hampshire College) for a BA in cognitive science and philosophy before studying a PhD in neuroscience, and is a published fiction writer (both short-stories and a novel). I suppose what a tech degree is needs to be defined, but in Canada a computer science degree would involve "breadth electives" that cover humanities and the social sciences. It is at least two of each depending on if one is studying for a BSc or a BA (a BA would require more). Even a degree in computer engineering in Canada would require at least one course in the humanities. I think this applies in the US as well. I believe computer science and computer engineering are the stereotypical tech degrees people picture when the argument is made that tech is lacking in an understanding of the humanities and ethics. It seems that the author of the article is about as far from a typical tech degree as I've outlined above, but you may be seeing something else that I've missed?
- quinnjh 5y ago
  Church turing paradox but in comments ~ ;p - to elaborate One take is that it’s bollocks- a mechanistic measure for a mechanism to “win” at has nothing to do with human creativity. One would argue that the arts should be taught with stem to help engineers interface with humanity- especially the indefinables. Another take is to ask what it is the machine isnt quite doing. As i believe its been said before “tell me exactly what the program can’t do and i will make it do exactly that”. But where does it end? To satisfy the argument we will need to define creativity, humanity, arts, not to mention bicker on the semantics of building the necessary shared ontology! The synthesist in me thinks the next Shakespeare surely could benefit from n+1 “monkeys and typewriters”

frankenst1 5y ago

The tasks asked to name 10 nouns which are as "different" from each other as possible. I chose 10 common words which all belonged to different concepts, I scored 75%. The results then stated that "different" refers to the probability of two words appearing closely to each other in known texts. With that in mind I retook the test and chose 10 words which are rare or vulgar. Scored 93%.