9 ms·
Robots are writing poetry, and many people can’t tell the difference
- boygobbo 5y agoAnd most people couldn't tell the difference between a genuine paper on quantum mechanics and one generated by a machine. Would that mean machines could write genuine papers on quantum mechanics? This article says more about most people's ignorance of poetry than anything else.
- gwern 5y agoFWIW, the best source of GPT-3 poetry, if you want to see what it can do beyond the cursory discussion in OP, remains my own https://www.gwern.net/GPT-3#poetry https://www.gwern.net/GPT-3#poetry page. (Pointing out, among other things, that GPT-3 can't rhyme and so Andrew Brown's sample is a bit misleading about capabilities.) I'm surprised it's neither mentioned nor linked, since there is not that much GPT-3 poetry out there... OP argues that GPT-3's poems 'fall dead' and never become 'a memorable unity'. I will just say that for me, a number of GPT-3's poems like "Uber-Poem" (https://www.gwern.net/GPT-3#uber-poem https://www.gwern.net/GPT-3#uber-poem) or "The Universe Is A Glitch" (https://www.gwern.net/GPT-3#the-universe-is-a-glitch https://www.gwern.net/GPT-3#the-universe-is-a-glitch) struck me as a good deal more lively and memorable than the vast majority of the poems I puzzle through reading the New Yorker or Paris Review.
- wizwit999 5y agoHonestly English poetry is just very bad.
- zdragnar 5y agoI agree. I think a lot of this falls to how poetry is taught in schools. There's a bunch of emphasis on the basic mechanics, but there is almost no insight at all into what makes a poem compelling. Take, for instance, the humble haiku. I was taught it was three lines with a set number of syllables. No insight into there Japanese language, the meaning behind recurring themes, or even it's origins in collaborative poetry called renku. You can draw a line from haiku, to wood block prints, to dada and surrealism, to beatnicks and the Beatles. Western / English poetry is treated even worse. Lessons usually boil down to "write out words so that some rhyme in the right places". There are some fantastic English poets throughout history and today, buried in a sea of unfortunate imitations.
- nullc 5y agoHave you heard of anyone attempting a transformer better specialized for poetry? ISTM that GPT3's tokenization isn't ideal as makes it needlessly complex to model how words sound. Maybe reducing the vocabulary to simple english so every word is a single token and using some embedding that was trained using word sounds, or something along those lines?
- gwern 5y agoI talked with one or two people were interested in further poetry work but I haven't heard of anything since. The most interesting recent news is someone claiming that InstructGPT can be prompted to do what looks like genuine rhyming, but they didn't share their prompt and I've been too busy to followup. The vocab tokenizing already mostly works, as you can see from the results. I think the solution really is as simple as switching to a byte encoding like ByT5, and perhaps someone spending 30 minutes to run a few megabytes of text through an IPA phonetic encoder to enrich the training corpus. Rhyming is just not that hard or complex compared to the things neural nets already do. It's purely artifactual. Then one can focus on more interesting and challenging tasks, like curating high-quality corpuses for finetuning.
- nullc 5y agoYeah, the challenge there is both understanding the word sounds and enough of the world to say something clever. It shouldn't just help with rhyming, puns that depend on understanding how words sound. One of these models could easily be vastly super human at cross-language puns if they only understood the sound of words. :P
- wolverine876 5y ago> a number of GPT-3's poems like "Uber-Poem" (https://www.gwern.net/GPT-3#uber-poem https://www.gwern.net/GPT-3#uber-poem) or "The Universe Is A Glitch" (https://www.gwern.net/GPT-3#the-universe-is-a-glitch https://www.gwern.net/GPT-3#the-universe-is-a-glitch) struck me as a good deal more lively and memorable than the vast majority of the poems I puzzle through reading the New Yorker or Paris Review. I read them and they are impressive and memorable - because they were generated by a computer. If a person wrote them they would be very unremarkable; nothing about them would stand out, in insight or language or form. What is remarkable is that a computer generated text that is passable as human poetry. If a high schooler or non-artistic college student wrote them, I wouldn't be surprised. (Also, and maybe I'm too literal here, if poems are seen as 'puzzles', that will miss the point for 99% of them.)
- pmoriarty 5y ago"if poems are seen as 'puzzles', that will miss the point for 99% of them" What is the point of 99% of them?
- brnaftr361 5y agoI like Tolstoy's conclusion in What is Art?, which I'm doing an injustice by summarizing as "universal emotional communication" which is contrasted with paid work, technical prowess, esoteric work, formulaic-derivative or pure aesthetics - anything that requires effort and "cultivation" to understand is not art.
- nescioquid 5y agoI've never seen Tolstoy's attempt at a such definition before, but from what you describe, it reminds me of what Joyce had Stephen Daedalus think about art (I wonder if it was informed at all by Tolstoy). As I recall it, the idea is that art starts with an unmitigated cry which is then made abstract, universal, in some way. I see no way for this to occur without some technical prowess, however. One can surely be a noble and delight in the way in which the peasants and serfs conduct their culture. But to surmise that their art was artless (i.e. without τέχνη) seems foolish and makes me think there must be more to it than what you've suggested in your comment. Otherwise, were I to throw a fit in the street, that would qualify as art; but were I to do it in hexameter, that would not be art. I think you are in essence right about what you're arguing, however. No cry comes from a machine, after all.
- sebmellen 5y agoThe video for The Universe is a Glitch is amazing: https://youtube.com/watch?v=5wl29laSOU4 https://youtube.com/watch?v=5wl29laSOU4
- practice9 5y agoDo you know if there is a page where you can guess whether a given poem is GPT-n generated or written by a human? Would be a cool small experiment to check if people can tell the difference, with actual data
- gwern 5y agoThere is not that I know of, although you could probably cobble one together from my samples.
- drusepth 5y agoThe "American poet Andrew Brown" here -- I'm honored, I don't think anyone has ever called me a real poet before! Looks like the full poem they're referring to is: > I think I’ll start to rain, > Because I don’t think I can stand the pain, > Of seeing you two, > Fighting like you do. For the record, the original tweet [1] doesn't mention rhymes, but the replies [2] do: I say generations that actually rhyme are "pretty rare" and that only half of poems generated with the prompt I was sharing were even worth editing into something good. Of course, that part doesn't get quoted in articles like this... [1] https://twitter.com/drusepth/status/1293937354978271234 https://twitter.com/drusepth/status/1293937354978271234 [2] https://twitter.com/drusepth/status/1294024126814752768 https://twitter.com/drusepth/status/1294024126814752768
- frereubu 5y agoDespite the headine, the article is actually pretty sceptical about the quality of current machine-generated poetry, aside from some fluff about the "inevitability" of AI being able to write human-quality poetry. There's nowhere in the article that I could see which claims that "many people can't tell the difference." (Headlines are often written by editors rather than the writers themselves, and this one seems to have generated a particularly big disconnection between the two - I think the write has been hard done by here). Besides which, often with these claims if you dig into them they start to fall apart. There's one about wine tasting that's often circulated - a study where people "couldn't tell the difference between red wine and white wine with added red food colouring". But IIRC the people in the study were first-year oenology (the study of wine) students, and I bet you (a) they'd be able to tell the difference between a Malbec and a Pouilly Fumé with red food colouring and (b) people who'd been tasting wine for longer (i.e. people who had more experience) would be able to tell the difference. I'm reminded of a radio comedy panel game where the punchline to a joke was "bite the wax tadpole", and there was quite a bit of laughter around the fact that it was probably the first time in human civilisation that those four words had been used in that order before. AI may eventually be able to write something reasonable, but there's something very specific about good poetry ("good" is of course subjective) that I think is likely to always set it apart.
- alasdair_ 5y ago> people who'd been tasting wine for longer (i.e. people who had more experience) would be able to tell the difference. I’m not so sure of this. Netflix had a fantastic documentary called “Som” that followed a group of people who were about to take the Master Sommelier exam (roughly fifty total in the whole world) and even these extremely experienced people would get even basic things like the varietal completely wrong quite frequently.
- frereubu 5y agoThanks for the pointer - I'll have a look.
- wolverine876 5y ago
- imranhou 5y agoIs it just me or more people think using "Robots" to identify software is out of place and just incorrect?
- Dylan16807 5y agoI feel like it's fine. "Bots" refer to software entities all the time, and there's no meaningful physical action needed for writing a poem. Using "robot" for something that's not highly autonomous is wrong, though.
- freeflight 5y agoDepends how much you want to abstract hardware and software to define "machine". But the words roots trace back to the old Church Slavonic word "robota" for servitude/forced labor/drudgery [0]. In that context even a calculator could be considered a "robot". [0] https://www.sciencefriday.com/segments/the-origin-of-the-word-robot/ https://www.sciencefriday.com/segments/the-origin-of-the-wor...
- tessierashpool 5y agoIn one example, American poet Andrew Brown asked the software to take the perspective of a cloud gazing down on two warring cities. GPT-3 delivered a rhyming poem that began, not uncharmingly, with “I think I’ll start to rain.” Stephen Marche, writing an article for The New Yorker, assigned GPT-3 maybe the trippiest poem in the English canon: Samuel Taylor Coleridge’s fifty-four-line “Kubla Khan”—an opium dream interrupted in the middle of its composition in 1797 and never completed. GPT-3’s mission? Finish the fragment. What the program fantasized was so sophisticated (“The tumult ceased, the clouds were torn, / The moon resumed her solemn course”) that readers unfamiliar with the original poem might have had a hard time discerning where Coleridge ended and the computer began. This is not robots writing poetry. This is poets using computers.
- greenie_beans 5y agoLet the record show that I hid some text generated by GPT-2 in my novella
- ed25519FUUU 5y agoThe obviously takeaway is not that poetry by robots is really good, but that most poetry by humans is really bad.
- visarga 5y ago> most poetry by humans is really bad I think human average is below top AI on a large number of tasks already. We don't actually know how many because the latest generation of models adapt so easily, it's a matter of asking (finding the right prompt). For example GPT-3 writing articles on demand - hard to tell which is real and which is fake. Translation works very well, and no human could cover the same number of language pairs. And not just translation, but also voice recognition and speech synthesis. Models also do well in symbolic math and can generate code at average human level in competitive programming. Short poetry is possible, music composition is possible. I think the average humans can write worse poetry and music. Driving cars? Controversial, but AI may be better than the average humans. DALL-E 2 is a better illustrator than the average human. Even protein folding is better with AI than we could do with any other method. Games - it's hard to find games where the top human still beats the top AI. The idea is that we're passing a threshold moment now. Especially GPT-3 and DALL-E 2 and family, they have a radical new kind of generality. It wasn't like this before 2020. Now it's 100x easier to get an AI to do a task, no need to label much data, just tell it what you want and give a couple of examples, like you would do with a human. It's gonna be interesting to see the effects ripple in all the fields in the next 5-10 years.
- wolverine876 5y agoMany discussions turn to political and inferiority issues, essentially, 'is this development a signal that machines are surpassing us?' But let's leave that aside and look at it as art: Many artists have used randomness; you could look at this software as a sort of controlled randomness. More interesting to me, artists also have created works challenging the source and significance of meaning, teasing that line between meaning and nothingness. There are so many possibilities with this tool. Also, it could be a sort of Rorschach Test, even adding a feedback loop.
- gotaquestion 5y agoIt seems no one has stopped to consider that some people just like poetry, whether it is from an "authenticated" human source, or AI and is close enough to language to engage the brain in a variety of ways. Does it matter?
- wrnr 5y agoThere was an art movement called dadaism in the 20th century where poets would cut up a newspaper and randomly rearrange a selection of letters, syllables and words into verse. People where flabbergasted as to how such a random process could produce such profound statements. Language already encodes meaning and our brains are wired to read into it. I always think of this when I read how amazing language models.
- swayvil 5y agoTo be fair, most people have very poor taste and can't tell good poetry from a hole in the ground.
- ShamelessC 5y agoTo be fairer, poetry like many other arts is highly subjective. The difference between a good poem and a bad one may have more to do with comparisons between other works at a given point in time than it does with any sort of "ideal poem". Same thing going on with dalle-2. Hell, even formal mathematics has trouble justifying "why" a particular problem is worth solving. I think people take into account the author, their sincerity, their experience, and how those words personally make the reader feel. This is the same reason I love Radiohead but would probably hate a Radiohead knockoff made by some AI. It's more about the people in the band and whether or not I relate with their message than it is about whether or not they are objectively good to listen to (they aren't, many people hate that band).
- swayvil 5y agoIt isn't "highly subjective". It's merely fuzzy, difficult to quantify and outside the understanding of the common dude Agreement about quality happens all the time. Smarts guys often concur. Good taste is real.
- ShamelessC 5y agoBeing highly subjective doesn't make its study less worthwhile in the slightest and study of subjective topics can still possess rigour and commonly held opinions derived from first principles (i.e. "taste"). Look no further than the study of art, music and literature and philosophy to see just how much humans enjoy deep dives into arbitrary rules. It's maybe even part of what makes us human.
- Geonode 5y agoMany people can't tell the difference between Pepsi and Coke in a blind test, which is insane. I wouldn't base any decisions on 'most people.'
- RcouF1uZ4gsC 5y agoI wonder if humans are so primed to find meaning in poetry that a lot of people would find something in anything. For example, take Bob Dylan and have him write 7 random words (for example using the Diceware password generator.) Then release this 7 words saying that this is a leak of one of the lines in an unpublished song. I bet that you will get entire articles analyzing those 7 words in great detail and coming to the conclusion that that line is one of the most profound lines ever written.
- notahacker 5y agoOr take "I am the Walrus" a mix of dream scenes and childhood nonsense rhymes scrambled[1] together as a riposte to taking Beatles and Dylan lyrics too seriously, and see people analysing it as using the Inuit symbol of death to say something about Paul... [1]I wasn't consciously thinking of the egg man when I picked this word...
- whateveracct 5y agoBut can the Poetry World tell the difference?
- Jimmy 5y agoBetween this and DALL-E 2, I’ve been ruminating on David Hilbert’s old quip about a student: “For a mathematician, he did not have enough imagination. But now he is a poet, and everything is fine.” As someone with experience in both STEM and the arts, I don’t feel particularly bad about saying that some “creative” pursuits are more creative than others, and my rough thesis is that the ones on the higher end of the scale are the ones that will take longer to fall to AI. Representational painting is, at the end of the day, a somewhat mechanical problem, so it’s not too surprising that AI is rapidly catching up to humans there. “Draw me a cat eating ice cream on the beach” is a fairly well-defined request with a clear range of acceptable solutions. I think the creativity exhibited in a long-form written work like say, Plato’s “Republic”, is of a much higher degree than your average painting or short poem. Not only is it an aesthetically beautiful work, but it contains a great deal of novel (for its time) intellectual content as well, filtered through the personal perspective of the author, and organized in an elaborate logical structure. When it comes to these types of large-scale creative works, AI hasn’t even begun to tackle the problem yet.
- picture 5y agoSo its "intention" that is hard to replicate by machines, huh. How far is the day when the first machine decide to make something truly for its own sake?
- tintedfireglass 5y ago"intention" refers to critical thinking and intelligence so yeah that's what is missing from AI
- Jimmy 5y agoWell, no, I wouldn’t say that my post was about “intention”. I could imagine, in principle, someone saying “hey GPT-3, write me a philosophical masterpiece” or “write me a hit novel” and it does. No intentionality required. It’s just that in actuality, it can’t do that yet.
- e-dt 5y agoWhy do we neglect poetry, anyway? After Mother Goose there's a great lacuna in poetic education up until, I suppose, a possible Literature degree. In school we study and restudy novels and Shakespeare-as-a-novel, which (despite the method of teaching) sets up students to enjoy novels. But we study poetry not at all, or only rarely. (Let alone long-form poetry!) And certainly there's a market for poetry: after all, rap is poetry, and the best examples of rap are up there with the greats. I need to amend my previous question, actually. Both poetry and prose have fairly popular public outlets in the form of the novel and rap. (Of course rap is a specific form of poetry, but the popular novel tends to take a stereotyped form as well: the romance, the detective story, formerly the pulp adventure...) The best examples of popular novels and rap ascend to the status of high art (really the best type of high art, the high art that is also low art.) But, crucially, a novel reader whose interest is piqued by very good popular novels can fairly seamlessly transition into reading literary fiction, as it shares in large part the form and characteristics of the novel. The rap listener awed by, e.g. Kendrick Lamar's lyricism, has no such path to literary poetry. How do we expect poetry to thrive in these conditions? And that's not even getting into the historical conditions which mean many people dismiss rap music.
- sph 5y agoI don't have a similarly elegant answer to your query, but it made me pause and consider why I dislike poetry and in lesser way, rap music. I dislike poetry (and rap) because it is too introspective, too personal, too self centred and painting a picture of the author's mindset that is lost on someone else. Other art forms try to elicit an emotion to the readers or viewers. They're telling a story, presenting an idea for someone else's eyes. Feels to me like no one can appreciate the poet's art and depth more than its author. As such, there is no bad poetry, which is why it's that easy to have robots convince us that random words are instead contents of a particularly troubled mind.
- nonstickcoating 5y agoThat's really interesting. In Germany poetry is handled differently. In the earlier years of what would be high-school there are poets like Jandel[0] and Morgenstern[1] in the curriculum who wrote a lot of funny, nonsensical poems playing with language a lot. Later (if you do schooling aimed at getting you to university) there is a lot of in-depth reading of romantic[2]as well as clacissist (think the latet works of Goethe and Schiller) poetry. Also, at least in my Latin classes, we also read Ovid, as well as Shakespeare's sonnets. These poems are analysed regarding their contents, as well as more formally concerning meter etc. [0] https://en.m.wikipedia.org/wiki/Ernst_Jandl https://en.m.wikipedia.org/wiki/Ernst_Jandl [1] https://en.m.wikipedia.org/wiki/Christian_Morgenstern https://en.m.wikipedia.org/wiki/Christian_Morgenstern [2] https://en.m.wikipedia.org/wiki/Romanticism https://en.m.wikipedia.org/wiki/Romanticism
- Dwedit 5y agoWhat have we here, laddie? Mysterious scribblings? A secret code? No! Poems, no less! Poems, everybody! The laddie reckons himself a poet! "Money get back, I'm all right, Jack, Keep your hands off my stack. New car, Caviar. Four star daydream, Think I'll buy me a football team." Absolute rubbish, laddie!
- noisy_boy 5y agoPoetry being the supremely subjective thing it is, quite possible that a "good" poem written by a robot is "better" than many written by humans - which would actually be a good thing - who is to judge? Atleast we won't be accusing robots of stealing "good paying jobs" of poets anytime soon.
- uwagar 5y agomaybe poets are robotic
- trophycase 5y agoAs someone who doesn't really get poetry, I probably couldn't tell the difference.
- lordgrenville 5y agoI would guess that an AI's ability to write poetry passable as human-authored is inversely correlated with length. So good haikus - certainly; Elegy Written in a Country Churchyard - never (before AGI).
- ebjaas_2022 5y agoI'm not sure about good short poems. A short poem is so condensed, that it allows the reader to project his own meaning into the text, even when there was no meaning to begin with (which, arguably, is the case with all current AI generated poetry). I think a short AI generated poem can be perceived as good, because of this. I still hope that I would not be tricked by it myself.
- kuhewa 5y ago> There is no question that poetry will be subsumed, and soon, into the ideology of data collection, existing on the same spectrum as footstep counters, high-frequency stock trading, and Netflix recommendations. Maybe this is how the so-called singularity—the moment machines exceed humans and, in turn, refashion us—comes about. This reminds me of in the mid to late oughts in biology we had similar futurists telling us the scientific discipline of taxonomy would be rendered obsolete by advances in DNA barcoding. It seems naive in hindsight. Genetic techniques are now established as a tool to aid taxonomy, but one among many. I wonder if those in literature would feel the same about this.
- bertylicious 5y agoI always found it funny how AI researchers are desperately trying to create consciousness so their machines will finally pass as human. All the while the key to being human lies in the unconscious.
- MomoXenosaga 5y agoProbably because nobody actually ever reads poetry. Also modern art can't be distinguished from trash. I for one welcome our AI overlords.
- chadcmulligan 5y agoA lot of it probably is, the problem with classical art, or say surrealism, is all the trash has been forgotten, and the only thing that is remembered is the good stuff. With modern (contemporary) art we have to wait for the winnowing to occur. Same with poetry, people do read and write poetry still, I've always loved Dylan Thomas. I feel his poetry is, perhaps, very approachable for techies: "Do not go gentle into that good night, Old age should burn and rave at close of day; Rage, rage against the dying of the light." ...
- optimalsolver 5y agoForm vs meaning: https://scottaaronson.blog/?p=6387 https://scottaaronson.blog/?p=6387
- deleted 5y ago[deleted]
- bitlax 5y agoHiggledy-piggledy Use machine learning to Add more bad poetry Into the mix O, the futility! Never so clever as My double dactyls and Dirty lim'ricks
- stemlord 5y agoAI art is just its own subset of conceptual interest in the area of generative/computational aesthetics. It won't take over / it's not some kind of paradigm shift for the art world because it doesn't fundamentally challenge what's important about complex interpersonal emotional communication in any new way just by mimicking it. Sol LeWitt has already separated agency one more degree by writing instruction sets for others to perform the artworks for the better part of the past century.
- mensetmanusman 5y agoThis is just another way of taking ‘images’ of emergent phenomena. Some people take pictures of nature, and some people take pictures text output that emerge from the corpus of all human writing.
- INGELRII 5y agoPoems and aphorisms are an art form that requires the reader to fill gaps in logical structure to create meaning. Humans naturally try to generate meaning from arbitrary nonsense if it's grammatically correct. Human curated samples will almost certainly be good. Consider MegaHAL from 1996 https://en.wikipedia.org/wiki/MegaHAL https://en.wikipedia.org/wiki/MegaHAL From all the stuff it generated we are left with handful of gems like: CHESS IS A FUN SPORT, WHEN PLAYED WITH SHOT GUNS. COWS FLY LIKE CLOUDS BUT THEY ARE NEVER COMPLETELY SUCCESSFUL. TIME. TIME. WHAT IS TIME? SWISS MANUFACTURE IT. FRENCH HORDE IT. ITALIANS WANT IT, HUH? WELL, I MUST BE THURSDAY. I NEVER COULD GET THE HANG OF THURSDAYS. I COULD SHAKE MY TINY FIST AND SWEAR I WASN'T WRONG, BUT WHAT'S THE SENSE IN ARGUING WHEN YOU'RE MUCH TOO BUSY RETURNING TO THE LAVATOR.
- ZeroGravitas 5y agoI can't help but feel this kind of thing is going to hurt AI. Poetry seems perfectly placed in the zone of "something that feels like a high-level human achievent" mixed with "most people have no clue about what this involves". It's like chess all over again. Smart people can play chess well. A machine that can play chess well is smart. No, this is a stupid conclusion that never held up to any scrutiny even in the decades before we had real functioning chess computers to base our intuitions on. There's another story on HN about AI doing IQ tests, the stupidity of which makes me angry. Mechanical calculators can outdo human beings at specific tasks just as car jack can lift more than a human. This doesn't make them "strong" or "smart", they're just tools. Cool, useful (sometimes) tools. Pretending they are magical beings doesn't really seem to help in any way, it doesn't even seem to be desirable if it was true which it obviously isn't so why do we keep doing it?
- gxonatano 5y agoI teach computational approaches to literature at a university, and I always give my students a test to see if they can tell which poems were written by a computer, and which were written by a human. They tend to score about 50%. But to me, a literary scholar, I know immediately, from a glance even, which ones aren't written by humans, because they're patently ridiculous. I guess if you're not really a reader of poetry, or don't understand it, it's all just nonsense anyway. But for the rest of us, not even the most advanced language models can pull off a poem, convincingly.
- caseyf 5y agowow Racter! I haven't thought about that in forever. I had Racter on my Mac in 1984 and spent plenty of time talking to it