4 ms·

FWIW, the best source of GPT-3 poetry, if you want to see what it can do beyond the cursory discussion in OP, remains my own https://www.gwern.net/GPT-3#poetry

by gwern 5y ago

FWIW, the best source of GPT-3 poetry, if you want to see what it can do beyond the cursory discussion in OP, remains my own https://www.gwern.net/GPT-3#poetry https://www.gwern.net/GPT-3#poetry page. (Pointing out, among other things, that GPT-3 can't rhyme and so Andrew Brown's sample is a bit misleading about capabilities.) I'm surprised it's neither mentioned nor linked, since there is not that much GPT-3 poetry out there... OP argues that GPT-3's poems 'fall dead' and never become 'a memorable unity'. I will just say that for me, a number of GPT-3's poems like "Uber-Poem" (https://www.gwern.net/GPT-3#uber-poem https://www.gwern.net/GPT-3#uber-poem) or "The Universe Is A Glitch" (https://www.gwern.net/GPT-3#the-universe-is-a-glitch https://www.gwern.net/GPT-3#the-universe-is-a-glitch) struck me as a good deal more lively and memorable than the vast majority of the poems I puzzle through reading the New Yorker or Paris Review.

wizwit999 5y ago
Honestly English poetry is just very bad.
- zdragnar 5y ago
  I agree. I think a lot of this falls to how poetry is taught in schools. There's a bunch of emphasis on the basic mechanics, but there is almost no insight at all into what makes a poem compelling. Take, for instance, the humble haiku. I was taught it was three lines with a set number of syllables. No insight into there Japanese language, the meaning behind recurring themes, or even it's origins in collaborative poetry called renku. You can draw a line from haiku, to wood block prints, to dada and surrealism, to beatnicks and the Beatles. Western / English poetry is treated even worse. Lessons usually boil down to "write out words so that some rhyme in the right places". There are some fantastic English poets throughout history and today, buried in a sea of unfortunate imitations.
nullc 5y ago
Have you heard of anyone attempting a transformer better specialized for poetry? ISTM that GPT3's tokenization isn't ideal as makes it needlessly complex to model how words sound. Maybe reducing the vocabulary to simple english so every word is a single token and using some embedding that was trained using word sounds, or something along those lines?
- gwern 4y ago
  I talked with one or two people were interested in further poetry work but I haven't heard of anything since. The most interesting recent news is someone claiming that InstructGPT can be prompted to do what looks like genuine rhyming, but they didn't share their prompt and I've been too busy to followup. The vocab tokenizing already mostly works, as you can see from the results. I think the solution really is as simple as switching to a byte encoding like ByT5, and perhaps someone spending 30 minutes to run a few megabytes of text through an IPA phonetic encoder to enrich the training corpus. Rhyming is just not that hard or complex compared to the things neural nets already do. It's purely artifactual. Then one can focus on more interesting and challenging tasks, like curating high-quality corpuses for finetuning.
  - nullc 4y ago
    Yeah, the challenge there is both understanding the word sounds and enough of the world to say something clever. It shouldn't just help with rhyming, puns that depend on understanding how words sound. One of these models could easily be vastly super human at cross-language puns if they only understood the sound of words. :P
wolverine876 5y ago
> a number of GPT-3's poems like "Uber-Poem" (https://www.gwern.net/GPT-3#uber-poem https://www.gwern.net/GPT-3#uber-poem) or "The Universe Is A Glitch" (https://www.gwern.net/GPT-3#the-universe-is-a-glitch https://www.gwern.net/GPT-3#the-universe-is-a-glitch) struck me as a good deal more lively and memorable than the vast majority of the poems I puzzle through reading the New Yorker or Paris Review. I read them and they are impressive and memorable - because they were generated by a computer. If a person wrote them they would be very unremarkable; nothing about them would stand out, in insight or language or form. What is remarkable is that a computer generated text that is passable as human poetry. If a high schooler or non-artistic college student wrote them, I wouldn't be surprised. (Also, and maybe I'm too literal here, if poems are seen as 'puzzles', that will miss the point for 99% of them.)
- pmoriarty 5y ago
  "if poems are seen as 'puzzles', that will miss the point for 99% of them" What is the point of 99% of them?
  - brnaftr361 5y ago
    I like Tolstoy's conclusion in What is Art?, which I'm doing an injustice by summarizing as "universal emotional communication" which is contrasted with paid work, technical prowess, esoteric work, formulaic-derivative or pure aesthetics - anything that requires effort and "cultivation" to understand is not art.
    - nescioquid 5y ago
      I've never seen Tolstoy's attempt at a such definition before, but from what you describe, it reminds me of what Joyce had Stephen Daedalus think about art (I wonder if it was informed at all by Tolstoy). As I recall it, the idea is that art starts with an unmitigated cry which is then made abstract, universal, in some way. I see no way for this to occur without some technical prowess, however. One can surely be a noble and delight in the way in which the peasants and serfs conduct their culture. But to surmise that their art was artless (i.e. without τέχνη) seems foolish and makes me think there must be more to it than what you've suggested in your comment. Otherwise, were I to throw a fit in the street, that would qualify as art; but were I to do it in hexameter, that would not be art. I think you are in essence right about what you're arguing, however. No cry comes from a machine, after all.
sebmellen 5y ago
The video for The Universe is a Glitch is amazing: https://youtube.com/watch?v=5wl29laSOU4 https://youtube.com/watch?v=5wl29laSOU4
practice9 5y ago
Do you know if there is a page where you can guess whether a given poem is GPT-n generated or written by a human? Would be a cool small experiment to check if people can tell the difference, with actual data
- gwern 4y ago
  There is not that I know of, although you could probably cobble one together from my samples.
drusepth 5y ago
The "American poet Andrew Brown" here -- I'm honored, I don't think anyone has ever called me a real poet before! Looks like the full poem they're referring to is: > I think I’ll start to rain, > Because I don’t think I can stand the pain, > Of seeing you two, > Fighting like you do. For the record, the original tweet [1] doesn't mention rhymes, but the replies [2] do: I say generations that actually rhyme are "pretty rare" and that only half of poems generated with the prompt I was sharing were even worth editing into something good. Of course, that part doesn't get quoted in articles like this... [1] https://twitter.com/drusepth/status/1293937354978271234 https://twitter.com/drusepth/status/1293937354978271234 [2] https://twitter.com/drusepth/status/1294024126814752768 https://twitter.com/drusepth/status/1294024126814752768

by gwern 5y ago

wizwit999 5y ago

Honestly English poetry is just very bad.

zdragnar 5y ago
I agree. I think a lot of this falls to how poetry is taught in schools. There's a bunch of emphasis on the basic mechanics, but there is almost no insight at all into what makes a poem compelling. Take, for instance, the humble haiku. I was taught it was three lines with a set number of syllables. No insight into there Japanese language, the meaning behind recurring themes, or even it's origins in collaborative poetry called renku. You can draw a line from haiku, to wood block prints, to dada and surrealism, to beatnicks and the Beatles. Western / English poetry is treated even worse. Lessons usually boil down to "write out words so that some rhyme in the right places". There are some fantastic English poets throughout history and today, buried in a sea of unfortunate imitations.

nullc 5y ago

Have you heard of anyone attempting a transformer better specialized for poetry? ISTM that GPT3's tokenization isn't ideal as makes it needlessly complex to model how words sound. Maybe reducing the vocabulary to simple english so every word is a single token and using some embedding that was trained using word sounds, or something along those lines?

gwern 4y ago
I talked with one or two people were interested in further poetry work but I haven't heard of anything since. The most interesting recent news is someone claiming that InstructGPT can be prompted to do what looks like genuine rhyming, but they didn't share their prompt and I've been too busy to followup. The vocab tokenizing already mostly works, as you can see from the results. I think the solution really is as simple as switching to a byte encoding like ByT5, and perhaps someone spending 30 minutes to run a few megabytes of text through an IPA phonetic encoder to enrich the training corpus. Rhyming is just not that hard or complex compared to the things neural nets already do. It's purely artifactual. Then one can focus on more interesting and challenging tasks, like curating high-quality corpuses for finetuning.
- nullc 4y ago
  Yeah, the challenge there is both understanding the word sounds and enough of the world to say something clever. It shouldn't just help with rhyming, puns that depend on understanding how words sound. One of these models could easily be vastly super human at cross-language puns if they only understood the sound of words. :P

wolverine876 5y ago

> a number of GPT-3's poems like "Uber-Poem" (https://www.gwern.net/GPT-3#uber-poem https://www.gwern.net/GPT-3#uber-poem) or "The Universe Is A Glitch" (https://www.gwern.net/GPT-3#the-universe-is-a-glitch https://www.gwern.net/GPT-3#the-universe-is-a-glitch) struck me as a good deal more lively and memorable than the vast majority of the poems I puzzle through reading the New Yorker or Paris Review. I read them and they are impressive and memorable - because they were generated by a computer. If a person wrote them they would be very unremarkable; nothing about them would stand out, in insight or language or form. What is remarkable is that a computer generated text that is passable as human poetry. If a high schooler or non-artistic college student wrote them, I wouldn't be surprised. (Also, and maybe I'm too literal here, if poems are seen as 'puzzles', that will miss the point for 99% of them.)

pmoriarty 5y ago
"if poems are seen as 'puzzles', that will miss the point for 99% of them" What is the point of 99% of them?
- brnaftr361 5y ago
  I like Tolstoy's conclusion in What is Art?, which I'm doing an injustice by summarizing as "universal emotional communication" which is contrasted with paid work, technical prowess, esoteric work, formulaic-derivative or pure aesthetics - anything that requires effort and "cultivation" to understand is not art.
  - nescioquid 5y ago
    I've never seen Tolstoy's attempt at a such definition before, but from what you describe, it reminds me of what Joyce had Stephen Daedalus think about art (I wonder if it was informed at all by Tolstoy). As I recall it, the idea is that art starts with an unmitigated cry which is then made abstract, universal, in some way. I see no way for this to occur without some technical prowess, however. One can surely be a noble and delight in the way in which the peasants and serfs conduct their culture. But to surmise that their art was artless (i.e. without τέχνη) seems foolish and makes me think there must be more to it than what you've suggested in your comment. Otherwise, were I to throw a fit in the street, that would qualify as art; but were I to do it in hexameter, that would not be art. I think you are in essence right about what you're arguing, however. No cry comes from a machine, after all.

sebmellen 5y ago

The video for The Universe is a Glitch is amazing: https://youtube.com/watch?v=5wl29laSOU4 https://youtube.com/watch?v=5wl29laSOU4

practice9 5y ago

Do you know if there is a page where you can guess whether a given poem is GPT-n generated or written by a human? Would be a cool small experiment to check if people can tell the difference, with actual data

gwern 4y ago
There is not that I know of, although you could probably cobble one together from my samples.

drusepth 5y ago

The "American poet Andrew Brown" here -- I'm honored, I don't think anyone has ever called me a real poet before! Looks like the full poem they're referring to is: > I think I’ll start to rain, > Because I don’t think I can stand the pain, > Of seeing you two, > Fighting like you do. For the record, the original tweet [1] doesn't mention rhymes, but the replies [2] do: I say generations that actually rhyme are "pretty rare" and that only half of poems generated with the prompt I was sharing were even worth editing into something good. Of course, that part doesn't get quoted in articles like this... [1] https://twitter.com/drusepth/status/1293937354978271234 https://twitter.com/drusepth/status/1293937354978271234 [2] https://twitter.com/drusepth/status/1294024126814752768 https://twitter.com/drusepth/status/1294024126814752768