4 ms·

I wonder if having access to characters actually helps rhyming in English all that much, as English rules of pronunciation are essentially rote-learned anyway.

I wonder if having access to characters actually helps rhyming in English all that much, as English rules of pronunciation are essentially rote-learned anyway. If it were not rote-learning, then it might make different mistakes, for example expecting two words to rhyme because they end with the same suffix. Perhaps it would be more effective to ask it to produce poems in the format: English0 IPA0 English1 IPA1, where each line is produced in both semantic and phonetic representations. This would give it the context necessary to “see” the rhymes without having to mess around with the tokenization.

telotortium 3y ago
Can GPT accurately transcribe English text to IPA? It looks like it can (https://chat.openai.com/c/c893cf54-5dfa-4774-9800-e6a74b7f9e2e https://chat.openai.com/c/c893cf54-5dfa-4774-9800-e6a74b7f9e...): Prompt: Eng Rhyming with IPA User Transcribe the following English text to IPA: English: I wonder if having access to characters actually helps rhyming in English all that much, as English rules of pronunciation are essentially rote-learned anyway. If it were not rote-learning, then it might make different mistakes, for example expecting two words to rhyme because they end with the same suffix. Perhaps it would be more effective to ask it to produce poems in the format: English0 IPA0 English1 IPA1, where each line is produced in both semantic and phonetic representations. This would give it the context necessary to “see” the rhymes without having to mess around with the tokenization. IPA: Response: aɪ ˈwʌndər ɪf ˈhævɪŋ ˈæksɛs tuː ˈkærəktərz ˈækʧuəli hɛlps ˈraɪmɪŋ ɪn ˈɪŋglɪʃ ɔːl ðæt mʌtʃ, æz ˈɪŋglɪʃ ruːlz ʌv prəˌnʌnsiˈeɪʃən ɑːr ɪˈsɛnʃəli roʊt-lɜrnd ˈɛniˌweɪ. ˈɪf ɪt wər nɑt roʊt-ˈlɜrnɪŋ, ðɛn ɪt maɪt meɪk ˈdɪfərənt mɪsˈteɪks, fɔr ɪgˈzæmpl̩ ɪkˈspɛktɪŋ tuː wɜrdz tuː raɪm bɪˈkɔz ðeɪ ɛnd wɪð ðə seɪm ˈsʌfɪks. pərˈhæps ɪt wʊd biː mɔr ɪˈfɛktɪv tuː ˈæsk ɪt tuː prəˈdus poʊəmz ɪn ðə ˈfɔrmæt: ˈɪŋglɪʃ0 ˈaɪpiːˈeɪ0 ˈɪŋglɪʃ1 ˈaɪpiːˈeɪ1, wɛr iʧ laɪn ɪz prəˈdusd ɪn boʊθ sɪˈmæntɪk ænd fəˈnɛtɪk ˌrɛprɪzɛnˈteɪʃənz. ðɪs wʊd gɪv ɪt ðə ˈkɑntɛkst ˈnɛsəˌsɛri tuː si ðə raɪmz wɪˈðaʊt ˈhævɪŋ tuː mɛs ɚˈaʊnd wɪð ðə ˌtoʊkənaɪˈzeɪʃən.
gwern 3y ago
I did some poking around with IPA way back in 2020 reasoning that if the phonetics were explicit maybe that'd be fine, but I didn't get anything that looked like a big improvement: https://gwern.net/gpt-3#ipa-rhyme-annotations https://gwern.net/gpt-3#ipa-rhyme-annotations My guess was that it doesn't see enough IPA to use it that way zero-shot, and the erasure of information by BPEs damages its learning fundamentally enough that you can't easily prompt your way into anything better. I speculated that because it's memorized so much, it shouldn't be too hard for it to learn to rhyme properly if you finetuned it on an IPA-encoded or non-BPE-tokenized poetry corpus, but I never got around to it and AFAIK no one else has tried that yet.

telotortium 3y ago

Can GPT accurately transcribe English text to IPA? It looks like it can (https://chat.openai.com/c/c893cf54-5dfa-4774-9800-e6a74b7f9e2e https://chat.openai.com/c/c893cf54-5dfa-4774-9800-e6a74b7f9e...): Prompt: Eng Rhyming with IPA User Transcribe the following English text to IPA: English: I wonder if having access to characters actually helps rhyming in English all that much, as English rules of pronunciation are essentially rote-learned anyway. If it were not rote-learning, then it might make different mistakes, for example expecting two words to rhyme because they end with the same suffix. Perhaps it would be more effective to ask it to produce poems in the format: English0 IPA0 English1 IPA1, where each line is produced in both semantic and phonetic representations. This would give it the context necessary to “see” the rhymes without having to mess around with the tokenization. IPA: Response: aɪ ˈwʌndər ɪf ˈhævɪŋ ˈæksɛs tuː ˈkærəktərz ˈækʧuəli hɛlps ˈraɪmɪŋ ɪn ˈɪŋglɪʃ ɔːl ðæt mʌtʃ, æz ˈɪŋglɪʃ ruːlz ʌv prəˌnʌnsiˈeɪʃən ɑːr ɪˈsɛnʃəli roʊt-lɜrnd ˈɛniˌweɪ. ˈɪf ɪt wər nɑt roʊt-ˈlɜrnɪŋ, ðɛn ɪt maɪt meɪk ˈdɪfərənt mɪsˈteɪks, fɔr ɪgˈzæmpl̩ ɪkˈspɛktɪŋ tuː wɜrdz tuː raɪm bɪˈkɔz ðeɪ ɛnd wɪð ðə seɪm ˈsʌfɪks. pərˈhæps ɪt wʊd biː mɔr ɪˈfɛktɪv tuː ˈæsk ɪt tuː prəˈdus poʊəmz ɪn ðə ˈfɔrmæt: ˈɪŋglɪʃ0 ˈaɪpiːˈeɪ0 ˈɪŋglɪʃ1 ˈaɪpiːˈeɪ1, wɛr iʧ laɪn ɪz prəˈdusd ɪn boʊθ sɪˈmæntɪk ænd fəˈnɛtɪk ˌrɛprɪzɛnˈteɪʃənz. ðɪs wʊd gɪv ɪt ðə ˈkɑntɛkst ˈnɛsəˌsɛri tuː si ðə raɪmz wɪˈðaʊt ˈhævɪŋ tuː mɛs ɚˈaʊnd wɪð ðə ˌtoʊkənaɪˈzeɪʃən.

gwern 3y ago

I did some poking around with IPA way back in 2020 reasoning that if the phonetics were explicit maybe that'd be fine, but I didn't get anything that looked like a big improvement: https://gwern.net/gpt-3#ipa-rhyme-annotations https://gwern.net/gpt-3#ipa-rhyme-annotations My guess was that it doesn't see enough IPA to use it that way zero-shot, and the erasure of information by BPEs damages its learning fundamentally enough that you can't easily prompt your way into anything better. I speculated that because it's memorized so much, it shouldn't be too hard for it to learn to rhyme properly if you finetuned it on an IPA-encoded or non-BPE-tokenized poetry corpus, but I never got around to it and AFAIK no one else has tried that yet.