3 ms·

> Human voice inflection is hard to mimic because you have to have a contextual understanding of what is being said I think that's correct if we are talking ab

> Human voice inflection is hard to mimic because you have to have a contextual understanding of what is being said I think that's correct if we are talking about the author reading his book. But if we are using another person, they have no way to learn the correct vocalization other than using the text. And that's the same input data that the AI is dealing with, so it should be possible for them to be as good as non-authors reading the book.

monkpit 4y ago
I disagree. In the case of audiobooks, a language model might be able to store context of the story similar to a reader, but a good reader/voice actor will be able to add emphasis or inflection to enhance the story - and decide to do so based upon the story itself. The AI won’t have understanding of the story itself to know what is appropriate or not. Also, you say only the author could read the story in a certain way. This kind of implies that all audiobook recordings are just 1 long take, with no direction. I am not an expert, but I would assume there’s some creative direction involved.
- rafaelero 4y ago
  > but a good reader/voice actor will be able to add emphasis or inflection to enhance the story - and decide to do so based upon the story itself But why do you think this AI (VALL-E) isn't doing the same? The training procedure implies that it must follow correct intonation in order to achieve lower test loss. Also, if there is indeed something specific about audiobooks that is harder to replicate, then we can just fine-tune these models on audiobook. Accuracy will jump considerably.
  - monkpit 4y ago
    Because the intonation can depend on the events of the story, which it can’t be trained on.
timgilbert 4y ago
My intuition is that in the case of audiobook narrators who adopt different voices for different characters, VALL-E would struggle to identify which character is speaking and thus produce the correct voice for a given portion of, say, text in quotation marks. In some bits of prose it can be difficult for a human reader to determine who is speaking, for that matter, but this is a classic weakness with the current crop of AI tools, and stems from a lack of actual understanding of the text.

monkpit 4y ago

I disagree. In the case of audiobooks, a language model might be able to store context of the story similar to a reader, but a good reader/voice actor will be able to add emphasis or inflection to enhance the story - and decide to do so based upon the story itself. The AI won’t have understanding of the story itself to know what is appropriate or not. Also, you say only the author could read the story in a certain way. This kind of implies that all audiobook recordings are just 1 long take, with no direction. I am not an expert, but I would assume there’s some creative direction involved.

rafaelero 4y ago
> but a good reader/voice actor will be able to add emphasis or inflection to enhance the story - and decide to do so based upon the story itself But why do you think this AI (VALL-E) isn't doing the same? The training procedure implies that it must follow correct intonation in order to achieve lower test loss. Also, if there is indeed something specific about audiobooks that is harder to replicate, then we can just fine-tune these models on audiobook. Accuracy will jump considerably.
- monkpit 4y ago
  Because the intonation can depend on the events of the story, which it can’t be trained on.

timgilbert 4y ago

My intuition is that in the case of audiobook narrators who adopt different voices for different characters, VALL-E would struggle to identify which character is speaking and thus produce the correct voice for a given portion of, say, text in quotation marks. In some bits of prose it can be difficult for a human reader to determine who is speaking, for that matter, but this is a classic weakness with the current crop of AI tools, and stems from a lack of actual understanding of the text.