5 ms·

Will we see this level of voice synthesis in the public domain? Maybe I am out of touch but I found those examples very impressive - more impressive than the jo

Will we see this level of voice synthesis in the public domain? Maybe I am out of touch but I found those examples very impressive - more impressive than the jobs vs rogan demo a few months back. But I am also saddened at a future where all this is locked up in corporate hands - obviously there is money needed and (licensed) data needed too which Apple can get at. Honestly I would rather eschew the ethics of it and just consume any and all voice data (youtube, podcasts, existing audiobooks, radio) that has transcripts available, perhaps because I assume corpos are already doing this, if it means we can have a free and open data model that people can run at home, maybe that makes me evil.

rickdeckard 4y ago
My guess is that such "digital narration" is on the brink of becoming available as a service to authors and publishers, with Amazon and Apple trying to get ahead of it by selling a product that can only be published on their platforms then. They are surely able to undercut human narration, but even on digital narration they have an unfair advantage of earning a share on every sale as well. Will be interesting to see how this develops. Either independent digital narration becomes competitive enough that a publisher simply gets it done once and then sells it on all platforms, or this new platform-exclusive model is so disruptive that it becomes even less economic to produce a Audiobook, effectively making Audiobooks exclusive to Apple and Amazon/Audible (and whoever else has such a digital narration engine).
- patentatt 4y ago
  I'm curious what the economics are here, because getting someone to just read a book can't possibly be that expensive, right? I'm not talking celebrity voice-over work, but just getting anybody with a passable voice to sit down a spend a day or so reading a book into a microphone? Does that really cost that much more than having someone sit down a listen to the whole AI-generated audio book to do QA? And then there's all of the engineers who have to work on the project, the hardware to run all of it, etc. Seems like if it takes months to do and has to be QA'd anyways, it can't be that much more cost effective. Now, if it's completely computer-generated and can be a push-button feature? Sure, that makes sense. I wonder how close they are to that.
  - hidelooktropic 4y ago
    If you're talking about using a trained voice actor, yes, they will cost an hourly rate to do this, which is reflective of the care and training they put into their craft. One should also expect that they don't simply press record, and then do an entire read through. They will go back and try different takes on segments If we're not talking about a trained voice actor, you may want to look at some thing like LibriVox, which is entirely volunteer run on public domain works, and while the efforts are appreciated, the quality is noticeably different.
  - cianmm 4y ago
    You need to take into account that nobody can perfectly read a book first time. I've done a reasonable amount of short-fiction narration for audio magazines, and even with many hours worth of script reading under my belt I still fumble every third or forth sentence. So I need to re-read those, and then I (or somebody else) needs to find those errors and edit them out, replacing with the fixed audio. Then people need to listen to the entire file at least once to ensure the whole thing makes sense and I didn't leave out a sentence somewhere or something. Audiobook narration is one of those things that is remarkably labour intensive, certainly much more than I'd have guessed before getting into it.
    - vmilner 4y ago
      This was demonstrated vividly when Andy Serkis read The Hobbit unabridged live for charity then did a separate recording as a commercial release. Both were great, but you could see how many minor flaws were in the live recording that would have had to be re-recorded in the commercial version. (I’ve no idea how long it took to record the Lord of the Rings which he did a year later)
  - r00fus 4y ago
    As an audible customer I can tell you clearly that I often buy audiobooks by a particular narrator (discovering new authors) because the narrator is so good. I mean, Siri is quite good at reading texts (I imagine that's a huge training corpus) but I think we'll be in "uncanny valley" for quite a few years. It's possible that the public just gets used to that.
deleted 4y ago
[deleted]
nmfisher 4y ago
This is achievable with a very modest (say 50-100k) budget for voice actors and compute. Less if you’re happier with lower quality. Speech synthesis is probably one of the few areas in ML that’s trivially accessible to smaller orgs. Even Stable Diffusion was only 600k which is hardly outside the reach of a startup. The only ridiculously expensive models reserved for the big end of town are the GPT3 etc language models, and I fully expect the data/compute requirements to come down considerably in the near future.
pmontra 4y ago
Not from the companies selling audiobooks IMHO. They will use this technology to save money on human speakers. If they release it into the public domain we'll end up with ebooks that can read themselves aloud and they'll lose part of the incomes from audio books. My Samsung phone can read ebooks with one of Samsung's voices right now, but it does an awful job at pauses. Basically, no commas. With a good voice I could turn each one of my ebooks in an audiobook.
- raffraffraff 4y ago
  I don't really think I could listen to an AI voice reading a book. Perhaps some technical stuff, but not fiction. Even the difference between a mediocre vs good voice actor is huge, and can mean the difference between finishing an audio book or stopping after a chapter. Edit: to be very specific, a really good voice actor will take on different voices depending on which character is speaking, and will act out scenes realistically. I honestly can't imagine any AI being able to do that.
kaba0 4y ago
There is https://commonvoice.mozilla.org/en https://commonvoice.mozilla.org/en though I’m not sure where and how is it being used.
- jonathankoren 4y ago
  Common Voice is more about building a dataset or how people talk, especially with accents. While there is/was a voice synthesis project at Mozilla it was rudimentary like 3 years ago
enlyth 4y ago
This is already easily achievable on consumer hardware, you can train something like Tacotron 2 + WaveRNN on your own computer to achieve similar, if not better results. Check out this repo: https://github.com/coqui-ai/TTS https://github.com/coqui-ai/TTS You can also clone someone's voice by finetuning a pretrained LJSpeech model and training a vocoder from scratch, I've had great success with as little as 15 minutes of speech.
- davidzweig 4y ago
  Check my other comment in this thread, you might try our dataset. :)
  - enlyth 4y ago
    Will check it out, thanks
- GordonS 4y ago
  > You can also clone someone's voice by finetuning a pretrained LJSpeech model and training a vocoder from scratch, I've had great success with as little as 15 minutes of speech. Are you able to point to any articles to help get started with this please?
  - enlyth 4y ago
    Unfortunately, I'm not aware of any beginner friendly tutorials. The way I learned it was just by experimenting with various GitHub repositories (e.g. https://github.com/fatchord/WaveRNN https://github.com/fatchord/WaveRNN or the one I linked earlier) but it takes a lot of trial and error. Might do a writeup at some point if I have time.
- rockemsockem 4y ago
  It really isn't though. The level of fidelity that Apple is demonstrating in those samples is very impressive. You can generate fine voices with little work using repos like that one, but to get to the level Apple has takes a lot of work. EDIT: "fine voices" not "find voices"
criddell 4y ago
> saddened at a future where all this is locked up in corporate hands I would guess this rolls out from big companies first because the first version is always the most difficult. It’s only going to get easier to do and I would totally expect end-user controlled TTS systems to get better and eventually exceed the capabilities of this version from Apple. Of course Apple isn’t going to sit still, so they will continue to improve as well. Are there examples from ten or twenty years ago of a technology that big companies had locked up that never made it out to end users? What we have might lag, but it seems like this stuff only ever gets easier to do.
- abraxas 4y ago
  Linux is a prime example of a technology that was at first far behind its proprietary counterparts but eventually dominated and nearly extinguished all non-free competitors.
  - FinalBriefing 4y ago
    For very specific uses. Linux is not a good options for general computers used by everyday people. That experience is still owned by large corporations.
    - fsflover 4y ago
      There are only two cases when you can't use Linux as you daily driver: * You must use proprietary software not supporting Linux * You must use a device with proprietary drivers not supporting Linux For me, it works flawlessly, since I avoid both cases.
- wahnfrieden 4y ago
  as AI requires larger and larger data sets and processing power (energy, money), how will public domain catch up to their wealth accumulation? it's not like linux where chipping away at functionality and incremental UX adds up sufficiently over time
carb 4y ago
You can play with it on https://uberduck.ai/ https://uberduck.ai/ and they have a very active Discord!
- idle_zealot 4y ago
  What exactly is "Open Source" about uberduck? It looks like a proprietary tts saas to me; no links to a git repo and the "developer" section just shows how to get an API key and hit their service.

rickdeckard 4y ago

My guess is that such "digital narration" is on the brink of becoming available as a service to authors and publishers, with Amazon and Apple trying to get ahead of it by selling a product that can only be published on their platforms then. They are surely able to undercut human narration, but even on digital narration they have an unfair advantage of earning a share on every sale as well. Will be interesting to see how this develops. Either independent digital narration becomes competitive enough that a publisher simply gets it done once and then sells it on all platforms, or this new platform-exclusive model is so disruptive that it becomes even less economic to produce a Audiobook, effectively making Audiobooks exclusive to Apple and Amazon/Audible (and whoever else has such a digital narration engine).

patentatt 4y ago
I'm curious what the economics are here, because getting someone to just read a book can't possibly be that expensive, right? I'm not talking celebrity voice-over work, but just getting anybody with a passable voice to sit down a spend a day or so reading a book into a microphone? Does that really cost that much more than having someone sit down a listen to the whole AI-generated audio book to do QA? And then there's all of the engineers who have to work on the project, the hardware to run all of it, etc. Seems like if it takes months to do and has to be QA'd anyways, it can't be that much more cost effective. Now, if it's completely computer-generated and can be a push-button feature? Sure, that makes sense. I wonder how close they are to that.
- hidelooktropic 4y ago
  If you're talking about using a trained voice actor, yes, they will cost an hourly rate to do this, which is reflective of the care and training they put into their craft. One should also expect that they don't simply press record, and then do an entire read through. They will go back and try different takes on segments If we're not talking about a trained voice actor, you may want to look at some thing like LibriVox, which is entirely volunteer run on public domain works, and while the efforts are appreciated, the quality is noticeably different.
- cianmm 4y ago
  You need to take into account that nobody can perfectly read a book first time. I've done a reasonable amount of short-fiction narration for audio magazines, and even with many hours worth of script reading under my belt I still fumble every third or forth sentence. So I need to re-read those, and then I (or somebody else) needs to find those errors and edit them out, replacing with the fixed audio. Then people need to listen to the entire file at least once to ensure the whole thing makes sense and I didn't leave out a sentence somewhere or something. Audiobook narration is one of those things that is remarkably labour intensive, certainly much more than I'd have guessed before getting into it.
  - vmilner 4y ago
    This was demonstrated vividly when Andy Serkis read The Hobbit unabridged live for charity then did a separate recording as a commercial release. Both were great, but you could see how many minor flaws were in the live recording that would have had to be re-recorded in the commercial version. (I’ve no idea how long it took to record the Lord of the Rings which he did a year later)
- r00fus 4y ago
  As an audible customer I can tell you clearly that I often buy audiobooks by a particular narrator (discovering new authors) because the narrator is so good. I mean, Siri is quite good at reading texts (I imagine that's a huge training corpus) but I think we'll be in "uncanny valley" for quite a few years. It's possible that the public just gets used to that.

deleted 4y ago

[deleted]

nmfisher 4y ago

This is achievable with a very modest (say 50-100k) budget for voice actors and compute. Less if you’re happier with lower quality. Speech synthesis is probably one of the few areas in ML that’s trivially accessible to smaller orgs. Even Stable Diffusion was only 600k which is hardly outside the reach of a startup. The only ridiculously expensive models reserved for the big end of town are the GPT3 etc language models, and I fully expect the data/compute requirements to come down considerably in the near future.

pmontra 4y ago

Not from the companies selling audiobooks IMHO. They will use this technology to save money on human speakers. If they release it into the public domain we'll end up with ebooks that can read themselves aloud and they'll lose part of the incomes from audio books. My Samsung phone can read ebooks with one of Samsung's voices right now, but it does an awful job at pauses. Basically, no commas. With a good voice I could turn each one of my ebooks in an audiobook.

raffraffraff 4y ago
I don't really think I could listen to an AI voice reading a book. Perhaps some technical stuff, but not fiction. Even the difference between a mediocre vs good voice actor is huge, and can mean the difference between finishing an audio book or stopping after a chapter. Edit: to be very specific, a really good voice actor will take on different voices depending on which character is speaking, and will act out scenes realistically. I honestly can't imagine any AI being able to do that.

kaba0 4y ago

There is https://commonvoice.mozilla.org/en https://commonvoice.mozilla.org/en though I’m not sure where and how is it being used.

jonathankoren 4y ago
Common Voice is more about building a dataset or how people talk, especially with accents. While there is/was a voice synthesis project at Mozilla it was rudimentary like 3 years ago

enlyth 4y ago

This is already easily achievable on consumer hardware, you can train something like Tacotron 2 + WaveRNN on your own computer to achieve similar, if not better results. Check out this repo: https://github.com/coqui-ai/TTS https://github.com/coqui-ai/TTS You can also clone someone's voice by finetuning a pretrained LJSpeech model and training a vocoder from scratch, I've had great success with as little as 15 minutes of speech.

davidzweig 4y ago
Check my other comment in this thread, you might try our dataset. :)
- enlyth 4y ago
  Will check it out, thanks
GordonS 4y ago
> You can also clone someone's voice by finetuning a pretrained LJSpeech model and training a vocoder from scratch, I've had great success with as little as 15 minutes of speech. Are you able to point to any articles to help get started with this please?
- enlyth 4y ago
  Unfortunately, I'm not aware of any beginner friendly tutorials. The way I learned it was just by experimenting with various GitHub repositories (e.g. https://github.com/fatchord/WaveRNN https://github.com/fatchord/WaveRNN or the one I linked earlier) but it takes a lot of trial and error. Might do a writeup at some point if I have time.
rockemsockem 4y ago
It really isn't though. The level of fidelity that Apple is demonstrating in those samples is very impressive. You can generate fine voices with little work using repos like that one, but to get to the level Apple has takes a lot of work. EDIT: "fine voices" not "find voices"

criddell 4y ago

> saddened at a future where all this is locked up in corporate hands I would guess this rolls out from big companies first because the first version is always the most difficult. It’s only going to get easier to do and I would totally expect end-user controlled TTS systems to get better and eventually exceed the capabilities of this version from Apple. Of course Apple isn’t going to sit still, so they will continue to improve as well. Are there examples from ten or twenty years ago of a technology that big companies had locked up that never made it out to end users? What we have might lag, but it seems like this stuff only ever gets easier to do.

abraxas 4y ago
Linux is a prime example of a technology that was at first far behind its proprietary counterparts but eventually dominated and nearly extinguished all non-free competitors.
- FinalBriefing 4y ago
  For very specific uses. Linux is not a good options for general computers used by everyday people. That experience is still owned by large corporations.
  - fsflover 4y ago
    There are only two cases when you can't use Linux as you daily driver: * You must use proprietary software not supporting Linux * You must use a device with proprietary drivers not supporting Linux For me, it works flawlessly, since I avoid both cases.
wahnfrieden 4y ago
as AI requires larger and larger data sets and processing power (energy, money), how will public domain catch up to their wealth accumulation? it's not like linux where chipping away at functionality and incremental UX adds up sufficiently over time

carb 4y ago

You can play with it on https://uberduck.ai/ https://uberduck.ai/ and they have a very active Discord!

idle_zealot 4y ago
What exactly is "Open Source" about uberduck? It looks like a proprietary tts saas to me; no links to a git repo and the "developer" section just shows how to get an API key and hit their service.