6 ms·

Is it possible to use this (or Eleven Labs) to generate a voice model to plug into an Android phone's TTS? I have a friend with a paralysed larynx who is often

Is it possible to use this (or Eleven Labs) to generate a voice model to plug into an Android phone's TTS? I have a friend with a paralysed larynx who is often using his phone or a small laptop to type in order to communicate. I know he would love it if it was possible to take old recordings of him speaking and use that to give him back "his" voice, at least in some small measure.

Share6323 3y ago
That would be awesome
klankbrouwerij 3y ago
Your friend can take a look at solutions from Acapela [0], SpeakUnique [1] or VOCALiD [2]. Not sure whether they have a solution for Android though. I recently saw a video from google about a custom voice they created for somebody with ALS but I can't seem to find it online (Does anybody have a link?). Creating custom voices is not yet available on Android though. The latest iOS release (iOS 17) does support creating personalized voices. ModelTalker [3] is a long-term (research?) project to create custom voices for people with speech disabilities. Their TTS seem to support Android so that might be another option. [0] https://www.acapela-group.com/ https://www.acapela-group.com/ [1] https://www.speakunique.co.uk/ https://www.speakunique.co.uk/ [2] https://vocalid.ai/ https://vocalid.ai/ [3] https://www.modeltalker.org/ https://www.modeltalker.org/
- anotherevan 3y ago
  Hi and thanks for the suggestions. Looking through them, it looks like you need to do what they call "voice banking" before you lose your voice. Basically reading a script they provide. Unfortunately my friend's voice is too far gone for that to be possible. Hoping for something where they can use old recordings to generate a voice.
  - jokethrowaway 3y ago
    From my tests I think Audiobox from Meta is the most promising (even better than Eleven Labs) - too bad it's closed source and they force you to read some randomly generated sentences (to prevent the case of someone generating a cloned voice without consent). Right now Eleven Labs is your best bet. xTTS is just not there quality wise. The version available in the studio is marginally better than the OSS version but it's still pretty far from being believable. The non-nerfed version of Tortoise (the author decided to ruin their own project but forks exist) was decent at voice cloning but it takes a lot of tries. I'm pretty sure we already have the technology to do what you want and help your friend, it's just a matter of time until it gets better and more software comes out.
  - droopyEyelids 3y ago
    https://appleinsider.com/articles/23/11/30/apples-new-disability-tool-helps-users-with-disabilities-communicate-with-their-personal-voice https://appleinsider.com/articles/23/11/30/apples-new-disabi...
  - cjbprime 3y ago
    Some of the recent transformer models can work with audio clips just a few seconds long. I'm sure the final output is less good, but perhaps your friend has audio clips that would work for that from e.g. home movies.
refulgentis 3y ago
Alas, no (made some contributions to TTS at G, and worked on Android). iOS has this built in :/ which may bode well, there's no greater Google product manager than "whatever apple just shipped." I'm doing some xplatform on device inference stuff (see FONNX on GitHub) and it'll be one of 100 items that'll stick on my mind for a while, I hope I find time and I'll try to ping you Edit: is an Android app with a keyboard and "speak" button that does API calls to eleven labs sufficient for something worth trying?
- anotherevan 3y ago
  > I'll try to ping you Thanks. Use the email address in my profile if anything eventuates. > is an Android app with a keyboard and "speak" button that does API calls to eleven labs sufficient for something worth trying? Maybe. Obviously something with local processing would be preferred, but it might be an option when internet connectivity is good. Is there such an app?
  - refulgentis 3y ago
    There isn't an ElevenLabs app like that, but I think that's the most expedient method, by far. (i.e. O(days) instead of O(months)) (warning: detailed opinionated take, I suggest skimming) Why? Local inference is hard. You need two things: the clips to voice model (which we have here, but bleeding edge), and text + voice -> speech model. Text to voice to speech, locally, has excellent prior art for me, in the form of a Raspberry Pi-based ONNX inference library called [Piper](https://github.com/rhasspy/piper https://github.com/rhasspy/piper). I should just be able to copy that, about an afternoon of work! :P Except...when these models are trained, they encode plaintext to model input using a library called eSpeak. eSpeak is basically f(plaintext) => ints representing phonemes. eSpeak is a C library and written in a style I haven't seen in a while and depends on other C libraries. So I end up needing to port like 20K lines of C to Dart...or I could use WASM, but over the last year, I lost the ability to be able to reason through how to get WASM running in Dart, both native and web. Re: ElevenLabs I had looked into the API months ago and vaguely remembered it was _very_ complete. I spent the last hour or two playing with it, and reconfirmed that. They have enough API surface that you could build an app that took voice recordings, created a voice, and then did POSTs / socket connection to get audio data from that voice at will. Only issue is pricing IMHO, $0.18 for 1000 characters. :/ But this is something I feel very comfortable saying wouldn't be _that_ much work to build and open source with a "bring your own API key" type thing. I had forgotten about Eleven Labs till your post, which made me realize there was an actually meaningful and quite moving use case for it. All of Elevens advantages (cloning, peak quality by a mile) come into play, and the disadvantages are blunted: local voice cloning isn't there yet, and $0.18 / 1000 characters doesn't matter as much when it's interpersonal exchanges instead of long AI responses

Share6323 3y ago

That would be awesome

klankbrouwerij 3y ago

Your friend can take a look at solutions from Acapela [0], SpeakUnique [1] or VOCALiD [2]. Not sure whether they have a solution for Android though. I recently saw a video from google about a custom voice they created for somebody with ALS but I can't seem to find it online (Does anybody have a link?). Creating custom voices is not yet available on Android though. The latest iOS release (iOS 17) does support creating personalized voices. ModelTalker [3] is a long-term (research?) project to create custom voices for people with speech disabilities. Their TTS seem to support Android so that might be another option. [0] https://www.acapela-group.com/ https://www.acapela-group.com/ [1] https://www.speakunique.co.uk/ https://www.speakunique.co.uk/ [2] https://vocalid.ai/ https://vocalid.ai/ [3] https://www.modeltalker.org/ https://www.modeltalker.org/

anotherevan 3y ago
Hi and thanks for the suggestions. Looking through them, it looks like you need to do what they call "voice banking" before you lose your voice. Basically reading a script they provide. Unfortunately my friend's voice is too far gone for that to be possible. Hoping for something where they can use old recordings to generate a voice.
- jokethrowaway 3y ago
  From my tests I think Audiobox from Meta is the most promising (even better than Eleven Labs) - too bad it's closed source and they force you to read some randomly generated sentences (to prevent the case of someone generating a cloned voice without consent). Right now Eleven Labs is your best bet. xTTS is just not there quality wise. The version available in the studio is marginally better than the OSS version but it's still pretty far from being believable. The non-nerfed version of Tortoise (the author decided to ruin their own project but forks exist) was decent at voice cloning but it takes a lot of tries. I'm pretty sure we already have the technology to do what you want and help your friend, it's just a matter of time until it gets better and more software comes out.
- droopyEyelids 3y ago
  https://appleinsider.com/articles/23/11/30/apples-new-disability-tool-helps-users-with-disabilities-communicate-with-their-personal-voice https://appleinsider.com/articles/23/11/30/apples-new-disabi...
- cjbprime 3y ago
  Some of the recent transformer models can work with audio clips just a few seconds long. I'm sure the final output is less good, but perhaps your friend has audio clips that would work for that from e.g. home movies.

refulgentis 3y ago

Alas, no (made some contributions to TTS at G, and worked on Android). iOS has this built in :/ which may bode well, there's no greater Google product manager than "whatever apple just shipped." I'm doing some xplatform on device inference stuff (see FONNX on GitHub) and it'll be one of 100 items that'll stick on my mind for a while, I hope I find time and I'll try to ping you Edit: is an Android app with a keyboard and "speak" button that does API calls to eleven labs sufficient for something worth trying?

anotherevan 3y ago
> I'll try to ping you Thanks. Use the email address in my profile if anything eventuates. > is an Android app with a keyboard and "speak" button that does API calls to eleven labs sufficient for something worth trying? Maybe. Obviously something with local processing would be preferred, but it might be an option when internet connectivity is good. Is there such an app?
- refulgentis 3y ago
  There isn't an ElevenLabs app like that, but I think that's the most expedient method, by far. (i.e. O(days) instead of O(months)) (warning: detailed opinionated take, I suggest skimming) Why? Local inference is hard. You need two things: the clips to voice model (which we have here, but bleeding edge), and text + voice -> speech model. Text to voice to speech, locally, has excellent prior art for me, in the form of a Raspberry Pi-based ONNX inference library called [Piper](https://github.com/rhasspy/piper https://github.com/rhasspy/piper). I should just be able to copy that, about an afternoon of work! :P Except...when these models are trained, they encode plaintext to model input using a library called eSpeak. eSpeak is basically f(plaintext) => ints representing phonemes. eSpeak is a C library and written in a style I haven't seen in a while and depends on other C libraries. So I end up needing to port like 20K lines of C to Dart...or I could use WASM, but over the last year, I lost the ability to be able to reason through how to get WASM running in Dart, both native and web. Re: ElevenLabs I had looked into the API months ago and vaguely remembered it was _very_ complete. I spent the last hour or two playing with it, and reconfirmed that. They have enough API surface that you could build an app that took voice recordings, created a voice, and then did POSTs / socket connection to get audio data from that voice at will. Only issue is pricing IMHO, $0.18 for 1000 characters. :/ But this is something I feel very comfortable saying wouldn't be _that_ much work to build and open source with a "bring your own API key" type thing. I had forgotten about Eleven Labs till your post, which made me realize there was an actually meaningful and quite moving use case for it. All of Elevens advantages (cloning, peak quality by a mile) come into play, and the disadvantages are blunted: local voice cloning isn't there yet, and $0.18 / 1000 characters doesn't matter as much when it's interpersonal exchanges instead of long AI responses