4 ms·

In CGI there were always these milestones which I observed getting reached. Like trees with leaves finally looking close to realistic, wind blowing in grass loo

In CGI there were always these milestones which I observed getting reached. Like trees with leaves finally looking close to realistic, wind blowing in grass looking almost realistic, hair, jelly, and it were usually Pixar shorts pointing out what they have been focusing on and then seeing it applied to their movies. Then mocap, mapping digital faces on real actors which was first mind-blowing to see in Pirates of the Caribbean, then the apes in one of the Planet of the Apes movies... So much in the CGI industry has already reached a point where the hardest problems seem to have been solved. When I now clicked play on the first Synthesized Dialoge from Dialogue Synthesis "Where did you go last summer? | I went to Greece, it was amazing.", I was blown away. It's as if we've now reached one of those milestones where a problem appears to be fixed or cracked. Machines will be able to really sound like humans, indistinguishable from them. 10-5 years ago, if you wanted to deal with TTS, the best option you had was to let your Android phone render a TTS into an audio file, because everything else sounded really bad. Specially Open Source stuff sounded absolutely horrible. So how long will it be until we will be able to download something of this quality onto a future-gen Raspberry Pi which can do some AI processing, where we make an HTTP call and it starts speaking through the audio out in a perfect voice without relying on the cloud? 5 years?

bckr 3y ago
I would bet 2 years tops
nine_k 3y ago
In you need a really compact form factor, you can buy a Jetson right now and run more complex models on it. It's pricey though.
amelius 3y ago
Another question, how long until we have systems that can sing 10 octaves and we don't need/want any actual human singers anymore?
- jayd16 3y ago
  People like to song along though.
  - tialaramex 3y ago
    People like playing drums too, but a drum machine means that if you're not any good at it or too busy but you need drum sounds you can have drum sounds. There are rights issues if the result is it replaces a particular singer, if you made it so that Sneaker Pimps can fire Kelli but still have her voice on subsequent songs that's a problem. But suppose you're a bedroom musician, and you realise you've got a piece that really wants somebody with a different voice than yours to make it work - you can pay someone, but technology like this offers a cheaper, easier option.
- ttul 3y ago
  As a choral singer, if there’s an app that one day allows me to sing with a fake choir of extremely good singers, I would enjoy doing that all day long. And it would allow my actual choir to practice way more, making our performances far better.
  - nraford 3y ago
    This exists right now! Not as an app exactly, but you should check out Holly Herndon and Mat Dryhurt’s suite of tools called “Holly Plus”: https://holly.plus/ https://holly.plus/ I’m pretty sure you can access their model somehow and even train your own voice using their “spawning” approach. She did an awesome TED talk demonstrating this: https://www.ted.com/talks/holly_herndon_what_if_you_could_sing_in_your_favorite_musician_s_voice?language=en https://www.ted.com/talks/holly_herndon_what_if_you_could_si... Here’s a cool example, using Dolly Parton’s song “Jolene”: https://www.youtube.com/watch?v=kPAEMUzDxuo https://www.youtube.com/watch?v=kPAEMUzDxuo I don’t think it’s quite at the level of consumer use yet, but I know they’re working on it. Definitely check it out.
- PaulDavisThe1st 3y ago
  To answer that question, it is necessary to answer the question "why did we want actual human singers in the past?" If you think it is because there was no alternative, then you will see no barriers to the adoption of the sort of system you imagine. But if you think it is because all human art is a form (however weird, tangled and opaque) of story telling, and for it to work, the person experiencing the art needs to be believe that there must be a story to be told, then your imagined system is not that interesting. However, we are seeing an overwhelming propensity even today for people (even people who ought to know better) to ascribe intentionality and even emotion to computer systems that demonstrably do not have it (and as far back as Eliza in the 1970s). It seems likely to me that most people will rapidly come to believe in a sufficiently intentional and emtional backstory to your putative wunder-singer that their belief will satisfy their desire for "a person behind the art".
JonathanFly 3y ago
>So how long will it be until we will be able to download something of this quality onto a future-gen Raspberry Pi which can do some AI processing, where we make an HTTP call and it starts speaking through the audio out in a perfect voice without relying on the cloud? 5 years? It's probably possible roughly whenever the larger Whisper models can run on it. Probably the next Raspberry Pi, running quantized or optimized versions of some audio model. It may be almost possible right now if you tried really realy hard, and you used a small model fine-tuned on a single voice, instead of something larger and more general purpose that can do any voice. I think whisper-tiny works on a Pi on real time, right? And that's not leveraging the GPU on the Pi. (https://github.com/ggerganov/whisper.cpp/discussions/166 https://github.com/ggerganov/whisper.cpp/discussions/166) Edit: looks like medium is 30x slower on the Pi than tiny model, so I may have been overly optimistic. I didn't realize Whisper tiny was that much faster than medium. This method works pretty well with Tortoise, letting you use the super fast Tortoise quality settings but get quality similar to the larger models. Fine-tuning the whole thing on just one voice removes a lot of the cool capabilities of course. With Tortoise, that would still be way too slow for a Pi but potentially that same strategy could work with faster models like SoundStorm. In terms of quality there's still a lot of room to go with long term coherence, like long audio segments. When a real person reads an audiobook the words at the top the page have a pretty big impact on how many words at the bottom the page are read. And there can be some impact at any distance, page 10 to page 300. When you try audiobooks on super high end TTS models and listen carefully you really notice the mismatch. It's like the reader recorded the paragraphs out of order, or a video game voice lines where you can tell the actors recorded all the lines separately, and were not reacting to each other's performance. You can bump the context windows, a minute, two minutes. That's gonna get you closer and probably good enough for some books. In the short term a human could simply adjust all the all the audio samples and manually tweak things to sound correct. So this will enable fan-created audiobooks where they take the time to get it right. But for fully automated books the mismatch drives me nuts. The performance is just soooo close for certain segments that when you get a tonal mismatch it hurts.
rossjudson 3y ago
Over the years I've also watched CGI progress and been amazed by the steps there -- the joy of seeing yet another complex part of reality rendered was a constant. These days, though...every new technique developed to simulate and replicate human creativity and behaviors builds on a constant sense of unease. If I view or read, do I have the right to know if it's generated?

bckr 3y ago

I would bet 2 years tops

nine_k 3y ago

In you need a really compact form factor, you can buy a Jetson right now and run more complex models on it. It's pricey though.

amelius 3y ago

Another question, how long until we have systems that can sing 10 octaves and we don't need/want any actual human singers anymore?

jayd16 3y ago
People like to song along though.
- tialaramex 3y ago
  People like playing drums too, but a drum machine means that if you're not any good at it or too busy but you need drum sounds you can have drum sounds. There are rights issues if the result is it replaces a particular singer, if you made it so that Sneaker Pimps can fire Kelli but still have her voice on subsequent songs that's a problem. But suppose you're a bedroom musician, and you realise you've got a piece that really wants somebody with a different voice than yours to make it work - you can pay someone, but technology like this offers a cheaper, easier option.
ttul 3y ago
As a choral singer, if there’s an app that one day allows me to sing with a fake choir of extremely good singers, I would enjoy doing that all day long. And it would allow my actual choir to practice way more, making our performances far better.
- nraford 3y ago
  This exists right now! Not as an app exactly, but you should check out Holly Herndon and Mat Dryhurt’s suite of tools called “Holly Plus”: https://holly.plus/ https://holly.plus/ I’m pretty sure you can access their model somehow and even train your own voice using their “spawning” approach. She did an awesome TED talk demonstrating this: https://www.ted.com/talks/holly_herndon_what_if_you_could_sing_in_your_favorite_musician_s_voice?language=en https://www.ted.com/talks/holly_herndon_what_if_you_could_si... Here’s a cool example, using Dolly Parton’s song “Jolene”: https://www.youtube.com/watch?v=kPAEMUzDxuo https://www.youtube.com/watch?v=kPAEMUzDxuo I don’t think it’s quite at the level of consumer use yet, but I know they’re working on it. Definitely check it out.
PaulDavisThe1st 3y ago
To answer that question, it is necessary to answer the question "why did we want actual human singers in the past?" If you think it is because there was no alternative, then you will see no barriers to the adoption of the sort of system you imagine. But if you think it is because all human art is a form (however weird, tangled and opaque) of story telling, and for it to work, the person experiencing the art needs to be believe that there must be a story to be told, then your imagined system is not that interesting. However, we are seeing an overwhelming propensity even today for people (even people who ought to know better) to ascribe intentionality and even emotion to computer systems that demonstrably do not have it (and as far back as Eliza in the 1970s). It seems likely to me that most people will rapidly come to believe in a sufficiently intentional and emtional backstory to your putative wunder-singer that their belief will satisfy their desire for "a person behind the art".

JonathanFly 3y ago

>So how long will it be until we will be able to download something of this quality onto a future-gen Raspberry Pi which can do some AI processing, where we make an HTTP call and it starts speaking through the audio out in a perfect voice without relying on the cloud? 5 years? It's probably possible roughly whenever the larger Whisper models can run on it. Probably the next Raspberry Pi, running quantized or optimized versions of some audio model. It may be almost possible right now if you tried really realy hard, and you used a small model fine-tuned on a single voice, instead of something larger and more general purpose that can do any voice. I think whisper-tiny works on a Pi on real time, right? And that's not leveraging the GPU on the Pi. (https://github.com/ggerganov/whisper.cpp/discussions/166 https://github.com/ggerganov/whisper.cpp/discussions/166) Edit: looks like medium is 30x slower on the Pi than tiny model, so I may have been overly optimistic. I didn't realize Whisper tiny was that much faster than medium. This method works pretty well with Tortoise, letting you use the super fast Tortoise quality settings but get quality similar to the larger models. Fine-tuning the whole thing on just one voice removes a lot of the cool capabilities of course. With Tortoise, that would still be way too slow for a Pi but potentially that same strategy could work with faster models like SoundStorm. In terms of quality there's still a lot of room to go with long term coherence, like long audio segments. When a real person reads an audiobook the words at the top the page have a pretty big impact on how many words at the bottom the page are read. And there can be some impact at any distance, page 10 to page 300. When you try audiobooks on super high end TTS models and listen carefully you really notice the mismatch. It's like the reader recorded the paragraphs out of order, or a video game voice lines where you can tell the actors recorded all the lines separately, and were not reacting to each other's performance. You can bump the context windows, a minute, two minutes. That's gonna get you closer and probably good enough for some books. In the short term a human could simply adjust all the all the audio samples and manually tweak things to sound correct. So this will enable fan-created audiobooks where they take the time to get it right. But for fully automated books the mismatch drives me nuts. The performance is just soooo close for certain segments that when you get a tonal mismatch it hurts.

rossjudson 3y ago

Over the years I've also watched CGI progress and been amazed by the steps there -- the joy of seeing yet another complex part of reality rendered was a constant. These days, though...every new technique developed to simulate and replicate human creativity and behaviors builds on a constant sense of unease. If I view or read, do I have the right to know if it's generated?