10 ms·
SoundStorm: Efficient Parallel Audio Generation
- mg 3y agoI wonder if work marketplaces like UpWork and Fiverr will adapt quickly enough to this new situation, where many of their services, which in the past were done by humans, can now be done by software. Their current marketplace interface seems inadequate for this. Instead of contacting a human and then wait for them to finish the work, buyers will want to get results right away. Therefore they will have to change their platform to work like an app store. Where the sellers connect their services and buyers can use these services.
- throw47474777j 3y agoWhy wouldn't people just use existing software markets?
- mg 3y agoFor example?
- throw47474777j 3y agoApp Stores, the web, etc. How else does software as a service get sold? It’s not a new thing. Probably a lot of these things will just end up as features in existing systems.
- mg 3y agoExisting appstores like the ones on iOS and Android mostly target casual use cases, mobile devices and on-device software. Not "buy once" experiences for work via software as a service. They also do not offer a unified experience. Two "text-to-speach" apps could have completely different user interfaces. The web does not have good discovery and reputation management and also does not provide a unified interface. That is why market places like Booking.com, Amazon, Spotify etc have become so big.
- Legend2440 3y agoWhy does everybody focus on "how will this replace humans?" It's just a really good text-to-speech.
- ImHereToVote 3y agoPersonally I can't wait for all the streets to be lined with the homeless like in SF. So good.
- akaij 3y agoIt's kinda sad to see you believe that this is the inevitable outcome.
- pjmlp 3y agoWell, if we imagine that the only thing that will be left are physical jobs that can't be done by computers. At least until they get clever enough to start a transformers line factory.
- Legend2440 3y agoThis is the lump of labor fallacy. It's not about "what jobs will be left", it's about the new jobs we'll invent with all the time we'll have on our hands. There was never a fixed number of jobs, there's a fixed number of workers.
- pjmlp 3y agoWell, we can also return to feudalism.
- ImHereToVote 3y agoI didn't think you understand. We aren't about to automate all things humans can "currently" do. We are about to automate everything that separates us from a brick. What can possibly be left, and why would an unemployed person pay for whatever what is left?
- seydor 3y ago> where many of their services, which in the past were done by humans, can now be done by software. Their users are already using AI to do the work that they are supposed to do. i think that's fine
- bleepblop 3y agoWhere I see the benefit is in dialogue replacement. It takes a lot of time to call back an actor to the stage after they have finished the project. They might have moved onto another project or depending on how in demand they are they have a tight schedule and you're time with them is limited. Then on top of all that, some actors are just not very good at the process. So you have them in a room for a long time trying to get the right performance; sometimes which is difficult because that performance was done in some type of specific environment. Having a tool at my fingertips where I can feed it some of the actor's previous lines and be able to belch out something to fill the gaps with set parameters and be able to move along in the project without all the logistics would be heaven. It would however, kill an entire field of expertise. It would also devalue the actor as well. Though it's already happening. There are already programs on the market that replace voice actors all together and are being used in the video game space. For the work I do, I can see the benefits it could bring. But I am also fully aware it probably will be heavily abused.
- famouswaffles 3y agoIt's good that Bing, Bard are using the latest Microsoft, Google Cloud offerings but it would be nice to see these speech advances (along with audio palm - https://google-research.github.io/seanet/audiopalm/examples/ https://google-research.github.io/seanet/audiopalm/examples/ etc) hit public api's and/or user interfaces. Bard's TTS is alright but it's clearly behind. On that note, Bing's English/Korean TTS is really good. I also didn't realize Microsoft uses the best offerings for free TTS on edge so it blows google's default tts voices away.
- ShamelessC 3y ago> public api's and/or user interfaces sigh. Google used to release _some_ models. Guess the fun early days are coming to an end.
- Legend2440 3y agoGoogle is a business and this is clearly a valuable product.
- ShamelessC 3y agoSure, but there was a time not too long ago when companies were still in the "good will" phase of handing out even highly valuable models like CLIP, guided-diffusion, etc. Come to think, it was mostly OpenAI doing this. And they kinda still do? But far more selectively. I'm just preemptively romanticizing that.
- rasz 3y agoProduct is something you sell to make money. The only real Google product is users sold to advertisers.
- vore 3y agoUh, what about all of their paid cloud offerings?
- JonathanFly 3y agoInteresting that SoundStorm was trained to produce dialog between two people using transcripts annotated with '|' marking changes in voice. But the exact same '|' characters seem to mostly work in the Bark model out of the box and also produce a dialog? Maybe a third or a bit more of Bark outputs are a dialog person talking to themselves -- and it often misses a voice change. But the pipe characters do reliably produce audio that sounds like a dialog in the performance style. https://twitter.com/jonathanfly/status/1675987073893904386 https://twitter.com/jonathanfly/status/1675987073893904386 Is there some text-audio data somewhere in the training data that uses | for voice changes? Amusingly, Bark tends to render the SoundStorm prompts sarcastically. Not sure if that's a difference in style in the models, or just Google cherry picking the more straightforward line readings as the featured samples.
- famouswaffles 3y agoThe creators won't say as far as i know but bark looks to be trained on lot of youtube corpora (rather than typical ML audio datasets) where audio may have transcripts like that and why stuff like [laughs] work
- JonathanFly 3y agoYeah I often try to think about what might be in a YouTube caption when finding prompts that work in Bark. But pipe character isn't one I remember seeing on YouTube. Maybe it's part of some other audio dataset though. Or maybe it's on YouTube but only in non English videos.
- neilv 3y agoIn the future, will children think it's normal to talk like, "Hey, what up, Youtube! ... Be sure to like and subscribe! ... Smash that like button! ... Let me know in the comments down below!"? I wonder how ML trained on the tone transitions to a sponsored segment dripping with secret shame... would infect general speech.
- paulmd 3y ago
- tagyro 3y agoI've wasted (counting) about 300 seconds of my life listening to these audio files and they all sound and seem fake...
- tagyro 3y agojust to be clear, one could mistake them for some (voice) actor reading a book (maybe) but even to my untrained ear they sound fake and artificial. Am i missing something?
- kvn8888 3y agoIt's meant to sound artificial. The focus is on speed and consistency
- svantana 3y agoI found that in my (high quality) studio monitors, the audio sounded fine and hard to distinguish from 24kHz wav. But in headphones, the artifacts were pretty obvious. So probably some reverberation will do a lot to cover up artifacts. In the paper, they only do a subjective comparison between the generated audio and the soundstream-encoded original audio, which seems a bit disingenuous. Listening to soundstream audio in headphones, I can hear those same artifacts.
- jeffbee 3y agoDid you read the paper? They intentionally steered the quality to ensure they sound fake. Their generated speech is "very easy to detect" according to the reference at the end of the paper.
- butz 3y agoWith all recent advances, are there any decent TTS voices for Linux that are not complicated to set up for regular user?
- anigbrowl 3y agoGood for fraudsters and spammers, bad for anyone who ever hoped to make a living from voice acting. I'm perplexed by AI technologists' seemingly incessant drive to automate away the existence of artistic performers.
- croes 3y agoWhy spare artists if everyone else gets replaced by technology?
- anigbrowl 3y agoThey don't, otherwise there would be many former CEOs living in tents. In reality, those who control large amounts of capital are quite willing (and increasingly, say so in the open) to to deprive others of their livelihoods, homes, and ability to feed themselves in order to realize a marginal increase in their own wealth.
- brucethemoose2 3y agoThe function of CEOs and board members are closer to being replaced than, say, roofers or garbage workers. It will be interesting to see what happens when that pressure is applied...
- krapp 3y agoCapitalism will never allow capitalists to be made redundant - that's not how it works.
- brucethemoose2 3y agoIt will if its more profitable, which it is.
- Legend2440 3y agoYou are being deliberately pessimistic. There are a million fantastic, practical uses for text-to-speech.
- qwertox 3y agoIn CGI there were always these milestones which I observed getting reached. Like trees with leaves finally looking close to realistic, wind blowing in grass looking almost realistic, hair, jelly, and it were usually Pixar shorts pointing out what they have been focusing on and then seeing it applied to their movies. Then mocap, mapping digital faces on real actors which was first mind-blowing to see in Pirates of the Caribbean, then the apes in one of the Planet of the Apes movies... So much in the CGI industry has already reached a point where the hardest problems seem to have been solved. When I now clicked play on the first Synthesized Dialoge from Dialogue Synthesis "Where did you go last summer? | I went to Greece, it was amazing.", I was blown away. It's as if we've now reached one of those milestones where a problem appears to be fixed or cracked. Machines will be able to really sound like humans, indistinguishable from them. 10-5 years ago, if you wanted to deal with TTS, the best option you had was to let your Android phone render a TTS into an audio file, because everything else sounded really bad. Specially Open Source stuff sounded absolutely horrible. So how long will it be until we will be able to download something of this quality onto a future-gen Raspberry Pi which can do some AI processing, where we make an HTTP call and it starts speaking through the audio out in a perfect voice without relying on the cloud? 5 years?
- bckr 3y agoI would bet 2 years tops
- nine_k 3y agoIn you need a really compact form factor, you can buy a Jetson right now and run more complex models on it. It's pricey though.
- amelius 3y agoAnother question, how long until we have systems that can sing 10 octaves and we don't need/want any actual human singers anymore?
- jayd16 3y agoPeople like to song along though.
- elAhmo 3y agoThis is nothing short of amazing. It is exciting, a bit scary as well, what the future will bring. It just makes me sad that I cannot open this page on Safari. It will not play a single audio, yet Chrome plays it fine. So here we are, able to generate audio, video, code, do amazing things with AI, but a simple website that has text and audio is not working on the most popular laptop out there.
- lern_too_spel 3y agoIf you want to use the modern multimedia web, you can't use Lynx or Safari.
- elAhmo 3y agoThey are not comparable. I understand that some applications might prefer "other browsers", but a simple page that plays audio snippets was really a disappointment.
- nwoli 3y agoSeems like we wouldn’t be far at all from just correlating this to face movement (including subtle iris movement and blinks, not just the mouth). As long as you clearly label it as CGI it’s harmless and I’m excited for the day to come. Might be quite fun to chat with a little buddy this way
- asutekku 3y agoThe most impressive part of this is that they are seemingly able to produce 30 seconds of TTS with just 3 seconds of source material. That is super cool and honestly much more further in the curve that I expected it to be.
- deleted 3y ago[deleted]
- binary132 3y agoWhen people wax eloquent about how the artisans will just find something new to do for work, what they fail to mention is that the new work is often a menial and lower-paid job. When Amazon puts mom and pop shops out of business, they don’t go start new businesses, they go get jobs at Wal-Mart.
- willemmerson 3y agoI don't have anything intelligent to say about this but it's ALOT of fun making all the samples play at the same time - sort of like the HTML version of Ableton Live.
- squarefoot 3y agoImpressive, although the first example fails at the very last second when the virtual guy's voice glides up while saying "what?" and the autotune-ish pitch correction effect is clearly audible. Other examples are nothing short of incredible. If it really can output minutes of credible voice just by training it with a few seconds, next step will be to make it sing. I foresee a legal storm approaching when someone will use a similar technology to put for example Elvis' voice -without naming him- in some advertising, so that fans would recognize him but the audio wouldn't match neither any of his lyrics nor any existing songs by him.
- linhns 3y agoI don't believe the first example is created by Soundstorm. It sounds coercive and modulated to me.
- bozhark 3y agoClose. Too flat of a sound overall.
- varunpant 3y agohttps://github.com/lucidrains/soundstorm-pytorch https://github.com/lucidrains/soundstorm-pytorch
- deleted 3y ago[deleted]
- up2isomorphism 3y agoNo, I am not interested play a game generated this way because the exact reason I want to listen to NPC conversations is because they are written by human.
- mgaunard 3y agoWhy would you choose to synthesize audio using such awful valley girl accents?
- globular-toast 3y agoThat name is a blast from the past. Anyone remember Nvidia SoundStorm?