54 ms·
BASE TTS: The largest text-to-speech model to-date
- unsupp0rted 3y ago> Echoing the widely-reported "emergent abilities" of Large Language Models when trained on increasing volume of data, we show that BASE TTS variants built with 10k+ hours start to exhibit advanced understanding of texts that enable contextually appropriate prosody.
- IronWolve 3y agoAwhile ago, when amazon had its text limited but unlimited free use of its neural tts, I was converting an ebook to audiobook, it was amazing how it could sound so lifelike and inflections of the voice. Amazing. Amazon really had the best sounding TTS I've seen compared to paid microsoft and google. Hands down better. But technology is getting better for opensource, I'd expect in a year or 2, home use will be on par in quality with paid services. I cant wait for realtime video translate, so shows with non-english subs can be translated into english speech. You can do it now with some services, upload a video and lang/voice/mouth will convert to any language.
- revenga99 3y agoWow. I could see this as threatening audio book narrators. However I would still prefer a real narrator to this in its current state. I think what it might be missing is different voices/accents for different characters.
- swashboon 3y agoAudible doesn't allow AI narration or much Public Domain stuff at the moment. The only thing keeping it from happening is the markets trying to keep back a flood of crap from over taking / drowning / diluting the more well crafted options and causing the consumers to get really annoyed.
- TOMDM 3y agoLet's be honest, the moment Amazon thinks their tts is good enough, they'll be offering AI audible deals to every author on their platform
- swashboon 3y agoYea, hard to say because the obvious implementation would be to just have it built into phones once the model is potentially portable enough - I see this happening quicker as a more general TTS functionality much like Google is doing with 'subtitles anywhere' aka Live Caption. Paired with translations we maybe pretty close to the universal translator type functionality. I could see end users being able to customize their voice assistant even more or maybe having multiple based on if its talking for you or to you. Anyways the problem with this is it makes the product 'ai audiobook' basically worthless, why not just buy the eBook and have my personalized translator turn it into an audio book. Now you just have market differentiation between cheap ebook + ai narrator vs expensive + professional narration. Though narration costs are already pretty cheap - it really does not factor into the cost of publishing an audio book that much unless its really a bottom of the barrel book.
- TOMDM 3y agoI'm looking forward to my on device TTS, but Amazon has a decent moat with the DRM on their Kindles. At least they'll have to remain somewhat competitive once consumers decide they want the AI audiobooks and the like.
- swashboon 3y agoThinking about this more - the copyright implications become much more interesting once its no longer a recording. Does it could as a private performance if you have headphones on? Is it a public performance if you listen to live TTS through your speakers in public?
- coredog64 3y agoThe 80% solution: Pair with a professional narrator who has consented to have their voice modeled by this (see the note at the bottom about what they held back from open sourcing). This generates a beta, and then you can pay the human narrator to rework specific sections you’re unhappy with.
- dshpala 3y agoI think Google's product has that: https://play.google.com/books/publish/autonarrated/ https://play.google.com/books/publish/autonarrated/
- pparanoidd 3y agoThat sounds pretty bad though
- dataminded 3y agoAs an avid consumer of audio books (150+/year) - we are well past the point where narrators are necessary. Professional audio books take too long to release, are too expensive, are concentrated on a limited number of platforms and just aren't THAT much better than the automated stuff for the long tail of books.
- geor9e 3y agoFolks probably will think me silly for this, but I prefer TTS. I have access to voice actor audiobooks but I pick the .epub files instead. I made a little extension to inject window.speechSynthesis with "Microsoft Steffan Online (Natural) - English (United States)" at rate=6 when I hit a hotkey. At high speed it's much clearer and natural sounding than a sped up voice actor recording.
- superkuh 3y agoI also prefer TTS. The spin voice actors put on the text always distracts me. With text to speech I only get what's in the text itself. I wrote a Perl/Tk GUI script for my file manager to manage text to speech through Festival 1.96 w/voice_nitech_us_awb_arctic_hts. Unlike neural network AI models it runs fine even on very slow machines.
- minimaxir 3y agoThe emotion examples are interesting. One of the current most obvious indicators of AI-generated voices/voice cloning is a lack of emotion and range, which make them objectively worse compared to professional voice actors, unless a lack of emotion and range is the desired voice direction. But if you listen to the emotion examples, the range essentially what you'd get from an audiobook narrator, not more traditional voice acting.
- tsumnia 3y agoSadly it's not my forte but I expect in the near future we'll see an additional "emotion" embedding or something similar. Actors regularly use 'action words' (verbs) [1] to help add context to lines. A model then could study a text, determine an appropriate verb/emotion range to work from, then produce the audio with that additional context. [1] https://indietips.com/subtext-action-verb/ https://indietips.com/subtext-action-verb/
- minimaxir 3y agoThe bottleneck is the annotations: there's no easy way to annotate "emotions" on the scale of data needed to have the model learn the necessary verbal tics. In contrast, image data on the intent for image generation models is very highly annotated in most cases.
- biomcgary 3y agoJust run an LLM in sentiment analysis mode to annotate.
- rhdunn 3y agoThat doesn't factor in line delivery. You can have the words say/mean one thing (e.g. "I'm fine.") and the delivery say/mean another (defensive, distraught, etc.). It also does not account for where stresses, emphasis, pauses, etc. are placed to enhance the delivery of a given text. How do you get sentiment analysis to properly annotate an audiobook that has a dramatic reading, or something akin to the narration of the Game of Thrones or Harry Potter books where the narrators switch characters, accents, manarisms to portray the written content?
- maxglute 3y agoAre there any decent TTS models that can be ran locally that plugs into existing software like SAPI without too much lag?
- dvt 3y agoBark and Tortoise work fairly well. Bark does super fast inference[1] on my M1. [1] https://github.com/SaladTechnologies/bark https://github.com/SaladTechnologies/bark
- modeless 3y agoXTTS has a streaming mode with ~300ms latency and sounds good, though it has hallucination issues. StyleTTS2 sounds good and doesn't hallucinate as much. It doesn't support streaming but it's fast so it can still respond quickly. But neither of them sound as good as Eleven Labs or OpenAI or this one.
- SparkyMcUnicorn 3y ago> ... capable of mimicking speaker characteristics with just a few seconds of reference audio ... we have decided against open-sourcing this model as a precautionary measure. Disappointed yet again.
- deleted 3y ago[deleted]
- someplaceguy 3y agoSomeone should send the developers this audio recording I have of Jeff Bezos saying that he changed his mind and wants the model to be released as open-source.
- mrfakename 3y agoSadly they didn't release the code or models
- CamperBob2 3y agoIt's for Your Own Good, don't you know
- chankstein38 3y agoI'm so glad they are all so protective of my safety! Lord knows I'm a child incapable of controlling myself or having my own morals! /s
- chankstein38 3y agoAgreed. It hardly feels worth even reading through the paper since, from my perspective, it may as well just be made up. I can also write "Hey guys I made a good TTS it's really cool and great and the voices sound really natural" and put some samples together. If I never release any code or models or anything, it may as well have not been published.
- Terretta 3y ago> really cool and great ... and put some samples together There are samples on the page which demonstrate it completely failing. Now as to whether you'd make that up is 4D chess.
- echelon 3y agoThe value of this stuff is going to zero. Don't worry about it. Product over model. Models and weights are a race to the bottom. Everyone is doing it and competing on data efficiency, methodology, MOS, etc. Groups all over are releasing their data and weights. It doesn't matter if Amazon doesn't, other labs will do it to get ahead and to get attention. This is going to be entirely pedestrian within a year. ElevenLabs is not a unicorn. It's an early-forming bubble.
- LarsDu88 3y agoSounds about as good as ElevenLabs.io Hopefully if this ships on AWS, it will support SSML tags. I used Elevenlabs.io for all the voices in my VR game (https://roguestargun.com https://roguestargun.com), but its still lacking on the emotion front which is all one-shot
- ghostbrainalpha 3y agoGame looks great. Are you supporting Flight Sticks?
- LarsDu88 3y agoEventually yes. Honestly I have joystick mappings setup in the games input configuration, but I no longer own a joystick or hotas, so somebody is gonna have to verify this for me. Gamedev ain't my day job, and the reality is most folks outside of hardcore flightsim enthusiasts don't own joysticks
- JanSt 3y agoI would love an API for this.. any information on availability?
- qwertox 3y agoInteresting. Just a couple of hours ago I came across MetaVoice-1B [0] (Demo [1]) and was amazed by the quality of their TTS in English (sadly no other languages available). If this year becomes the year when high quality Open Source TTS and ASR models appear that can run in real-time on an Nvidia RTX 40x0 or 30x0, then that would be great. On CPU even better. Also note the Ethical Statement on BASE TTS: > An application of this model can be to create synthetic voices of people who have lost the ability to speak due to accidents or illnesses, subject to informed consent and rigorous data privacy reviews. However, due to the potential misuse of this capability, we have decided against open-sourcing this model as a precautionary measure. [0] https://github.com/metavoiceio/metavoice-src https://github.com/metavoiceio/metavoice-src [1] https://ttsdemo.themetavoice.xyz/ https://ttsdemo.themetavoice.xyz/
- nshm 3y agoMetavoice is one of a dozen GPT-based TTS systems around starting from Tortoise. And not that great honestly. You can clearly hear "glass scratches" in their sound, it is because they trained on MP3-compressed data. There are much more clear sounding systems around. You can listen for StyleTTS2 to compare.
- qwertox 3y agoI had forgotten about StyleTTS2, and it was discussed here on HN a couple of months ago. Maybe that's what made me feel that there's something going on.
- ionwake 3y agoIs it possible to run Metavoice and other pytorch systems on Apple silicon EG the M1? I keep getting issues.
- popalchemist 3y agoI've tested both. StyleTTS2 is impressive, especially its speed, but the prosody is lacking, compared to Metavoice.
- standardly 3y ago
- nshm 3y agoErr, I deeply respect Amazon TTS team but this paper and synthesis is..... You publish the paper in 2024 and include YourTTS in your baselines to look better. Come on! There is XTTS2 around! Voice sounds robotic and plain. Most likely a lot of audiobooks in training data and less conversational speech. And dropping diffusion was not a great idea, voice is not crystal clear anymore, it is more like a telephony recording.
- thorum 3y agoxtts2 is great, but it looks like this model is probably more consistent with its output and has a better grasp of meaning in long texts.
- oersted 3y agoThe Spanish voice has an interesting accent: 85% Castillian (from Spain) pronunciation, with a few unexpected Latin American tonalities and phonemes (especially "s") sprinkled in. I guess it's what you'd expect from averaging a large amount of public-domain recordings. I think there's a bias towards Spain vs Latin America due to socioeconomic reasons, the population is obviously much smaller.
- dontreact 3y agoHow would socioeconomic factors lead to bias in a model? I figured there would be way more recordings in Latin American Spanish that u supervised learning would anchor on more
- sebmellen 3y agoOpen question: does anyone know of a TTS model which can synchronize the output to an SRT or other subtitle file?
- zamadatix 3y agoTo answer directly first: I don't know of any model with this built in. To answer more generally: but it should be pretty straightforward to use any old TTS model, the subtitle timestamps, and set the according delay until the next subtitle change and get the same effect. The alternative (changing the speed of the generated voice) is also possible via the same method but the problem there, and the problem when directly driven by a model, is subtitles don't clue you in on when e.g. someone is talking slow or there was a pause in conversation so that subtitle staid up a little longer than a normal one. What you'd need to solve that is a model which takes both the video and the subtitle info, a bit more difficult. Of course it's also a question about what the end goal is. It's pretty rare to have significant subtitles but no audio so if the ultimate goal was e.g. changing a actor's voice you'd probably get much better results with an audio->audio model than a TTS->audio model. Likely similar kinds of stories for many other use cases.
- solarized 3y agoFrom the ethical statement. > However, due to the potential misuse of this capability, we have decided against open-sourcing this model as a precautionary measure. Another irony. Elevenlabs had SaaS-ed this feature. I bet they'll jump on releasing this as SaaS ASAP. Money always trumps ethics, right?
- precompute 3y agoAh, so that's where all the Alexa recordings went.
- mrfakename 3y agoLooks like the website (amazon-ltts-paper.com) now redirects to amazon.science. They took out the "Ethical Statement" section. (The original page can still be accessed from the Wayback Machine: https://web.archive.org/web/20240215005705/https://amazon-ltts-paper.com/ https://web.archive.org/web/20240215005705/https://amazon-lt...)
- somesun 3y agois there any open sourced library can reach the quality of Microsoft tts and support multi-language