15 ms·
VibeVoice: A Frontier Open-Source Text-to-Speech Model
- Havoc 1y agoMIT license - very nice!
- em-bee 1y agowhat does that mean in this context? it seems to depend on an LLM. so can i run this completely offline? if i have to sign up and pay for an LLM to make it work, then it's not really more useful than any other non-free system
- ComputerGuru 1y agoThe application of known FOSS licenses to what is effectively a binary-only release is misleading and borderline meaningless.
- Havoc 1y agoIt is an unfortunate recycling of an existing regime that no doubt offends Stallman to his very core, but I wouldn't call it meaningless. If you're in a company and need a model which one do you think you're getting past compliance & legal - the one that says MIT or the one that says "non-commercial use only"?
- deleted 1y ago[deleted]
- watsonmusic 1y agoMicrosoft is cool
- egorfine 1y ago[deleted - I'm an idiot]
- x187463 1y agoWhisper is speech-to-text. VibeVoice is text-to-speech.
- mpeg 1y agoThere is a text-to-speech version of whisper, but IMHO the quality is much worse than the demos of this model.
- x187463 1y agoAre you referring to this? https://github.com/WhisperSpeech/WhisperSpeech https://github.com/WhisperSpeech/WhisperSpeech Or is there some OpenAI official Whisper TTS?
- mpeg 1y agoYep, nothing official that I know, but that one is fairly popular so maybe they were referring to it (although AFAIK it's not frontier?)
- egorfine 1y agoI stand corrected
- baal80spam 1y agoWow. I admit that I am not a native speaker, but this looks (or rather, sounds) VERY impressive and I could mistake it for hearing two people talking.
- x187463 1y agoThe giveaway is they will never talk over each other. Only one speaker at a time, consistently.
- tracker1 1y agoFair enough... though it would be possible to generate that and edit to overlay the speech, introducing stuttering/pauses at the beginning and end of statements then edit the output to overlay the steps. Would probably want to do similar to balance crossfade anyway... having each speaker's input offset from center instead of straight mono.
- kaptainscarlet 1y agoAlso the lack of stutter and perfect flow of speech are a dead giveaway
- kridsdale1 1y agoAnd longer pause between turns than humans would do.
- tracker1 1y agoYeah, a lot of the TTS has gotten really impressive in general. Definitely a clear leap from the TTS stuff I worked with for training simulations a bit over a decade ago. Aside: Installing a sound card (unused) on a windows server just to be able to generate TTS was interesting. It was required by the platform, even if it wasn't used for it. I generally don't like a lot of the AI generated slop that's starting to pop up on YouTube these days... I do enjoy some of the reddit story channels, but have completely stopped with it all now. With the AI stuff, it really becomes apparent with dates/ages and when numbers are spoken. Dates/ages/timelines are just off as far as story generation, and really should be human tweaked. As to the voice gen, saying a year or measurement is just not how English speakers (US or otherwise) speak.
- wewewedxfgdf 1y agoI'm really hoping one day there will be TTS does that does really nice British accents - I've surveyed them all deeply, none do. Most that claim to do a British accent end up sounding like Kelsey Grammer - sort of an American accent pretending to be British.
- rafaelmn 1y agoThe Spontaneous Emotion dailog sounds like a team member venting through LLMs. They could have skipped the singing part, it would be better if the model did not try to do that :)
- eibrahim 1y agoHahahah. Thats what I thought too
- kridsdale1 1y agoIt did get me to look up the song [1] again though, which is a great stimulator of emotion. The robot singing has a long way to go. 1. https://music.youtube.com/watch?v=xl8thVrlvjI&si=dU6aIJIPWSsb_pOv https://music.youtube.com/watch?v=xl8thVrlvjI&si=dU6aIJIPWSs...
- malnourish 1y agoThis is clearly high quality but there's something about the voices, the male voices in particular, which immediately register as computer generated. My audio vocabulary is not rich enough to articulate what it is.
- lvncelot 1y agoAfter hearing them myself, I think I know what you mean. The voices get a bit warbly and sound at times like they are very mp3-compressed.
- heeton 1y agoI'm no audio engineer either, but those computer voice sound "saw-tooth"y to me. From what I understand, it's more basic models/techniques that are undersampling, so there is a series of audio pulses which give it that buzzy quality. Better models are produced smoother output. https://www.perfectcircuit.com/signal/difference-between-waveforms https://www.perfectcircuit.com/signal/difference-between-wav...
- codebastard 1y agoI would describe it as blockly, as if we visualise the sound wave it seems to be without peaks and cut upwards and downwards producing a metallic boxy echo.
- jofzar 1y agoYeah it sounds super low bitrate to me, reminds me of someone on Bluetooth microphone
- glenstein 1y agoVery good and I could see how I might believe they are real people if I let my guard down. The male voice sounded a little sedated though and there was a smoothness to it that could be samey over long stretches. Still not at the astonishing level of Google Notebook text to speech which has been out for a while now. I still can't believe how good that one is.
- throwaw12 1y agoWill there be a support for SSML to have more control of conversation?
- giancarlostoro 1y agoI really hope someone within Microsoft is naming their open source coding agent Microsoft VibeCode. Let this be a thing. Its either that or "Lo" then you can have Lo work with Phi, so you can Vibe code with Lo Phi. https://techcommunity.microsoft.com/blog/azure-ai-foundry-blog/introducing-phi-4-microsoft%E2%80%99s-newest-small-language-model-specializing-in-comple/4357090 https://techcommunity.microsoft.com/blog/azure-ai-foundry-bl...
- watsonmusic 1y agogenius
- simiones 1y agoKnowing the history of Microsoft marketing, it will either be called something like "Microsoft Copilot Code Generator for VSCode" or something like "Zunega"...
- giancarlostoro 1y agoWell don't forget "Microsoft SQL" ;) They'll name something as though they invented it and now have the worse possible way to google it.
- loloquwowndueo 1y ago“Microsoft Word” haha reminds me of the old joke : “Microsoft Works” is an oxymoron.
- giancarlostoro 1y agoOh my goodness, I forgot about "Microsoft Works" you just shot me back in time to the 2000s
- esafak 1y agoYou misquoted Microsoft "Works"
- 1y ago
- regularfry 1y agoOk, this is nit-picking, but it's very obvious that the sample voices these were trained with were captured in different audio environments. There's noticeable reverb on the male voice that's not there on the other. So that's a useful next step: for multi-voice TTS models, make them sound like they're in the same room.
- simiones 1y agoI read the comments praising these voices as very life like, and went to the page primed to hear very convincing voices. That is not at all what I heard though. The voices are decent, but the intonation is off on almost every phrase, and there is a very clear robotic-sounding modulation. It's generally very impressive compared to many text-to-speech solutions from a few years ago, but for today, I find it very uninspiring. The AI generated voice you hear all over YouTube shorts is at least as good as most of the samples on this page. The only part that seemed impressive to me was the English + (Mandarin?) Chinese sample, that one seemed to switch very seamlessly between the two. But this may well be simply because (1) I'm not familiar with any Chinese language, so I couldn't really judge the pronunciation of that, and (2) the different character systems make it extremely clear that the model needs to switch between different languages. Peut-être que cela n'aurait pas été si simple if it had been switching between two languages using the same writing system - I'm particularly curious how it would have read "simple" in the phrase above (I think it should be read with the French pronunication, for example). And, of course, the singing part is painfully bad, I am very curious why they even included it.
- rcarmo 1y agoOne of the things this model is actually quite good at is voice cloning. Drop a recorded sample of your voice into the voices folder, and it just works.
- watsonmusic 1y agobonus usage
- MengerSponge 1y ago> (1) I'm not familiar with any Chinese language, so I couldn't really judge the pronunciation of that https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect
- IshKebab 1y agoI agree. For some reason the female voices are waaay more convincing than the male ones too, which sound barely better than speech synthesis from a decade ago.
- viggity 1y agoI feel like this is a step in the right direction, but a lot of emotive text-to-speech models are only changing the duration and loudness of each word, the timing/pauses are better too. I would love to have a model that can make sense of things like stressing particular syllables or phonemes to make a point.
- watsonmusic 1y agothis model is superb
- aargh_aargh 1y agoIs there a current, updated list (ideally, a ranking) of the best open weights TTS models? I'm actually more interested in STT (ASR) but the choices there are rather limited.
- xnx 1y agoClick leaderboard in the hamburger menu: https://huggingface.co/spaces/TTS-AGI/TTS-Arena-V2 https://huggingface.co/spaces/TTS-AGI/TTS-Arena-V2
- odie5533 1y agoBest TTS: VibeVoice, Chatterbox, Dia, Higgs, F5 TTS, Kokoro, Cosy Voice, XTTS-2.
- kroaton 1y agoUnmute.sh (same team as Kokoro) gets slept on, but it's really good.
- Uehreka 1y agoYes: https://huggingface.co/models?pipeline_tag=text-to-speech https://huggingface.co/models?pipeline_tag=text-to-speech Generally if a model is trending on that page, there’s enough juice for it to be worth a try. There’s a lot of subjective-opinion-having in this space, so beyond “is it trending on HF” the best eval is your own ears. But if something is not trending on HF it is unlikely to be much good.
- amelius 1y agoI tried some TTS models a while ago, but I noticed that none of them allowed to put markup statements in the text. For example, it would be nice to do something like: Hey look! [enthusiastic] Should we tell the others? Maybe not ... [giggles] etc. In fact, I think this kind of thing is absolutely necessary if you want to use this to replace a voice actor.
- deleted 1y ago[deleted]
- data-ottawa 1y agoEleven labs has some models with support for that. https://elevenlabs.io/blog/v3-audiotags https://elevenlabs.io/blog/v3-audiotags
- strangescript 1y agoThe male voices seem much worse than the female voices, borderline robotic. Every sample of their website starts with a female voice. They clearly are aware of the issue.
- jsomedon 1y agoI felt the same, male voice feels kinda artificial.
- baxuz 1y agoLooking forward to the day when tts and speech recognition will work on Croatian, or other less prevalent languages. It seems that it's only variants of English, Spanish and Chinese which are somewhat working.
- lukax 1y agoHave you tried Soniox for speech recognition? It supports Croatian. Or are you just looking for self-hosted open-source models? Soniox is very cheap ($0.1/h for async, $0.12/h for real-time) and you get $200 free credits on signup. https://soniox.com/ https://soniox.com/ Disclaimer: I used to work for Soniox
- baxuz 1y agoI meant in general purpose tools from Google and Apple. Most of this assistant and "AI" stuff is practically useless for me because I refuse to talk to my devices in English. In Android Auto / CarPlay I can't even get voice guidance that works properly, much less reading notifications, or composing a reply using STT
- faxmeyourcode 1y agoI tried the colab notebook that they link to and couldn't replicate the quality for whatever reason. I just swapped out the text and let it run on the introduction paragraph of Metamorphosis by Franz Kafka and it seemingly could not handle the intricacies.
- ementally 1y agothey vibecoded their demo website? the text is invisible on Firefox.
- double_one 1y agoSame problem here. A quick refresh solved it for me — maybe try that?
- recursive 1y agoWorks for me
- swiftcoder 1y agoAh, yes, the Furious 7 soundtrack. Definitely something everyone recalls
- closewith 1y agoThe most popular song of the year from one of the most popular movie franchises that had been in the global news due to the death of its star. Probably the most memorable song from a soundtrack of the century so far.
- agos 1y agoI'm Just Ken (Barbie), Skyfall, Let it Go (Frozen), Remember Me (Coco), Happy (from Despicable Me 2), a Star is Born (Shallow), are all arguably wayyyyy more memorable and these are just off the top of my head. We've had quite a few memorable songs in soundtracks this millennium. edit: I had forgotten about Jai Ho (Slumdog Millionaire) and Lose Yourself (8 mile)
- TheAceOfHearts 1y agoUnfortunately it's not usable if you're GPU-poor. Couldn't figure out how to run this with an old 1080. I tried VibeVoice-1.5B on my old CPU with torch.float32 and it took 832 seconds to generate a 66 second audio clip. Switching from torch.bfloat16 also introduced some weird sound artifacts in the audio output. If you're GPU-poor the best TTS model I've tried so far is Kokoro. Someone else mentioned in this thread that you cannot add annotations to the text to control the output. I think for these models to really level up there will have to be an intermediate step that takes your regular text as input and it generates an annotated output, which can be passed to the TTS model. That would give users way more control over the final output, since they would be able to inspect and tweak any details instead of expecting the model to get everything correctly in a single pass.
- tempodox 1y agoThis is ludicrous. macOS has had text-to-speech for ages with acceptable quality, and they never needed energy- and compute-expensive models for it. And it reacts instantly, not after ridiculous delays. I cannot believe this hype about “AI”, it’s just too absurd.
- NitpickLawyer 1y ago> with acceptable quality Compared to IBMs Steven Hawking's chair, maybe. But apple tts is not acceptable quality in any modern understanding of SotA, IMO.
- selkin 1y agoDifferent use cases: If you need a not-visual output of text, SoyA is a waste of electrons. If you want to try and mimic a human speaker, then it ain’t. Question is why would you need to have the computer sound more human, except for “because I can”.
- NitpickLawyer 1y agoI tried listening to audiobooks generated with tts. It takes me out of it most of the time, and I lose focus. That podcast thing from google was the first time I felt like I could listen to an entire thing without feeling the uncanny valley thing. And I knew it was genAI. So I'm looking for that, but for my content. Grab a bunch of articles (long form, deeply researched) and "podcast" them but with natural voices, sans hype. Or books. Have them ready when I'm out and about.
- anarticle 1y agoThe first example sounds like a cry for help. Some of them have tone wobbles which iirc was more common in early TTS models. Looks like the huge context window is really helping out here.
- ml_basics 1y agowhat's the relationship between this work and the recently announced voice models from Microsoft AI? https://microsoft.ai/news/two-new-in-house-models/ https://microsoft.ai/news/two-new-in-house-models/
- ehutch79 1y agoThe examples are kind of off-putting. We're definitely in uncanny valley territory here.
- sciencesama 1y agoNeed this for mac
- double_one 1y agoI tried it on my MacBook Pro — works great!
- agos 1y agoseemingly supports only English, Indian and Chinese
- qwertytyyuu 1y agoWoah they even immitate the western chinese accent well
- Meneth 1y agoOpen-source, eh? Where's the training data, then?
- Joel_Mckay 1y agoMost scraped data is often full of copyright, usage agreement, and privacy law violations. Making it "open" would be unwise for a commercial entity. =3
- zoobab 1y agoOpen source is being abused to not provide the actual source. Stop this.
- Joel_Mckay 1y agoA lot of code has multiple FOSS licenses that are not contaminating like GPL. GPL violations do occur on code, but have nothing to do with the training Data. For example, many academic data sets are not public domain, and can't be used in a commercial context. A GPL claim on that data is often an argument of which thief showed up first. Rule #24: A lawyers Strategic Truth is to never lie, but also avoid voluntarily disclosing information that may help opponents. Thus, a business will never disclose they paid a fool to break laws for them... =3
- deleted 1y ago[deleted]
- nullc 1y agoPerhaps, but it is not Open Source in the traditional sense if they do not provide the preferred form for modifications.
- Joel_Mckay 1y agoThere are also some weird OSS license rules that only trip the disclosure obligation when distributing the build to end users. Indeed, these adversarial behaviors do not follow the spirit of FOSS community standards. If a project started as FOSS, than FOSS it should remain. =3
- watsonmusic 1y agoone of the best models built by Microsoft
- nextworddev 1y agoStill haven’t found anything better than kokoro tts. Anyone know something better?
- deleted 1y ago[deleted]
- weeb 1y agodoes anyone know of recent TTS options that let you specify IPA rather than written words? Azure lets you do this, but something local (and better than existing OS voices) would be great for my project.
- andybug 1y agoI'm using Kokoro via https://github.com/remsky/Kokoro-FastAPI https://github.com/remsky/Kokoro-FastAPI. It has a `generate_audio_from_phonemes()` endpoint that I'm sure maps to the Kokoro library if you want to use it directly. My usage is for Chinese, but the phonemes it generated looked very much like IPA.
- lagniappe 1y agoBots should never sing.
- cindyllm 1y ago[dead]
- bityard 1y agoI thought the name sounded familiar, I'm guessing its no relation to this project which has been around for 7 months? https://github.com/mpaepper/vibevoice https://github.com/mpaepper/vibevoice
- Insanity 1y agoWhat an odd name to me, becaus "Vibe" is, in my mind, equal to somewhat poor quality. Like "Vibe Coding". But that's probably just some bias from my side.
- mxfh 1y agoVibe coding just became a term this spring. I doubt that that the substantial part, like giving it a project code name and getting company approval of this research project started after that. It's not libe vibe has a negative connotation in general yet.
- Insanity 1y ago'Vibe' as a word / product was definitely less common though. I kinda doubt that 'VibeVoice' is _not_ a consequence of 'VibeCode'. But I do agree with you in that generally there's probably no negative connotation (yet).
- andrew_lettuce 1y agoVibe always meant "specific feel" and makes sense related to AI coding "by touch" vs. understanding what's actually happening. It's just the results have now made the word pejorative.
- lxe 1y agoThere are 2 "best" TTS models out right now: HiggsAudio and VibeVoice. I found that Higgs is both faster and much higher fidelity than Vibe. Can't speak to expressiveness, but don't sleep on it.
- tehlike 1y agoThe comments in the html code is chinese, which is very interesting.
- cush 1y agoTo me this is like early generative AI art, where the images came out very "smooth" and visually buttery, but instead there's no timbre to the voices. Intonation issues aside, these models could use a touch of vocal fry and some body to be more believable
- cush 1y agoI tried using the demo but it just errors out
- crvdgc 1y agoVery impressive that it can reproduce the Mandarin accent when speaking English and English accent when speaking Mandarin.
- bazlan 1y agoSad to not see vui on the comparisons! A 100M podcast model https://huggingface.co/spaces/fluxions/vui-space https://huggingface.co/spaces/fluxions/vui-space
- stuffoverflow 1y agoVibeVoice-Large is the first local TTS that can produce convincing Finnish speech with little to no accent. I tinkered with it yesterday and was pleasantly surprised at how good the voice cloning is and how it "clones" the emotion in the speech as well.
- data-ottawa 1y agoLooks like the repo went private https://github.com/microsoft/VibeVoice https://github.com/microsoft/VibeVoice I was trying to get this working on strix halo.
- ndkap 1y agoHere is AI being as close as possible to the most animated person I know and here I am sounding robotic in every conversation I have, despite my best efforts to sound otherwise. Sometimes, I just wish I could have an AI speak for me
- mpaepper 1y agoUnfortunate naming given I named my repo which does open source locally running speech to text vibevoice 7 months ago: https://github.com/mpaepper/vibevoice https://github.com/mpaepper/vibevoice
- davorak 1y agoAny insight on my the code and the large model were removed? Some copies are floating around and are MIT licensed. In cases like this I do not know why the projects are yanked. If the project was mistakenly released under MIT, copied elsewhere, is any damage control possible by yanking the copies you have control over? Mostly seems like bad PR, if minor.
- fivestones 1y agoWondering this too.
- androiddrew 1y agoOk anyone have a link to the code and weights?
- lyu07282 1y agoDid they delete the repo? It's 404 for me now: https://github.com/microsoft/VibeVoice https://github.com/microsoft/VibeVoice
- RealtyDAO 1y agothey must have removed it.. been down for hrs.
- lyu07282 1y agoRepo is back but code is gone, with this statement: > 2025-09-05: VibeVoice is an open-source research framework intended to advance collaboration in the speech synthesis community. After release, we discovered instances where the tool was used in ways inconsistent with the stated intent. Since responsible use of AI is one of Microsoft’s guiding principles, we have disabled the repo until we are confident that out-of-scope use is no longer possible. What was that about?
- enigma101 1y agoonly microsoft could come up with such a name rofl
- defrost 1y agoLippy got vetoed.