10 ms·

Show HN: AI dub tool I made to watch foreign language videos with my 7-year-old

348 points by leobg 3y ago

Hey HN! I love watching YouTube with my 7-year-old daughter. Unfortunately, the best stuff is often in English (we're German). So I made an AI tool that translates videos directly, using the original voices. All other sounds, as well as background music, are preserved, too. Turns out that it works for many other language pairs, too. So far, it can create dubs in English, Mandarin Chinese, Spanish, Arabic, French, Russian, German, Italian, Korean, Polish and Dutch. The main challenge in building this was to get the balance right between translating the original meaning and getting the timing right. Especially for language pairs like English -> German, where the target ist often longer than the source ("bat" -> "Fle-der-maus", "speed" -> "Ge-schwin-dig-keit"). Let me know what you think! :)

2099miles 3y ago
Talked about this idea last month since astrobiology still isn’t all dubbed to English. Thank you for actually making the tool, it’s awesome, huge Kudos.
iamjackg 3y ago
Is this using XTTS? I recognize a funny/weird glitch with Italian voices saying "punto" (full stop) at the end of every sentence.
poulsbohemian 3y ago
This is interesting because I'm the opposite of you - a German speaking American, who watches a lot of German language content on YouTube. Are you specifically looking for children's content? I ask because almost anything I would watch in English, I can find an equivalent content producer in German.
poulsbohemian 3y ago
You deserve a lot of credit for doing this... often when I am watching German content that has English subtitles, I wonder if the subtitles are already being produced by AI... I sometimes find the subtitles more confusing (even though they are in my native language) than the German, as though they almost had to have been automatically produced rather than by an actual translator using contextual clues, etc.
pell 3y ago
This is very impressive. I think the way you are timing the audio is clever. What kind of model are you using?
pcchristie 3y ago
This is extremely impressive. Congratulations!
- leobg 3y ago
  Thank you!
deleted 3y ago
[deleted]
auct 3y ago
Can you add ukrainian?
- leobg 3y ago
  Hey! As a target/output language? As a source language, it’s already supported.
  - oneshtein 3y ago
    Yes, as target language. And remove Яussian language, please, unless you are Яepublican. You can add it back after the war.
oldge 3y ago
Very cool, any chance to see a on device release so we can run this locally? Topaz ai has a pretty good model for this if you are looking to monetize.
cushpush 3y ago
This is really amazing I'm very impressed and happy you released this. Can you share more details about your development rhythm of this helpful piece of software?
bufferoverflow 3y ago
I speak Russian, and I gotta say, the Lex sample is incredible. It sounds like real dubbing. Maybe not pro-level dubbing, but it's very very good. They voices are also very close to Lex's and Elon's. Congrats! Very well done.
- leobg 3y ago
  Thank you! Yeah, I also had a big grin on my face, hearing Lex and Elon suddenly talk in another language. :) Consistency isn't perfect yet, as we're building the voice from scratch basically for each utterance. One the one hand, you want that, because the utterance might be more upbeat, or lower pitch, than the speaker's "average" voice. On the other hand, it sometimes introduces variance that makes the listener's brain go, "Uh.... is that another person speaking now?". If I had to dub 200 videos of a single YouTube channel, I would be able to fine-tune the voices of the main characters, and reserve the ad-hoc cloning for guest characters.
artninja1988 3y ago
Why do you keep the original audio track on the dubbed version? I think it sounds pretty distracting, although you do want to keep sounds other than the original voice I guess
- askhan 3y ago
  I think these little bits of the original sound really works well, helping us hear the original as well and keeping it less uncanny. What an amazing project!
  - leobg 3y ago
    Thank you. Exactly. I had the OG voice removed at first. But I added it back in for exactly this reason. It also serves as a tool for AI accountability: It lets you "see" that the cloned voice is indeed saying the same thing as the original voice. That being said, it would be trivial to turn the OG voice off for anyone who wants to.
- maxglute 3y ago
  I dig it, it's like amateur underground over dubs of bootleg hollywood movies. Don't want to conflate amateur AI voice with actual personality on screen. The seperation is part of the experience.
- gardenhedge 3y ago
  News channels do that for interviews
simple10 3y ago
This looks amazing! Thanks for sharing. I signed up for the private beta.
- leobg 3y ago
  Thank you so much!
lxe 3y ago
What does this use behind the scenes? This type of stuff can get pretty expensive if you're relying on elevenlabs or heygen.
- brody_hamer 3y ago
  Yea my guess would be elevenlabs, which just recently announced this exact featureset.
- leobg 3y ago
  Let's just say this is NOT a wrapper around Elevenlabs or Heygen. I've looked at commercial voice cloning before. But, as you said, the prices seemed ridiculous. Before this, I made audiobooks for my daughter. Old, out-of-print books, turned into speech. If I remember correctly, with Elevenlabs a single book would have cost me > $100. At that price level, I can read the damn thing myself. What good is computer generated voice if it isn't at least 10x cheaper than doing it yourself? I'm just one guy. With me, it's just my time, one or two commercial licenses, and other than that just the raw price of running those GPUs.
scrollaway 3y ago
I know this is HN so I don't want to distract from the technical achievement and how genuinely useful this can be. I also don't want to tell you how to raise your kid. You do you, it's not my family. But I want to share how important it is to watch foreign spoken language movies and TV, especially as a kid, to be able to speak multiple languages later in life. You'll notice that in every country where TV and movies are regularly dubbed in the local language, the English levels go to shit. Dubbing is partially responsible for this because kids are not exposed to a different language on a regular basis. I remember wanting to watch a dubbed movie with my mom as a kid, and she told me "We will watch the original instead, dubbed movies don't have a soul". It stuck with me. She was absolutely right. Today I am working on my sixth spoken language. Causation not guaranteed, merely implied.
- true_religion 3y ago
  Counter point. My parents didn’t let me watch dubbed shows, and didn’t speak our native language because rhetorical wanted me to speak unaffected English. I can’t speak any languages, but in school my English was insanely good. To the point of perfect scores in the college scholastic exams, and when I was in uni for engineering, I took on an English major for fun with essentially no impact on my work load. You can generalize but you can also specialize.
- Baeocystin 3y ago
  I can't say I agree with dubbed movies having no soul. Greater accessibility to a wider audience is not something to deride, or hold in contempt. That being said, I do agree that listening to other languages is a great thing. My father was a linguist, and when we would watch subtitled media, we'd play a game where we'd try and hear the cognates, pick out the most common words, figure out the basics of the grammar as we went along. It was a lot of fun!
  - leobg 3y ago
    One of my favorite movies was "Scent Of A Woman". But when I watched it in the German-dubbed version, I was appalled. It made the whole movie suddenly seem like a comedy. To me, the translation had killed its "soul", for lack of a better term. I still want my kids to learn English. And ideally also one or two other languages, like Chinese. As Nietzsche said: "So you have mounted on horseback? And now ride briskly up to your goal? Well, my friend - but your lame foot is also riding with you!"
sorenjan 3y ago
I know Germany dub most video, but wouldn't a seven year old be able to read subtitles? It's a great way for her to learn English, it's how most Swedes learn it before starting school. I think there's a pretty strong correlation between countries' average English proficiency and how common dubbing is. https://haonowshaokao.com/2013/05/18/does-dubbing-tv-harm-language-learning/ https://haonowshaokao.com/2013/05/18/does-dubbing-tv-harm-la... Edit: I forgot to mention that the samples on the website is impressive and well made. How do you do the speaker diarization and voice cloning?
- poulsbohemian 3y ago
  >I know Germany dub most video, but wouldn't a seven year old be able to read subtitles? I gotta say... while sometimes it is a necessarily evil, I would so rather not have to read subtitles. I often want to listen to a show so that I can also continue working on catching up on email, etc, IE: I can't read two things at once, but I can listen to one thing and continue working on something else.
- supafastcoder 3y ago
  I think it’s a cultural difference. I’m also from a non-dubbing country (Netherlands) and I can’t stand dubbed content either. On the other hand people tell me they can’t stand subtitles because it “reveals” what they’re going to say before they say it.
  - lukan 3y ago
    "people tell me they can’t stand subtitles because it “reveals” what they’re going to say before they say it." I love watching movies in the original language, but this is something I hate as well, but something that can be avoided. Some movies get it right, though. The timing, just the words that are spoken and even different colors for different persons speaking (very rare, cannot even remember where I have seen it). That should be standard, but with most movies you can be lucky if the subs even match the plot and do not reveal too much.
    - 3y ago
nextworddev 3y ago
[flagged]
sss111 3y ago
can you add hindi as an output language, been meaning to build something like this for my parents. You saved me some work haha!
jianshen 3y ago
Wow this is amazing. If there was a locally running version available, I would gladly pay money for it.
- leobg 3y ago
  Thanks! Well, to make local happen I'd have to learn more about local app development. I'd also be worried about having to support a bunch of different platforms, and being beholden to ever changing rules made by App Stores and OS makers. I actually work on a 2015 Mac with a 2019 operating system. There are many great looking AI apps that I'd love to run but can't. Besides, it seems to me that making this centralized makes economic sense. I can just keep the GPU busy with lots of videos from many customers. I'm sure that's what most people think who build something: "The world would be so much better if everyone just came here and used this." :)
solardev 3y ago
This is really impressive! Can't wait to see this more fleshed out. I'd gladly pay for something like this (by the video, ideally). Some page feedback though: It seems to me that the video just keeps playing, with no way to restart it or scrub through the timeline. Each time I click a language, it changes the spoken audio but just keeps playing where it left off. That makes it hard to compare the same passage across different languages. Separately, I think there are also some errors in translation. For Sample 3 (about the vines), the original in Mandarin Chinese says something like "if this tree gets grabbed, the weed will climb up and wrap around it, and the tree won't be able to photosynthesize and will die". But the English mistranslation says "If it gets scared by people, it gets pulled off and messed with. It can't function. The evil effects? It just dies." There are also timing issues where the translations don't match up with the original subtitles or dialogue, and certain parts of the original audio just seem to be altogether ignored and not translated. Maybe displaying the translated subtitles, allow with a way for users to report errors, would help...?
- leobg 3y ago
  Thank you very much! Yes. You cannot control the video playback on the demo page. I made it so because I wanted a way to showcase how you can switch between languages. You can go from Elon speaking English to German, Russian and Chinese, each with just one click. Activating the player controls would have made the UI more complex and distracting. And it would have also made it harder for me to sync the timing between languages. Of course, the real output would be a proper player, with all of the controls. Or, for creators, raw files (video and/or audio, plus SRT subtitles). I also noticed problems in the translation of the Chinese video. I put it up there anyway, because I figured most people coming to my site would be English speakers, and being able to understand a Chinese video might be another interesting aspect, in addition to the idea of being able to turn your own English content into languages you don't speak. If this had been a pitch deck, I would have cherry picked the samples. But I wanted to share where the project is right now and see if anyone was interested. Premature optimization is the root evil of all programming. I think Knuth said that. And it's a trap I regularly fall into. So I tried to be disciplined this time. But if any Chinese YouTuber would ask me to dub their work today, I'd make darn sure that the translations were close to perfect. Meaning I'd allow the system to make changes to the way things are phrased if that's necessary for the purpose of timing or cultural context. But I wouldn't allow it to skip a thought from the original video, or say something something different. I've translated books by hand in the past. So this is something I care about. If the demo isn't perfect in this regard, it's because I didn't know if anyone was going to even look at my project. When I first posted this yesterday, my submissions didn't go beyond one comment for several hours. I already thought I had built another solution looking for a problem. :) If you're seeing dropped phrases, that's most likely because my arranging function failed. Basically, the translation ran longer than the original. The algorithm tried to speed it up and fit it in. But it failed and dropped it. Better handling of these overruns are on my to-do list. Neither drops nor speedups should be tolerated. In terms of self-correction, I plan to feed the translated audio back into the transcription engine. Then, an LLM can compare the translation with the original transcript. If anything is missing, the pipeline will be force to run again with slightly different parameters. There shouldn't be a human neccessary in the loop. Translation is what Transformers are best at.
paxcoder 3y ago
[dead]
gagabity 3y ago
This is great, I tried to do a similar thing once but my language is one of those that AI doesnt do well. I think you can look into muting the original voice in the video, I remember I saw there is some AI/tech that can separate audio into voice and nonvoice. Yandex browser does this in the browser, you open a YT video and it offers to translate, a few seconds later the voices are all in Russian, it's probably the most interesting production use of AI I have seen and for free. It's to Russian only unfortunately.
maxglute 3y ago
Very passable. Waiting for something local like this for foreign language PLEX and podcasts. As someone who views/listens to things at 2x/3x speed 10-15 bucks an hour is cost prohibitive.
- leobg 3y ago
  Perhaps you could ask some of your favorite podcast hosts to make a deal with me. Running a training on their voice once and then just re-using that will be much cheaper. Also, customers who buy in bulk will help me focus on this full time. There is huge potential for making this faster and cheaper. (Even using OpenAI is silly. Technically, I neither need GPT-4's knowledge nor instruction tuning. Both is unnecessarily adding cost and latency. But it helped me get the demo out.) Basically, the deal for Podcasters / YouTubers would be: - Get all their episodes converted into 15+ languages - Increase their reach today, while the novelty is high and the market is still uncrowded - They get to tell their sponsors that they now have reach across the language boundary
  - maxglute 3y ago
    I don't know state of podcast ecosystem, but I think you should reach out to listennotes.com whose also a 1 man job that seems to elevate discovery and looks like has reasonable reach for producers. Or go hit up some popular western podcasts, you've definitely got something here and the execution is good enough.
theogravity 3y ago
Wonder how accuracy of the translation is measured (if at all).
- leobg 3y ago
  One idea I have is to use back-translation. After generating the new language audio, feed it back into the transcription, and then have an LLM compare it to the transcript from the original. Penalty for any thought/detail that is missing. If too bad, start from scratch.
daremon 3y ago
This is really amazing! Well done. I already joined the beta but I want to point out another use case here as well: In many countries (ie Greece where I'm from) movies and TV shows never get dubbed. We rely on subtitles. This means that if you can't see well (disability or age-related eye problems) and if your English is not excellent, then you are doomed to only watch locally produced movies & shows. This can be a real life-changer.
- leobg 3y ago
  Thank you! With movies, I think I could get into legally challenging territory. I guess all AI apps are, in a way. But with movies, there's an entire industry behind enforcing copyright. So I must tread carefully on that front. I made the jump from the courtroom into VS Code years ago. I really don't want to go backwards.
  - daremon 3y ago
    I honestly don't see how movies are different with any content ie YouTube videos. I am pretty sure MrBeast etc have the same lawyers as any big studio. Could this run locally? I would certainly pay for that and you're off the hook on how anyone uses it.
waldrews 3y ago
Impressively done! It sounds like you're doing 1) doing voice recognition with voice time clues, which Whisper and the like provide, breaking it up into sentence (or similar) units; you don't need to time match individual words, but you need to time match at coarser grain. 2) using a translation engine that allows for multiple alternative translations 3) cloning the original voice, regardless of language 4) choosing the translation that has the best time match (possibly by syllable counting, or by actually rendering and timing the translations). If there isn't a close translation, maybe you're asking ChatGPT to forcibly rephrase? 5) Maybe some modest pitch-corrected rate control to pick out path that gets you closest to the timing? Did I get any of that right?
- euazOn 3y ago
  I also noticed that the third sample with Chinese sounds slightly sped up in the first English segment, so there may be also an element of postprocessing the dub (speeding it up/slowing it down).
  - leobg 3y ago
    Yes. Though I don't like this solution. It breaks the flow. And it also doesn't really fully solve the problem. Overruns still accumulate if they happen too frequently. One second here, one second there... the further you get into the video, the worse it gets. I think it would be better to either slow down the underlying video or solve the overrun issue on the translation level. A good professional dubber will find translations that will even out in terms of timing. That's something an AI should be able to do better instead of worse.
  - odiroot 3y ago
    The last sample from BBC is really hilarious when translated to Polish. Something definitely went wrong and the voice speaks like a drunkard.
- waldrews 3y ago
  Ooh and you're probably doing a split into voice and non-voice tracks of the original, and keeping non-voice at original volume, but lowering the voice track.
lIIllIIllIIllII 3y ago
This might be a game-changer for preserving declining languages.
- leobg 3y ago
  I'm not sure. Making this work is limited by the availability by reliable transcription models. Which, in turn, are limited the availability of large training corpora. Those don't exist for rare languages. Also, if people choose to listen to a Nepali speaker through an AI translator, that does give speakers of this language "a voice" - but it doesn't really preserve that language. You might argue, on the contrary, that it may remove any remaining incentives to learn that language.
azamba 3y ago
What would be required to add a new output language? E.g Portuguese? I know it’s supported as input.
deleted 3y ago
[deleted]
crtasm 3y ago
What potential issues with copyright are there from offering (paid) access to this tool to run on sources including Youtube, and with the output containing the source audio?
- Freak_NL 3y ago
  You're creating a derived work, so you would be violating someone's IP unless the licence is permissive, and making money of it. That usually attracts the attention of whoever owns the IP.
  - dns_snek 3y ago
    Who's "you"? I'm obviously not a lawyer, but instinct tells me that end users are the ones creating a derived work by uploading a video they may or may not have the right to distribute. Linked website is just a tool, perhaps a cloud-based one, but still just a tool.
    - crtasm 3y ago
      Not just files a user uploads: "You can select videos from YouTube"
      - dns_snek 3y ago
        Thanks, I missed that. I can see how that would complicate things.
- leobg 3y ago
  You could have asked the same question when Google started building their index. Or OpenAI trained their models. I'm in Germany. I'm a licensed lawyer. I see the dangers. The safest path will be to simply offer the production of multi language translations to content owners themselves. Which is also going to be more efficient - translating the thing at the source, rather than having consumers each create their own translation. But the original intent for this has been to have my computer translate a video I want to watch with my kids in my private home. Technically, it's not "my computer" in the sense of being just the device that's physically in my home. There's stuff that happens in the cloud. Technically, copies are being made. So one could argue the point. For most people today, getting your content seen and consumed is the highest you can achieve. To sue someone from another country who cares enough to pay someone else to translate it for him would seem bonkers. But I'm sure there are lawyers who are desperate for work. Who cannot code, and can't be bothered to learn, but still want to do something "in AI". I'll at least give them a hard time. And dare they use ChatGPT hallucinated references on me! :)
hbarka 3y ago
Wow, this is impressive. Is there anything like it for live translation?
- leobg 3y ago
  Not on my to do list currently. EzDubs say they do live [0]. Also, a friend of mine mentioned some Samsung / Android app that does this? [0] https://ezdubs.ai https://ezdubs.ai
patrickhogan1 3y ago
Awesome! Large % of foreign streams have no proper dub.
godzillabrennus 3y ago
Would love this for Plex.
- leobg 3y ago
  Would love to. Can you broker a deal? :)
jeroenhd 3y ago
I've only ever experienced Dutch dubs in kids' TV but I feel like these examples show that your Dutch model may need some work. I can't judge other languages well, but I found the Taiwanese documentary dub especially hard to follow. I wouldn't have expected Dutch to be in there for how little the language is spoken and how often Dutch speakers will understand English, though! /offtopic It seems to do a pretty interesting thing where the first male voice has a bit of a Flemish/southern accent while the second male voice has an accent much closer to "Netherlands TV" Dutch. Reminded me a bit of the Lion King dub where the dub studio used Flemish voice actors to do the jungle animals (and Dutch voice actors for the savannah animals) to underline the "different world" Simba arrived in.
- leobg 3y ago
  Yes, that issue is also present in the German translation. I'm planning to monitor the output quality. Basically, feeding the translated audio back into the transcriber. Then compare it to the original transcript. Like a higher level loss function. I'll need this already because I don't speak all of these languages myself. But I can also use it to make the pipeline self-regulate and generate a new, better version if the last one scored too poorly.
  - jeroenhd 3y ago
    Interesting, I can see how that approach would catch the weird voice lines. Just the different ways the languages get picked up and processed by the AI system could be interesting. If you find anything cool, I'd love to read a blog post about it!
cheriot 3y ago
Great tool! Curious if you've made more progress on diarization than what's described in this article? https://aipressroom.com/streamline-diarization-using-ai-as-an-assistive-technology-zoo-digitals-story/ https://aipressroom.com/streamline-diarization-using-ai-as-a...
- leobg 3y ago
  They use pyannote/speaker-diarization. I tried that, but it wasn't accurate enough for my purposes. Made a confusion matrix with voice samples from The Simpsons characters. It looked... well, confused. Am using a mix now of speaker embeddings and other signals (end of sentence, pause before/after, etc.). As you can see in the demos, it already works well for interview situations. It's when there are 3+ speakers and they talk over each other that the system gets confused.
changoplatanero 3y ago
Why doesn't youtube have something like this built in.
- madduci 3y ago
  I guess it's computationally expensive? OP states in the website that their solution takes 1 hour of processing for 30 minutes long videos . Now imagine offering this for all the YouTube videos available: - either it's done on their servers (hard to believe due to high costs) - either it's done on client side (which is also difficult, due to lack of processing power)
  - leobg 3y ago
    Well, I'm also shamefully unoptimized at the moment. YouTube added auto-captions years ago. Long before there was Whisper, let alone things like Whisper.cpp. I imagine what I'm doing now is computationally no more expensive than what they did back then.
- Freak_NL 3y ago
  And automatically forced on every user depending on whatever their Google account is set to, just like the video titles which now get auto-translated without any way to turn this off. We're sliding back into a monolingual world. No thanks.
  - leobg 3y ago
    As a native German, I also resent it when Google/Amazon/whoever tries to force a translation on me when I prefer the original language. So I wouldn't want to live in a world where everything would be dubbed into German for me. Not even if they used my tool :) Regarding YouTube: AFAIK, YouTube allows you to add multi-lingual voice tracks to your videos. Then, if the viewer has a preferred language set, the video will play in that language. Else in the language inferred by his browser/OS. But the user can also switch back to the original language, or any other language, right in the player.
    - 3y ago
Gunnerhead 3y ago
Amazing! I love this for dubbing, but was wondering if anyone knows of an AI powered subtitle generator for YouTube videos? I know YouTube has closed caption, but it’s terrible.
- fbnt 3y ago
  https://fluen.ai https://fluen.ai gives very accurate subtitles (and subtitle translations).
- leobg 3y ago
  Speakz actually generates subtitles as a byproduct. The idea is that you put in a video, you select the target languages. And then you get out, for each target language, an audio track and an SRT subtitles file. Someone else here asked about generating only subtitles, with no audio, as a cheaper option. So I'll probably add that as an option.
  - Gunnerhead 3y ago
    I would love that to help learn another language!
leke 3y ago
I'm looking for a tool that will take a foreign language and automatically generate subtitles in my language. Anyone know of such a tool?
ocolegro 3y ago
So when is the crunchyroll integration rolling out?
exitb 3y ago
I suppose you're paying a lot for the voice cloning, so do consider that in many countries voice-overs are done with a single generic voice. Would you consider a lower price point service doing just that? I'm doing something similar and using GPT-4 for translation. What's unique about it, is that you can specifically prompt it to avoid long translations by rephrasing things, so you can buy yourself some time for the "Fledermaus".
- leobg 3y ago
  Using a single speaker makes sense when you're paying for human voice talents. But if you're using a computer to generate the voice, why not generate a voice that sounds like the original speaker? Much more fun to hear Elon speak Chinese :)
  - exitb 3y ago
    To be blunt, because of the price. I’m running the whole pipeline of my toy project for much less than $5 per hour. If the voice cloning is the long pole in the tent, I’d just consider dropping it. Moreover, it’s a cultural custom. I’m from Poland and here the voice-over narrator is supposed to be generic and bland, so your brain learns to tune him out and take the emotional cues from the original voice.
    - leobg 3y ago
      Yes, in Germany we have generic voice-over narrators, too. In documentaries, etc.. They usually match the gender of the speaker, but that's it. Personally, I read most of my books with the iOS app Voice Dream Reader. That app still uses old TTS voices. They sounded great 3 years ago, but now sound robotic when compared to Elevenlabs or WaveNet. But, as you say, you learn to tune out the voice. I can read entire novels like this, and I still "hear" the different voices and personalities. It just all happens in my imagination. How much I'd need to charge to make the project worthwhile depends on many factors. And I didn't want to name a price now and then backpaddle a month from now and say it'll actually cost more. My pipeline right now is super unoptimized, to the point of being embarrassing. This can all be made to run much faster and cheaper. I agree with you that if the voice cloning part of my pipeline causes a significant chunk of the cost that the end user pays for the service, that I should then offer the option of using a "bland" voice instead for a lower price.
mannycalavera42 3y ago
it's a lovely parenting story. Let me tell you there is also a huge opportunity for the opposite use case. My elderly parents speak only one (non-Eglish) language. I would love to have a (cheaper) way to provide my parents with translated videos with the addition of (translated) subtitles. Subs are important because elderlies can have hearing issues great work, inclusivity is love
- felixarba 3y ago
  I second this! I was recently looking into a way to build something like this for my grandfather, but wasn't even sure where to start from the hardware side. I wanted to have hardware plug into TV receiver, generate subtitles for live TV program and then play it back on TV. Delay would likely be less than a minute but even a few minutes is not a problem really. Many people with a hearing problem would benefit from this and with AI getting so good at Speech-to-text, this can be done for quite a large population. If anyone has a recommendation on where to start with this, I'd appreciate it! Was thinking of using Whisper for subtitle generation, but not sure about hardware that can take in, and output HDMI and run this software
  - leobg 3y ago
    I keep thinking about something similar. Also hardware. Also for my grandparents. My grandma is 95. Her vision is bad. Even using the phone (I'm talking old school landline) is getting hit and miss, because she can't see the buttons. Years ago, I set her up with an Echo Show. That works well enough for her to say "Call Leo". But Alexa is dumb. Sometimes, she'll mishear something and start playing music. Or start a monthly subscription... :) So what I'd like: - box - screen - far-field mic array - AI backend You could do a number of things with it: - manage a grocery shopping list (AI will notice duplicates and other oddities and ask) - communicate with the outside world (initiate calls, send emails and faxes, including to local businesses) - optional human oversight and/or permission settings (preventing the AI, say, from ordering groceries for more than $50 a week without a family member approving the order) Something like your "subtitle mode" could also work: "Listen to what is currently being spoken in the room (including the TV), and display it on the screen". My grandma has her TV running all day. So maybe one could ditch the screen and make it a "set top box". Add an IR port to it, so it can control also the TV itself. Something like that might work.
Timwi 3y ago
Would have been nice to get a download link for a program I can just run locally.
- bbitmaster 3y ago
  Alas! Didn't get the memo? We're In the era where AI tools are all "software as a service," and you must pay for individual inferences from the model. How could they charge for inferences if they gave you the model to download?
  - Timwi 3y ago
    Amen, bro.
  - leobg 3y ago
    I don't have access to any special model that I'm holding back from you in order to rent-seek. Anyone can learn to build something like this. The parts are all available out there. There's Whisper. There's Mistral-7b. There's Tortoise, Coqui, SV2TTS. There's Python. The bigger question is: Would you want to? I've been building web apps for several years now. I've sunk thousands of dollars into those projects. And literally years of my time. If I calculated my hourly wage, I'd be below a teenager mowing lawn. In Rwanda. And by a factor of 10x, probably. The real ROI here is the learning. And that's not something I'm "taking" from anyone.
    - rubymamis 3y ago
      You are awesome, man! Keep it going and all the best of luck!
welder 3y ago
I do the opposite and seek out videos in other languages for my kids to watch. Now they're learning German, Spanish, Chinese, and Japanese.
android521 3y ago
any opensource available?
sanjananb 3y ago
[dead]
ipsum2 3y ago
Does this use seamless4t or similar projects?
pavelboyko 3y ago
Please consider adding Simplified English as an output language option, preferably with a level, e.g., A2, B1, etc. This way, I can adjust the language complexity to my kids' level and then gradually remove the crutches as they improve in English.
- leobg 3y ago
  Yes! I love this! So you'd be translating English to Simplified English? Or are you talking from another source language? I've already been playing with this concept w.r.t. books: I take a non-fiction book. I'll have an LLM translate it with a specific audience in mind (say, a 7 year old girl with a certain background), explaining concepts and words that are likely unknown to that audience. And then converting the whole thing into an audiobook. Optional parental controls built in ("exclude violence", etc.). Nowhere near showtime, though. Another thing I'd love to work on is filtering existing content. There are millions of videos on YouTube. Right now, finding quality stuff that's fun to watch with my kid depends a lot on dumb luck. But what if I could filter by topic (semantic whitelist/blacklist, i.e. not keyword dependent), personality traits (OCEAN, MBTI), values (e.g. "curiousity") and language (reading level, vocabulary, words per minute, etc.)? I'd love that.
  - vidarh 3y ago
    I'd love what they suggested as well, for other languages. I'm working on improving my French (and occasionally German), and I'm at a stage where I can follow along some French shows reasonably well if they're not speaking too fast (one of the first French phrases my French teacher in school taught us was "plus lentement, s'il vous plaît" - "slower/slowly please", for a reason), and if they're not speaking any particularly difficult accents, and not too much slang, but it's limiting and I'm often forced to keep English subtitles on as a consequence and it's sometimes too much of a crutch. It doesn't help that my hearing isn't what it was. Being able to "step down" the difficulty so that I can either turn off subtitles entirely or rely on French subtitles, or even much "difficult speech" and "simple subtitles" or vice versa seems like it'd be very useful in getting over that hump faster.
stuckkeys 3y ago
Cool project. What is the tech stack behind it. I can already guess few. 11labs, OpenAI…dreamtalk. There are so many similar services to what you are doing. What sets you apart? You should partner with local media or outside. Good luck! Check out https://www.flawlessai.com https://www.flawlessai.com they been around since 2018. Interesting stuff. When I first saw it in 2021 I was blown away.
YKreator 3y ago
Congratulations on this project! We spent 6 years developing the best solution for generating perfect subtitles automatically (https://www.Checksub.com https://www.Checksub.com). 2 years ago, with the arrival of new generations of AI, we decided to go a step further and add the possibility of automatically dubbing videos. But automatic dubbing requires manual adjustments for a comfortable result for the audience. For example, https://www.HeyGen.com https://www.HeyGen.com generates a video automatically, but offers very few editing options. That's why we focus on two things: 1 - to provide the best possible automatic quality 2 - to offer an advanced editor that lets you fine-tune your dubbing without having to go back to editing software. In any case, I'm delighted to see people working on this problem. I hope it will help develop this sector.
- luxpir 3y ago
  Thanks for Checksub, another happy user here. We take multilingual, AI-cloned audio from you guys (split from background noise, of course), after we've processed the subs professionally at our end, then we align everything in your tool and send off to a third service for lip sync. The result has blown away a few clients now. The CEO speaking 5 languages with perfect lip sync in their own tone of voice is quite convincing. Hopefully we can get it all in one tool soon.
- leobg 3y ago
  Great website! You're based in France? You should put a demo on your website. If there is one, I don't see it (or rather: hear). If you're interested in collaborating, me email is in my bio.
hombre_fatal 3y ago
What a great application of AI. The samples were amazing.
- leobg 3y ago
  Thank you!
k__ 3y ago
Would be awesome if it could also deep-fake the voice.

348 points by leobg 3y ago

2099miles 3y ago

Talked about this idea last month since astrobiology still isn’t all dubbed to English. Thank you for actually making the tool, it’s awesome, huge Kudos.

iamjackg 3y ago

Is this using XTTS? I recognize a funny/weird glitch with Italian voices saying "punto" (full stop) at the end of every sentence.

poulsbohemian 3y ago

This is interesting because I'm the opposite of you - a German speaking American, who watches a lot of German language content on YouTube. Are you specifically looking for children's content? I ask because almost anything I would watch in English, I can find an equivalent content producer in German.

poulsbohemian 3y ago

You deserve a lot of credit for doing this... often when I am watching German content that has English subtitles, I wonder if the subtitles are already being produced by AI... I sometimes find the subtitles more confusing (even though they are in my native language) than the German, as though they almost had to have been automatically produced rather than by an actual translator using contextual clues, etc.

pell 3y ago

This is very impressive. I think the way you are timing the audio is clever. What kind of model are you using?

pcchristie 3y ago

This is extremely impressive. Congratulations!

leobg 3y ago
Thank you!

deleted 3y ago

[deleted]

auct 3y ago

Can you add ukrainian?

leobg 3y ago
Hey! As a target/output language? As a source language, it’s already supported.
- oneshtein 3y ago
  Yes, as target language. And remove Яussian language, please, unless you are Яepublican. You can add it back after the war.

oldge 3y ago

Very cool, any chance to see a on device release so we can run this locally? Topaz ai has a pretty good model for this if you are looking to monetize.

cushpush 3y ago

This is really amazing I'm very impressed and happy you released this. Can you share more details about your development rhythm of this helpful piece of software?

bufferoverflow 3y ago

I speak Russian, and I gotta say, the Lex sample is incredible. It sounds like real dubbing. Maybe not pro-level dubbing, but it's very very good. They voices are also very close to Lex's and Elon's. Congrats! Very well done.

leobg 3y ago
Thank you! Yeah, I also had a big grin on my face, hearing Lex and Elon suddenly talk in another language. :) Consistency isn't perfect yet, as we're building the voice from scratch basically for each utterance. One the one hand, you want that, because the utterance might be more upbeat, or lower pitch, than the speaker's "average" voice. On the other hand, it sometimes introduces variance that makes the listener's brain go, "Uh.... is that another person speaking now?". If I had to dub 200 videos of a single YouTube channel, I would be able to fine-tune the voices of the main characters, and reserve the ad-hoc cloning for guest characters.

artninja1988 3y ago

Why do you keep the original audio track on the dubbed version? I think it sounds pretty distracting, although you do want to keep sounds other than the original voice I guess

askhan 3y ago
I think these little bits of the original sound really works well, helping us hear the original as well and keeping it less uncanny. What an amazing project!
- leobg 3y ago
  Thank you. Exactly. I had the OG voice removed at first. But I added it back in for exactly this reason. It also serves as a tool for AI accountability: It lets you "see" that the cloned voice is indeed saying the same thing as the original voice. That being said, it would be trivial to turn the OG voice off for anyone who wants to.
maxglute 3y ago
I dig it, it's like amateur underground over dubs of bootleg hollywood movies. Don't want to conflate amateur AI voice with actual personality on screen. The seperation is part of the experience.
gardenhedge 3y ago
News channels do that for interviews

simple10 3y ago

This looks amazing! Thanks for sharing. I signed up for the private beta.

leobg 3y ago
Thank you so much!

lxe 3y ago

What does this use behind the scenes? This type of stuff can get pretty expensive if you're relying on elevenlabs or heygen.

brody_hamer 3y ago
Yea my guess would be elevenlabs, which just recently announced this exact featureset.
leobg 3y ago
Let's just say this is NOT a wrapper around Elevenlabs or Heygen. I've looked at commercial voice cloning before. But, as you said, the prices seemed ridiculous. Before this, I made audiobooks for my daughter. Old, out-of-print books, turned into speech. If I remember correctly, with Elevenlabs a single book would have cost me > $100. At that price level, I can read the damn thing myself. What good is computer generated voice if it isn't at least 10x cheaper than doing it yourself? I'm just one guy. With me, it's just my time, one or two commercial licenses, and other than that just the raw price of running those GPUs.

scrollaway 3y ago

I know this is HN so I don't want to distract from the technical achievement and how genuinely useful this can be. I also don't want to tell you how to raise your kid. You do you, it's not my family. But I want to share how important it is to watch foreign spoken language movies and TV, especially as a kid, to be able to speak multiple languages later in life. You'll notice that in every country where TV and movies are regularly dubbed in the local language, the English levels go to shit. Dubbing is partially responsible for this because kids are not exposed to a different language on a regular basis. I remember wanting to watch a dubbed movie with my mom as a kid, and she told me "We will watch the original instead, dubbed movies don't have a soul". It stuck with me. She was absolutely right. Today I am working on my sixth spoken language. Causation not guaranteed, merely implied.

true_religion 3y ago
Counter point. My parents didn’t let me watch dubbed shows, and didn’t speak our native language because rhetorical wanted me to speak unaffected English. I can’t speak any languages, but in school my English was insanely good. To the point of perfect scores in the college scholastic exams, and when I was in uni for engineering, I took on an English major for fun with essentially no impact on my work load. You can generalize but you can also specialize.
Baeocystin 3y ago
I can't say I agree with dubbed movies having no soul. Greater accessibility to a wider audience is not something to deride, or hold in contempt. That being said, I do agree that listening to other languages is a great thing. My father was a linguist, and when we would watch subtitled media, we'd play a game where we'd try and hear the cognates, pick out the most common words, figure out the basics of the grammar as we went along. It was a lot of fun!
- leobg 3y ago
  One of my favorite movies was "Scent Of A Woman". But when I watched it in the German-dubbed version, I was appalled. It made the whole movie suddenly seem like a comedy. To me, the translation had killed its "soul", for lack of a better term. I still want my kids to learn English. And ideally also one or two other languages, like Chinese. As Nietzsche said: "So you have mounted on horseback? And now ride briskly up to your goal? Well, my friend - but your lame foot is also riding with you!"

sorenjan 3y ago

I know Germany dub most video, but wouldn't a seven year old be able to read subtitles? It's a great way for her to learn English, it's how most Swedes learn it before starting school. I think there's a pretty strong correlation between countries' average English proficiency and how common dubbing is. https://haonowshaokao.com/2013/05/18/does-dubbing-tv-harm-language-learning/ https://haonowshaokao.com/2013/05/18/does-dubbing-tv-harm-la... Edit: I forgot to mention that the samples on the website is impressive and well made. How do you do the speaker diarization and voice cloning?

poulsbohemian 3y ago
>I know Germany dub most video, but wouldn't a seven year old be able to read subtitles? I gotta say... while sometimes it is a necessarily evil, I would so rather not have to read subtitles. I often want to listen to a show so that I can also continue working on catching up on email, etc, IE: I can't read two things at once, but I can listen to one thing and continue working on something else.
supafastcoder 3y ago
I think it’s a cultural difference. I’m also from a non-dubbing country (Netherlands) and I can’t stand dubbed content either. On the other hand people tell me they can’t stand subtitles because it “reveals” what they’re going to say before they say it.
- lukan 3y ago
  "people tell me they can’t stand subtitles because it “reveals” what they’re going to say before they say it." I love watching movies in the original language, but this is something I hate as well, but something that can be avoided. Some movies get it right, though. The timing, just the words that are spoken and even different colors for different persons speaking (very rare, cannot even remember where I have seen it). That should be standard, but with most movies you can be lucky if the subs even match the plot and do not reveal too much.
  - 3y ago

nextworddev 3y ago

[flagged]

sss111 3y ago

can you add hindi as an output language, been meaning to build something like this for my parents. You saved me some work haha!

jianshen 3y ago

Wow this is amazing. If there was a locally running version available, I would gladly pay money for it.

leobg 3y ago
Thanks! Well, to make local happen I'd have to learn more about local app development. I'd also be worried about having to support a bunch of different platforms, and being beholden to ever changing rules made by App Stores and OS makers. I actually work on a 2015 Mac with a 2019 operating system. There are many great looking AI apps that I'd love to run but can't. Besides, it seems to me that making this centralized makes economic sense. I can just keep the GPU busy with lots of videos from many customers. I'm sure that's what most people think who build something: "The world would be so much better if everyone just came here and used this." :)

solardev 3y ago

This is really impressive! Can't wait to see this more fleshed out. I'd gladly pay for something like this (by the video, ideally). Some page feedback though: It seems to me that the video just keeps playing, with no way to restart it or scrub through the timeline. Each time I click a language, it changes the spoken audio but just keeps playing where it left off. That makes it hard to compare the same passage across different languages. Separately, I think there are also some errors in translation. For Sample 3 (about the vines), the original in Mandarin Chinese says something like "if this tree gets grabbed, the weed will climb up and wrap around it, and the tree won't be able to photosynthesize and will die". But the English mistranslation says "If it gets scared by people, it gets pulled off and messed with. It can't function. The evil effects? It just dies." There are also timing issues where the translations don't match up with the original subtitles or dialogue, and certain parts of the original audio just seem to be altogether ignored and not translated. Maybe displaying the translated subtitles, allow with a way for users to report errors, would help...?

leobg 3y ago
Thank you very much! Yes. You cannot control the video playback on the demo page. I made it so because I wanted a way to showcase how you can switch between languages. You can go from Elon speaking English to German, Russian and Chinese, each with just one click. Activating the player controls would have made the UI more complex and distracting. And it would have also made it harder for me to sync the timing between languages. Of course, the real output would be a proper player, with all of the controls. Or, for creators, raw files (video and/or audio, plus SRT subtitles). I also noticed problems in the translation of the Chinese video. I put it up there anyway, because I figured most people coming to my site would be English speakers, and being able to understand a Chinese video might be another interesting aspect, in addition to the idea of being able to turn your own English content into languages you don't speak. If this had been a pitch deck, I would have cherry picked the samples. But I wanted to share where the project is right now and see if anyone was interested. Premature optimization is the root evil of all programming. I think Knuth said that. And it's a trap I regularly fall into. So I tried to be disciplined this time. But if any Chinese YouTuber would ask me to dub their work today, I'd make darn sure that the translations were close to perfect. Meaning I'd allow the system to make changes to the way things are phrased if that's necessary for the purpose of timing or cultural context. But I wouldn't allow it to skip a thought from the original video, or say something something different. I've translated books by hand in the past. So this is something I care about. If the demo isn't perfect in this regard, it's because I didn't know if anyone was going to even look at my project. When I first posted this yesterday, my submissions didn't go beyond one comment for several hours. I already thought I had built another solution looking for a problem. :) If you're seeing dropped phrases, that's most likely because my arranging function failed. Basically, the translation ran longer than the original. The algorithm tried to speed it up and fit it in. But it failed and dropped it. Better handling of these overruns are on my to-do list. Neither drops nor speedups should be tolerated. In terms of self-correction, I plan to feed the translated audio back into the transcription engine. Then, an LLM can compare the translation with the original transcript. If anything is missing, the pipeline will be force to run again with slightly different parameters. There shouldn't be a human neccessary in the loop. Translation is what Transformers are best at.

paxcoder 3y ago

[dead]

gagabity 3y ago

This is great, I tried to do a similar thing once but my language is one of those that AI doesnt do well. I think you can look into muting the original voice in the video, I remember I saw there is some AI/tech that can separate audio into voice and nonvoice. Yandex browser does this in the browser, you open a YT video and it offers to translate, a few seconds later the voices are all in Russian, it's probably the most interesting production use of AI I have seen and for free. It's to Russian only unfortunately.

maxglute 3y ago

Very passable. Waiting for something local like this for foreign language PLEX and podcasts. As someone who views/listens to things at 2x/3x speed 10-15 bucks an hour is cost prohibitive.

leobg 3y ago
Perhaps you could ask some of your favorite podcast hosts to make a deal with me. Running a training on their voice once and then just re-using that will be much cheaper. Also, customers who buy in bulk will help me focus on this full time. There is huge potential for making this faster and cheaper. (Even using OpenAI is silly. Technically, I neither need GPT-4's knowledge nor instruction tuning. Both is unnecessarily adding cost and latency. But it helped me get the demo out.) Basically, the deal for Podcasters / YouTubers would be: - Get all their episodes converted into 15+ languages - Increase their reach today, while the novelty is high and the market is still uncrowded - They get to tell their sponsors that they now have reach across the language boundary
- maxglute 3y ago
  I don't know state of podcast ecosystem, but I think you should reach out to listennotes.com whose also a 1 man job that seems to elevate discovery and looks like has reasonable reach for producers. Or go hit up some popular western podcasts, you've definitely got something here and the execution is good enough.

theogravity 3y ago

Wonder how accuracy of the translation is measured (if at all).

leobg 3y ago
One idea I have is to use back-translation. After generating the new language audio, feed it back into the transcription, and then have an LLM compare it to the transcript from the original. Penalty for any thought/detail that is missing. If too bad, start from scratch.

daremon 3y ago

This is really amazing! Well done. I already joined the beta but I want to point out another use case here as well: In many countries (ie Greece where I'm from) movies and TV shows never get dubbed. We rely on subtitles. This means that if you can't see well (disability or age-related eye problems) and if your English is not excellent, then you are doomed to only watch locally produced movies & shows. This can be a real life-changer.

leobg 3y ago
Thank you! With movies, I think I could get into legally challenging territory. I guess all AI apps are, in a way. But with movies, there's an entire industry behind enforcing copyright. So I must tread carefully on that front. I made the jump from the courtroom into VS Code years ago. I really don't want to go backwards.
- daremon 3y ago
  I honestly don't see how movies are different with any content ie YouTube videos. I am pretty sure MrBeast etc have the same lawyers as any big studio. Could this run locally? I would certainly pay for that and you're off the hook on how anyone uses it.

waldrews 3y ago

Impressively done! It sounds like you're doing 1) doing voice recognition with voice time clues, which Whisper and the like provide, breaking it up into sentence (or similar) units; you don't need to time match individual words, but you need to time match at coarser grain. 2) using a translation engine that allows for multiple alternative translations 3) cloning the original voice, regardless of language 4) choosing the translation that has the best time match (possibly by syllable counting, or by actually rendering and timing the translations). If there isn't a close translation, maybe you're asking ChatGPT to forcibly rephrase? 5) Maybe some modest pitch-corrected rate control to pick out path that gets you closest to the timing? Did I get any of that right?

euazOn 3y ago
I also noticed that the third sample with Chinese sounds slightly sped up in the first English segment, so there may be also an element of postprocessing the dub (speeding it up/slowing it down).
- leobg 3y ago
  Yes. Though I don't like this solution. It breaks the flow. And it also doesn't really fully solve the problem. Overruns still accumulate if they happen too frequently. One second here, one second there... the further you get into the video, the worse it gets. I think it would be better to either slow down the underlying video or solve the overrun issue on the translation level. A good professional dubber will find translations that will even out in terms of timing. That's something an AI should be able to do better instead of worse.
- odiroot 3y ago
  The last sample from BBC is really hilarious when translated to Polish. Something definitely went wrong and the voice speaks like a drunkard.
waldrews 3y ago
Ooh and you're probably doing a split into voice and non-voice tracks of the original, and keeping non-voice at original volume, but lowering the voice track.

lIIllIIllIIllII 3y ago

This might be a game-changer for preserving declining languages.

leobg 3y ago
I'm not sure. Making this work is limited by the availability by reliable transcription models. Which, in turn, are limited the availability of large training corpora. Those don't exist for rare languages. Also, if people choose to listen to a Nepali speaker through an AI translator, that does give speakers of this language "a voice" - but it doesn't really preserve that language. You might argue, on the contrary, that it may remove any remaining incentives to learn that language.

azamba 3y ago

What would be required to add a new output language? E.g Portuguese? I know it’s supported as input.

deleted 3y ago

[deleted]

crtasm 3y ago

What potential issues with copyright are there from offering (paid) access to this tool to run on sources including Youtube, and with the output containing the source audio?

Freak_NL 3y ago
You're creating a derived work, so you would be violating someone's IP unless the licence is permissive, and making money of it. That usually attracts the attention of whoever owns the IP.
- dns_snek 3y ago
  Who's "you"? I'm obviously not a lawyer, but instinct tells me that end users are the ones creating a derived work by uploading a video they may or may not have the right to distribute. Linked website is just a tool, perhaps a cloud-based one, but still just a tool.
  - crtasm 3y ago
    Not just files a user uploads: "You can select videos from YouTube"
    - dns_snek 3y ago
      Thanks, I missed that. I can see how that would complicate things.
leobg 3y ago
You could have asked the same question when Google started building their index. Or OpenAI trained their models. I'm in Germany. I'm a licensed lawyer. I see the dangers. The safest path will be to simply offer the production of multi language translations to content owners themselves. Which is also going to be more efficient - translating the thing at the source, rather than having consumers each create their own translation. But the original intent for this has been to have my computer translate a video I want to watch with my kids in my private home. Technically, it's not "my computer" in the sense of being just the device that's physically in my home. There's stuff that happens in the cloud. Technically, copies are being made. So one could argue the point. For most people today, getting your content seen and consumed is the highest you can achieve. To sue someone from another country who cares enough to pay someone else to translate it for him would seem bonkers. But I'm sure there are lawyers who are desperate for work. Who cannot code, and can't be bothered to learn, but still want to do something "in AI". I'll at least give them a hard time. And dare they use ChatGPT hallucinated references on me! :)

hbarka 3y ago

Wow, this is impressive. Is there anything like it for live translation?

leobg 3y ago
Not on my to do list currently. EzDubs say they do live [0]. Also, a friend of mine mentioned some Samsung / Android app that does this? [0] https://ezdubs.ai https://ezdubs.ai

patrickhogan1 3y ago

Awesome! Large % of foreign streams have no proper dub.

godzillabrennus 3y ago

Would love this for Plex.

leobg 3y ago
Would love to. Can you broker a deal? :)

jeroenhd 3y ago

I've only ever experienced Dutch dubs in kids' TV but I feel like these examples show that your Dutch model may need some work. I can't judge other languages well, but I found the Taiwanese documentary dub especially hard to follow. I wouldn't have expected Dutch to be in there for how little the language is spoken and how often Dutch speakers will understand English, though! /offtopic It seems to do a pretty interesting thing where the first male voice has a bit of a Flemish/southern accent while the second male voice has an accent much closer to "Netherlands TV" Dutch. Reminded me a bit of the Lion King dub where the dub studio used Flemish voice actors to do the jungle animals (and Dutch voice actors for the savannah animals) to underline the "different world" Simba arrived in.

leobg 3y ago
Yes, that issue is also present in the German translation. I'm planning to monitor the output quality. Basically, feeding the translated audio back into the transcriber. Then compare it to the original transcript. Like a higher level loss function. I'll need this already because I don't speak all of these languages myself. But I can also use it to make the pipeline self-regulate and generate a new, better version if the last one scored too poorly.
- jeroenhd 3y ago
  Interesting, I can see how that approach would catch the weird voice lines. Just the different ways the languages get picked up and processed by the AI system could be interesting. If you find anything cool, I'd love to read a blog post about it!

cheriot 3y ago

Great tool! Curious if you've made more progress on diarization than what's described in this article? https://aipressroom.com/streamline-diarization-using-ai-as-an-assistive-technology-zoo-digitals-story/ https://aipressroom.com/streamline-diarization-using-ai-as-a...

leobg 3y ago
They use pyannote/speaker-diarization. I tried that, but it wasn't accurate enough for my purposes. Made a confusion matrix with voice samples from The Simpsons characters. It looked... well, confused. Am using a mix now of speaker embeddings and other signals (end of sentence, pause before/after, etc.). As you can see in the demos, it already works well for interview situations. It's when there are 3+ speakers and they talk over each other that the system gets confused.

changoplatanero 3y ago

Why doesn't youtube have something like this built in.

madduci 3y ago
I guess it's computationally expensive? OP states in the website that their solution takes 1 hour of processing for 30 minutes long videos . Now imagine offering this for all the YouTube videos available: - either it's done on their servers (hard to believe due to high costs) - either it's done on client side (which is also difficult, due to lack of processing power)
- leobg 3y ago
  Well, I'm also shamefully unoptimized at the moment. YouTube added auto-captions years ago. Long before there was Whisper, let alone things like Whisper.cpp. I imagine what I'm doing now is computationally no more expensive than what they did back then.
Freak_NL 3y ago
And automatically forced on every user depending on whatever their Google account is set to, just like the video titles which now get auto-translated without any way to turn this off. We're sliding back into a monolingual world. No thanks.
- leobg 3y ago
  As a native German, I also resent it when Google/Amazon/whoever tries to force a translation on me when I prefer the original language. So I wouldn't want to live in a world where everything would be dubbed into German for me. Not even if they used my tool :) Regarding YouTube: AFAIK, YouTube allows you to add multi-lingual voice tracks to your videos. Then, if the viewer has a preferred language set, the video will play in that language. Else in the language inferred by his browser/OS. But the user can also switch back to the original language, or any other language, right in the player.
  - 3y ago

Gunnerhead 3y ago

Amazing! I love this for dubbing, but was wondering if anyone knows of an AI powered subtitle generator for YouTube videos? I know YouTube has closed caption, but it’s terrible.

fbnt 3y ago
https://fluen.ai https://fluen.ai gives very accurate subtitles (and subtitle translations).
leobg 3y ago
Speakz actually generates subtitles as a byproduct. The idea is that you put in a video, you select the target languages. And then you get out, for each target language, an audio track and an SRT subtitles file. Someone else here asked about generating only subtitles, with no audio, as a cheaper option. So I'll probably add that as an option.
- Gunnerhead 3y ago
  I would love that to help learn another language!

leke 3y ago

I'm looking for a tool that will take a foreign language and automatically generate subtitles in my language. Anyone know of such a tool?

ocolegro 3y ago

So when is the crunchyroll integration rolling out?

exitb 3y ago

I suppose you're paying a lot for the voice cloning, so do consider that in many countries voice-overs are done with a single generic voice. Would you consider a lower price point service doing just that? I'm doing something similar and using GPT-4 for translation. What's unique about it, is that you can specifically prompt it to avoid long translations by rephrasing things, so you can buy yourself some time for the "Fledermaus".

leobg 3y ago
Using a single speaker makes sense when you're paying for human voice talents. But if you're using a computer to generate the voice, why not generate a voice that sounds like the original speaker? Much more fun to hear Elon speak Chinese :)
- exitb 3y ago
  To be blunt, because of the price. I’m running the whole pipeline of my toy project for much less than $5 per hour. If the voice cloning is the long pole in the tent, I’d just consider dropping it. Moreover, it’s a cultural custom. I’m from Poland and here the voice-over narrator is supposed to be generic and bland, so your brain learns to tune him out and take the emotional cues from the original voice.
  - leobg 3y ago
    Yes, in Germany we have generic voice-over narrators, too. In documentaries, etc.. They usually match the gender of the speaker, but that's it. Personally, I read most of my books with the iOS app Voice Dream Reader. That app still uses old TTS voices. They sounded great 3 years ago, but now sound robotic when compared to Elevenlabs or WaveNet. But, as you say, you learn to tune out the voice. I can read entire novels like this, and I still "hear" the different voices and personalities. It just all happens in my imagination. How much I'd need to charge to make the project worthwhile depends on many factors. And I didn't want to name a price now and then backpaddle a month from now and say it'll actually cost more. My pipeline right now is super unoptimized, to the point of being embarrassing. This can all be made to run much faster and cheaper. I agree with you that if the voice cloning part of my pipeline causes a significant chunk of the cost that the end user pays for the service, that I should then offer the option of using a "bland" voice instead for a lower price.

mannycalavera42 3y ago

it's a lovely parenting story. Let me tell you there is also a huge opportunity for the opposite use case. My elderly parents speak only one (non-Eglish) language. I would love to have a (cheaper) way to provide my parents with translated videos with the addition of (translated) subtitles. Subs are important because elderlies can have hearing issues great work, inclusivity is love

felixarba 3y ago
I second this! I was recently looking into a way to build something like this for my grandfather, but wasn't even sure where to start from the hardware side. I wanted to have hardware plug into TV receiver, generate subtitles for live TV program and then play it back on TV. Delay would likely be less than a minute but even a few minutes is not a problem really. Many people with a hearing problem would benefit from this and with AI getting so good at Speech-to-text, this can be done for quite a large population. If anyone has a recommendation on where to start with this, I'd appreciate it! Was thinking of using Whisper for subtitle generation, but not sure about hardware that can take in, and output HDMI and run this software
- leobg 3y ago
  I keep thinking about something similar. Also hardware. Also for my grandparents. My grandma is 95. Her vision is bad. Even using the phone (I'm talking old school landline) is getting hit and miss, because she can't see the buttons. Years ago, I set her up with an Echo Show. That works well enough for her to say "Call Leo". But Alexa is dumb. Sometimes, she'll mishear something and start playing music. Or start a monthly subscription... :) So what I'd like: - box - screen - far-field mic array - AI backend You could do a number of things with it: - manage a grocery shopping list (AI will notice duplicates and other oddities and ask) - communicate with the outside world (initiate calls, send emails and faxes, including to local businesses) - optional human oversight and/or permission settings (preventing the AI, say, from ordering groceries for more than $50 a week without a family member approving the order) Something like your "subtitle mode" could also work: "Listen to what is currently being spoken in the room (including the TV), and display it on the screen". My grandma has her TV running all day. So maybe one could ditch the screen and make it a "set top box". Add an IR port to it, so it can control also the TV itself. Something like that might work.

Timwi 3y ago

Would have been nice to get a download link for a program I can just run locally.

bbitmaster 3y ago
Alas! Didn't get the memo? We're In the era where AI tools are all "software as a service," and you must pay for individual inferences from the model. How could they charge for inferences if they gave you the model to download?
- Timwi 3y ago
  Amen, bro.
- leobg 3y ago
  I don't have access to any special model that I'm holding back from you in order to rent-seek. Anyone can learn to build something like this. The parts are all available out there. There's Whisper. There's Mistral-7b. There's Tortoise, Coqui, SV2TTS. There's Python. The bigger question is: Would you want to? I've been building web apps for several years now. I've sunk thousands of dollars into those projects. And literally years of my time. If I calculated my hourly wage, I'd be below a teenager mowing lawn. In Rwanda. And by a factor of 10x, probably. The real ROI here is the learning. And that's not something I'm "taking" from anyone.
  - rubymamis 3y ago
    You are awesome, man! Keep it going and all the best of luck!

welder 3y ago

I do the opposite and seek out videos in other languages for my kids to watch. Now they're learning German, Spanish, Chinese, and Japanese.

android521 3y ago

any opensource available?

sanjananb 3y ago

[dead]

ipsum2 3y ago

Does this use seamless4t or similar projects?

pavelboyko 3y ago

Please consider adding Simplified English as an output language option, preferably with a level, e.g., A2, B1, etc. This way, I can adjust the language complexity to my kids' level and then gradually remove the crutches as they improve in English.

leobg 3y ago
Yes! I love this! So you'd be translating English to Simplified English? Or are you talking from another source language? I've already been playing with this concept w.r.t. books: I take a non-fiction book. I'll have an LLM translate it with a specific audience in mind (say, a 7 year old girl with a certain background), explaining concepts and words that are likely unknown to that audience. And then converting the whole thing into an audiobook. Optional parental controls built in ("exclude violence", etc.). Nowhere near showtime, though. Another thing I'd love to work on is filtering existing content. There are millions of videos on YouTube. Right now, finding quality stuff that's fun to watch with my kid depends a lot on dumb luck. But what if I could filter by topic (semantic whitelist/blacklist, i.e. not keyword dependent), personality traits (OCEAN, MBTI), values (e.g. "curiousity") and language (reading level, vocabulary, words per minute, etc.)? I'd love that.
- vidarh 3y ago
  I'd love what they suggested as well, for other languages. I'm working on improving my French (and occasionally German), and I'm at a stage where I can follow along some French shows reasonably well if they're not speaking too fast (one of the first French phrases my French teacher in school taught us was "plus lentement, s'il vous plaît" - "slower/slowly please", for a reason), and if they're not speaking any particularly difficult accents, and not too much slang, but it's limiting and I'm often forced to keep English subtitles on as a consequence and it's sometimes too much of a crutch. It doesn't help that my hearing isn't what it was. Being able to "step down" the difficulty so that I can either turn off subtitles entirely or rely on French subtitles, or even much "difficult speech" and "simple subtitles" or vice versa seems like it'd be very useful in getting over that hump faster.

stuckkeys 3y ago

Cool project. What is the tech stack behind it. I can already guess few. 11labs, OpenAI…dreamtalk. There are so many similar services to what you are doing. What sets you apart? You should partner with local media or outside. Good luck! Check out https://www.flawlessai.com https://www.flawlessai.com they been around since 2018. Interesting stuff. When I first saw it in 2021 I was blown away.

YKreator 3y ago

Congratulations on this project! We spent 6 years developing the best solution for generating perfect subtitles automatically (https://www.Checksub.com https://www.Checksub.com). 2 years ago, with the arrival of new generations of AI, we decided to go a step further and add the possibility of automatically dubbing videos. But automatic dubbing requires manual adjustments for a comfortable result for the audience. For example, https://www.HeyGen.com https://www.HeyGen.com generates a video automatically, but offers very few editing options. That's why we focus on two things: 1 - to provide the best possible automatic quality 2 - to offer an advanced editor that lets you fine-tune your dubbing without having to go back to editing software. In any case, I'm delighted to see people working on this problem. I hope it will help develop this sector.

luxpir 3y ago
Thanks for Checksub, another happy user here. We take multilingual, AI-cloned audio from you guys (split from background noise, of course), after we've processed the subs professionally at our end, then we align everything in your tool and send off to a third service for lip sync. The result has blown away a few clients now. The CEO speaking 5 languages with perfect lip sync in their own tone of voice is quite convincing. Hopefully we can get it all in one tool soon.
leobg 3y ago
Great website! You're based in France? You should put a demo on your website. If there is one, I don't see it (or rather: hear). If you're interested in collaborating, me email is in my bio.

hombre_fatal 3y ago

What a great application of AI. The samples were amazing.

leobg 3y ago
Thank you!

k__ 3y ago

Would be awesome if it could also deep-fake the voice.