13 ms·
Open-source tool translates and dubs videos into other languages using AI
- vouaobrasil 2y agoUnlike other methods of automation, AI is replacing human beings too fast. And before you say, "new jobs will be created" -- look at history. After the computer, new jobs have been created, but what kind of jobs? Every year, we are becoming more entwined in wage slavery as the wealth accumulates at the top and jobs become more meaningless. So, no, new jobs will not be created, except the kind of jobs that crush the human spirit into oblivion so that the rich tech oligarchs can play God.
- nickthegreek 2y ago>AI is replacing human beings too fast. citation needed.
- RIMR 2y agoThis is a subjective statement. If you disagree with it, counter it. Asking for a citation doesn't make sense here.
- chompychop 2y agoHow is it subjective? Can't terms like "replacing" and "fast" be quantified by metrics related to rates of unemployment and adoption of AI systems for tasks previously manned by humans? I'm not saying the data is readily available, but I do see a route to objectively measuring this.
- bsaul 2y ago« It’s too sweet » falls into the same measurable phenomenon, and yet remains highly subjective
- SoftTalker 2y agoIt's the word "too" that makes it subjective.
- homarp 2y agocan't I with that creates video that have the world as potential users? and the video can be the products, or tutoriala for another products. This allows me to do more, not less.
- homarp 2y agosee also https://news.ycombinator.com/item?id=41330587 https://news.ycombinator.com/item?id=41330587
- Kiro 2y agoStrange submission to post this comment on. It's not like translating and dubbing videos is the highest form of labor.
- wongarsu 2y agoIt's not like it's mindless drudgery either. Losing as little as possible in translation while fitting within the constraints of subtitles (people can only read so fast) can be very challenging. And decent dubbing tries to approximately match the mouth movements, which puts further constraints on the translation and often requires creative text changes. There is a point about industry size (not that many countries dub all their movies), but it is one of the intellectually more challenging professions.
- Kiro 2y agoTranslation work is often associated with horrible working conditions and poor pay. It definitely fits OP's description of "jobs that crush the human spirit into oblivion." This might not be the case if you're a renowned translator of famous literature, but that is not representative of the majority of translators.
- numpad0 2y agoIt takes a fluent, language specific bilingual with domain knowledge just to verify work. That's a high skilled job. It's just massively underappreciated because (same).
- insane_dreamer 2y agoIt actually requires a fair bit of skill to do it correctly and accurately
- surfingdino 2y agoThere is finesse to it. You are translating from one culture to another.
- notkaiho 2y ago
- insane_dreamer 2y agoThe problem is that we are hurtling towards the unknown without a plan, driven by the “need” to make higher returns for shareholders and to capture the new market. It may be that some new types of employment magically appear that soak up the jobs lost, but you can be sure there is no one working on solving that problem since the goal is to eliminate labor not create it.
- numpad0 2y agoI don't think generative AI is replacing humans at all. It's like how SQL replaced software engineers, with added bonus of copyright doubts gatekeeping common folks from exploiting it. It's obviously killing open Internet fast and encouraging power concentration too. It's worst of couples of worlds.
- IndySun 2y ago>...new jobs will not be created, except the kind of jobs that crush the human spirit into oblivion... AI certainly means everyone will be able to create 'art' and as a result we'll have more art than we know what to do with, music and images are already confetti, soon so will full length 'films/movies'. That leaves anyone who can actually sing, paint, play, dance, in prime position to take up those mantles.
- pizza234 2y agoThis is a fallacy called "Lump of labour fallacy" (see https://en.wikipedia.org/wiki/Lump_of_labour_fallacy https://en.wikipedia.org/wiki/Lump_of_labour_fallacy).
- ZoomerCretin 2y agoCalling something a fallacy does not mean it truly is one. I have not seen strong evidence that there is not a "lump of labor" in extremely high-rent countries like the US. For a sufficiently small period of time, there absolutely is a "lump of labor". If I were to go into a county with robots and rent them out to employers for 1/4 of the wage of their current employees, and they all fired their employees and accepted my robots, do you really believe that all of ex-employees would be able to find work again within 6 months? Or even 70% of them? What about their new wages? Do you think these new jobs would pay as well? I have a hard time believing that to be the case.
- pizza234 2y agoThere are a number of misconceptions when it comes to the lump of labor fallacy. The prevalent one is confusing the small and the big picture. If a factory introduces automation, no doubt that the resulting job losses are a problem that must be addressed. However, the lump of labor refers to the big picture - in the same time span, other jobs are created elsewhere. If they weren't, considering that automation started at least 200 years ago (in the most limited sense of the term), the whole planet would be out of jobs by now.
- vouaobrasil 2y agoIf it is such a fallacy, then why are there fewer and fewer options for those who don't want to be entrenched in technology?
- IG_Semmelweiss 2y agoWho's upset here about the hordes of horse manure cleaners that lost their jobs due to the advent of cars? Does anyone miss the stench of a 1800s street in the summer heat? No. Lets ask the influencers, the twitch celebs, the podcasters id they would prefer to shovel horse crap, or play with their e-device all day. And to be completely fair, the manufacturing jobs may be gone because elected leaders were told it was OK export the vast majority of those jobs (millions) abroad, where labor doesn't have govt benefits or red tape. This makes new jobs go to foreigners instead of staying stateside. So, we all save a few pennies on each item made in china and sold on Amazon, at the price of nuking all US craftsmanship and artisans.
- lmm 2y ago> After the computer, new jobs have been created, but what kind of jobs? Every year, we are becoming more entwined in wage slavery as the wealth accumulates at the top and jobs become more meaningless. What are you talking about? Many of us have tech jobs with much more comfort, creativity and autonomy than the jobs they displaced, and computerisation has made it much more practical for those who dare to strike out their own rather than needing wealthy family or friends before you can even begin to think of starting a business.
- CaptainFever 2y agoI know, right. Jobs created by computers have by far been the best paying, and most comfortable jobs that exist in human history. We're privileged.
- SoftTalker 2y ago> After the computer, new jobs have been created, but what kind of jobs? Um, jobs where someone under age 30 can be earning hundreds of thousands of dollars a year programming them?
- vouaobrasil 2y ago> Um, jobs where someone under age 30 can be earning hundreds of thousands of dollars a year programming them? Not a very fulfilling job in my opinion, except for the money.
- hatenberg 2y agoTransformer based AI is basically printing machines for the knowledge economy from existing labor patterns. Nobody currently can say which patterns it cannot extract, hence "we always figured out new jobs" is ... challenged
- nsonha 2y agoso maybe don't cling on to "the job" so much and hoping it somehow can fulfil your life. If the job can be automated by a machine then isn't it already meaningless and mundance and bore you to death anyway? I agree with the point about "wealth accumulates at the top" though. Maybe Karl Marx was right about a thing or 2. Maybe the distribution of wealth should not fall into the hand of non-elected corporations. Whatever it is, it should be determined by a democratic process and not some "market mechanism" that is actually just arbitrary algorithms optimized for metrics no actual human cares about.
- fleischhauf 2y agothe reaction is wrong. it shouldnt be "oh no, jobs are being removed" but "nice, less work more automatization, let's make sure we all benefit through less work and not only the rich with more profits"
- pjc50 2y ago> let's make sure we all benefit through less work and not only the rich with more profits It's fairly clear by now that that is not what happens and that the real AI risk is not the "grey goo" one of everything being converted to nanomush, but the age-old one of landlordism soaking up all the returns to economic activity.
- cubbic 2y agoOh I made something similar but for Netflix! https://github.com/cubbK/dubbing_ai_netflix_client https://github.com/cubbK/dubbing_ai_netflix_client I want to learn swedish and because there are so few dubbed movies in Swedish I take the subtitles(Netflix is good at having subtitles in different languages) and text-to-speech it :)
- sam_perez 2y agoHow would you rate the quality of the dubs? Do you think it's usable for learning? Seems like you could end up with some quirky learnings.
- woodson 2y agoNot OP, but I agree that this could lead to questionable learning outcomes, especially since Whisper isn’t that good for low-resource languages. It’s probably fine for languages like English/Spanish/Mandarin, though.
- ainonsense44 2y agoWenn ist das nun Stück gitt und Schlottermeyer.
- sunnybeetroot 2y agoThis is fantastic! Any direction on if I wanted to change the language to something else?
- birktj 2y agoWhy not simply watch native Swedish content instead? There should be quite a lot globally available for free from SVT.
- carlosjobim 2y agoYou can only watch Nordic government channels within the Nordic countries. Unless you get a VPN for it, I guess.
- ranger_danger 2y agoAn English README would have been nice
- pcarion 2y agothere is one: https://github.com/jianchang512/pyvideotrans/blob/main/docs/EN/README_EN.md https://github.com/jianchang512/pyvideotrans/blob/main/docs/...
- deleted 2y ago[deleted]
- microflash 2y agoProbably you missed it: https://github.com/jianchang512/pyvideotrans/blob/main/docs/EN/README_EN.md https://github.com/jianchang512/pyvideotrans/blob/main/docs/...
- ranger_danger 2y agoYep I missed the "English" text right on the first line, sorry.
- netsharc 2y agoHuh, if it's good enough, movies/TV shows dubbed with AI-clone of the original voice would be great (if we can ignore the ethics of using the actor's voice and the loss of work for the dubbing companies and actors). For example here's how weird Friends is in German: https://www.youtube.com/watch?v=nCoNSZV--z0 https://www.youtube.com/watch?v=nCoNSZV--z0 . Or Italian: https://www.youtube.com/watch?v=wO5qTzvyQ1s https://www.youtube.com/watch?v=wO5qTzvyQ1s Can AI detect the emotional tone of sentences yet, and recreate it in the target language?
- programjames 2y agoYes, pop in a normalizing flow that removes tone and recreates it with a small audio sample as context. https://arxiv.org/abs/2312.01479 https://arxiv.org/abs/2312.01479
- woodson 2y agoWhat I found is that, for cross-language use-cases, this often just applies the intonation of the “context” sample to the created sample, which, if they are from different languages, usually gives the wrong result (in the sense that it sounds off).
- devindotcom 2y agothey're workin on it
- yieldcrv 2y agoYes, have you used OpenAI’s voice model? It uses and reacts to tones My favorite conversation has been getting it to tell me about marshmallow vs marshmellow spelling and pronunciation, it became very strict but patient with me It can reply in other languages too, but I cant detect dialect as well to say
- _moof 2y ago> (if we can ignore the ethics of using the actor's voice and the loss of work for the dubbing companies and actors) This is a shocking parenthetical.
- randomgiy3142 2y agoBecause translations are copyrighted so it is complex to get legal rights for them.
- CyberDildonics 2y agoIs this using an open source text to speech model or is it going out to some other internet service?
- tourmalinetaco 2y agoIt can use Whisper which is open source, although optionally it can also use GoogleSpeech.
- CyberDildonics 2y agoI took a look and that seems to be speech to text / speech recognition but not text to speech.
- lossolo 2y agoBased on english doc it seems it's not dubbing but voice over.
- LeoPanthera 2y agoFor years I've wanted this for live TV. Even just subtitles would be amazing. I've always wanted to be able to watch news TV from other countries.
- alphabetatheta 2y agoSome of my friends have been using: https://app.vivalabs.ai/ https://app.vivalabs.ai/ as a more managed/paid service recently
- cyberax 2y agoI would pay a lot for a tool that removes the freaking laugh track from videos. I just physically can't watch them. I wanted to watch the Blackadder series, but I couldn't even get through one episode.
- deckar01 2y agohttps://github.com/facebookresearch/demucs https://github.com/facebookresearch/demucs If you can train an instrument model on laugh tracks Demucs should do that.
- mrtksn 2y agoBack in high school, when I got my first PC a plumber came over to fix some stuff and when he saw the computer he got excited and asked some questions and one of the questions was “how do you translate the VCD with this, I have a movie to watch but hate subtitles”. I was like “silly dude doesn’t know how computers work” but maybe I was the silly one who can’t dare to imagine how something like that can work.
- codedokode 2y agoAs I understand, it first extracts text from original video into subtitles, translates them using external LLM, and then converts text to speech. All of this is done using thrid-party solutions, and the project seems to be just a GUI app that allows to integrate them. You obviously cannot use this to translate songs or movies because this method loses important information like voice, intonation, etc. So it is still better to use subtitles.
- ewuhic 2y agoThis one does dubbing, but is there an equivalent tool for subs?
- fbnt 2y agoNot open source, but https://fluen.ai https://fluen.ai does a good job at translating subs, while using your own or standards style guides for the target language - ie. reading speed, max characters, "chunking" sentences where it grammatically makes sense, re-adapting them etc.
- cyanydeez 2y agoSo uh, what's with AI products throwing out the gold standard in testing these claims.
- paulkon 2y agoIs there an open source speech-to-speech model which retains intonation, cadence and delivery?
- gagabity 2y agoCool what languages can it do? Yandex browser does the most impressive version of this and for free but only to Russian I believe, its quite amazing it does appropriate different voices and follows the correct intonation for everyone, just takes a few seconds for a YT video.
- nsonha 2y agoThis could be useful in combating fake news. In many videos especially in political news, foreign languages are dub over with sometimes nuanced translation that can skew audiences to (mis)understand the content in certain ways.
- kevingadd 2y agoA translation lacking nuance/precision (due to being the work of machine learning) can also cause significant misunderstandings, though. I'm not sure you win or lose in that regard by switching from humans to machines.
- nsonha 2y agothe errors in ML translation would be systemic and less likely to be of human arbitration, which is less dangerous. Human translation could be deliberately misleading at a specific phrase in a specific interview.
- pjc50 2y agoThe AI is quite capable of inserting its own translation errors.
- skummetmaelk 2y agoLocalization and dubbing is a sad endeavour. By trying to accomodate everyone's individual preference for information transmission we accomplish nothing more than reducing our ability to understand each other in the long run. Having a Babelfish is all well and good. Until it stops working, and you realise no one can understand each other any more. Ironically localization is often pushed by well meaning Americans who only speak one language. "Oh, you're in a French speaking region. You MUST want French language. Let me force it down your throat while I prance around virtue signalling about how inclusive we are"
- written-beyond 2y agoInteresting take, never really looked at it that way.
- kevingadd 2y agoAre you saying it's a bad thing if the creator of a work decides they want it localized or dubbed into other languages? I don't understand why you want to take that choice out of their hands. Which languages someone speaks isn't simply a matter of "individual preference". Learning a new language takes a lot of time and energy, and people only have the time to learn a handful of languages in most cases unless they can make a career out of linguistics. i.e. I know a sprinkling of words in various languages, and I've started learning Japanese, but I simply don't have the time to also learn Mandarin, Korean, Cantonese, etc. So I appreciate it when authors of works in those languages offer localizations into a language I can speak, or when third parties spend their time translating stuff for free to make it available to a wider audience. What's the advantage of closing knowledge and communication off from a wider audience? Maybe I'm misunderstanding and you're just angry about Google Translate/DeepL etc (which I have a strong distaste for since they're Fake)
- skeledrew 2y agoNah the localization itself is fine. Where it becomes problematic is when there's no opt-out at the app level. Or perhaps it should even be opt-in.
- underdeserver 2y agohttps://deepdub.ai/ https://deepdub.ai/ have been doing this commercially for a while.
- hulitu 2y ago> Open-source tool translates and dubs videos into other languages using AI Is there any assessment about how good the translation is ?