4 ms·

Forget subtitles: YouTube now dubs videos with AI-generated voices

snehk 3y ago
Anyone outside of English speaking countries should get ready to be detaulted to crappy AI voices instead of the original English without any option to disable it. Happened with subtitles as well. I don't think there's any way to disable subtitles once and for all on all devices. They come back even if you turn them off everywhere. I constantly have to disable them manually. If that crap starts with sound as well where I cannnot set a default, I'll just use a third party way to watch the videos.
superkuh 3y ago
It's cool and all but an awkward framing of the feature. Dubs don't really have the same functios as subtitle text. It's apples and oranges. As for the impact on people who want money to perform audio to audio translation services it sounds a lot like the luddites vs the power-looms. It's too bad for the people in that service industry but I'm 100% happy auto-translation is making everyone more able to communicate.
- vouaobrasil 3y ago
  I am wondering though: capitalism functions because of comparitive advantage. What happens when comparitive advantage vanishes in the realm of most people except the 0.1% in the tech companies. It's almost as if AI/technology is reaching a phase where it can reap every advantage, causing a sort of heat death analogous to the heat death in thermodynamics, where trade will eventually cease because no one has any meaningful contribution since AI can do it better. Maybe this will create a system where no one has to do any work and UBI will take care of everyone, but humans need a purpose and what happens to society when there is mass depression due to a widespread lack of purpose?
  - supriyo-biswas 3y ago
    Thank you for this comment, sums up my thoughts perfectly. I fear a regression of society to the days of zero-sum games. As an example, even though folks may say "open source models" are the answer, the only way to run said models at scale is to acquire physical resources such as electricity and hardware, and physical resources, by their very nature, are subject to zero-sum games.
  - falcor84 3y ago
    It's another case of "it is easier to imagine an end to the world than an end to capitalism". I for one can absolutely imagine having a "purpose" without a "job". Here are a few examples of activities that might give people a sense of purpose: art, traveling and exploring the world, lifelong learning, educating kids/adults about something you love, political activism, socializing in general, scientific inquiry, connecting with and helping animals, catching up on old literature/films/video games. And this is just me brainstorming for 2 minutes.
    - vouaobrasil 3y ago
      I am sure you can imagine such ways, and so can I. I personally hate working at a job and I don't even have a traditional full-time job anyway. I love doing a lot of things on your list as well and I feel like I have a purpose even without one.... but a lot of people are not so self-directed. And even if people are, a lot of them derive some satisfaction from taking their passion and sharing it with others. How are you going to share your passion in creating art when most people will be rather indifferent to art due to AI? And will most people even go to universities given that they don't need to go to get a job? Do you really think that if no one has any job whatsoever, they will all be self-directed enough to find true meaning in life? I could, and maybe you could to, but I really doubt if we are going to be transformed into a planet of traveling, science-teaching, animal caring, old video-game playing happy creatures any time soon.
      - falcor84 3y ago
        I don't have a full answer, and acknowledge that this will be a difficult transition, but humanity had lived for millennia before the modern concept of "a job", and many tribal societies still do, so we know it's possible. As for "happy", I don't think humans will ever be generally happy, but there's nothing new about that - our neural chemistry keeps us at a state of want as an adaptation; c'est la vie. As I see it, if you want something resembling lasting happiness, you need either drugs, or to relinquish "the self" via deep mindfulness, or to engineer ourselves beyond being human.
      - xboxnolifes 3y ago
  - visarga 3y ago
    > AI/technology is reaching a phase where it can reap every advantage No, the one who can reap every advantage of the new skills and knowledge learned by AI is actually you. They are all just one prompt away. > no one has any meaningful contribution since AI can do it better It won't be human vs AI, but human+AI vs human+AI. Has anyone noticed a trend? every AI task is fragile, they all need human oversight in critical scenarios. Looks like AI is only going to scale as much as we can scale oversight. It will have to be a human+AI team effort.
    - ben_w 3y ago
      > It won't be human vs AI, but human+AI vs human+AI. I think the metaphor is "centaur"; in chess we started with humans dominating, then AI got better and (after DeepBlue) centaurs dominated, then AI got better and humans only held them back. > Has anyone noticed a trend? every AI task is fragile, they all need human oversight in critical scenarios. Well, that's one way to prevent misaligned AI paper-clipping us. (Pity some of us keep being lazy and letting the fragile AI break stuff…) Given we don't know how hard it is to make a human-level general AI, there's no way to know right now if we're going to be in a world of (for any skill not already automated) human dominance, centaur dominance, or AI dominance, nor when any transitions between states will happen.
    - vouaobrasil 3y ago
      Short-term thinking only. human+AI will give way to AI...of couse, I can reap the advantages of AI. THAT is the problem: thinking about what I can do. That is the problem that plagues us. I was not thinking of I. I could easily reap the advantages of AI now, but what about future generations?
  - hfhdjdks 3y ago
    This is an old question in economics, but undoubtedly relevant now: "Is this new technology change different or similar to other technology changes?" In other technology changes, there are hundreds of jobs that disappear, but hundreds that appears...and, as far as I know, we haven't observed an increase in joblessness in the last, say 30-50 years. We went through the industrial revolution, the invention of personal computers and a big etc and it didn't happen. Is this one different? I don't have an answer, but I don't think the answer is obvious.
throwaway290 3y ago
If auto dub is as bad as auto generated subtitles, I am not holding my breath...
- l3x4ur1n 3y ago
  What are you on about? Youtube's auto generated subtitles are pretty good. granted, 5-10% of them are miss, but they usually convey at least enough to grasp the context. I would be lost without them (I'd have to be listening to really loud sound and/or focus more intensively to what is said)
  - throwaway290 3y ago
    Your estimate is way too optimistic. But also what it doesn't get is usually key subject content and what it does get is low hanging fruit guessable from context. Sometimes I try subtitles on something I cannot understand (because English is my second language) and that has not helped ever, it's always gibberish. I tried going subtitles only a few times and gave up because I have to rewind and turn on sound every few seconds.
    - danielbln 3y ago
      For me YouTube captions/transcription have always been really bad, but whenever YouTube switched to some Whisper equivalent (or Whisper itself, ha) it has improved tremendously and rarely gets things wrong, in my own experience.
  - Agentlien 3y ago
    This has really not been my experience... until just now. A few times before I've tried using auto generated subs and every time I turned them off within a minute because so much of it is nonsense. Last time was about a month ago. It seemed to get a lot of simple sentences right but every few it completely misparsed. Additionally, it seemed like every uncommon name or technical term became something else. In light of your comment I went to YouTube to evaluate it again and... it's a lot better than I remember! I watched a few minutes of a video analysing some music and everything, including music terms, was correct.
pasttense01 3y ago
Some of us who are English speaking still read subtitles on English speaking media because of lack of clarity of the voices or we are hard of hearing.
- almostnormal 3y ago
  If the dubbing machine can understand, it could produce a more readable version in the original language, not only a different one.
  - hiccuphippo 3y ago
    If it's the same machine as the one that generates the captions, then it's not very reliable yet.
    - NavinF 3y ago
      Have you tried recently? YouTube captions have improved substantially over the last year. In my experience it now performs at human level which might not be good enough for every use case but it does make the "transcript" button and full text search practical
rolph 3y ago
subtitles can be anything you want if you have an editor with subtitle functionality. it could be the dialogue from the video; it could be "thought bubbles" for the characters, or just ascii strings of whatever code. in a "good enough" sense end user subtitled video has potential use for steganographic methods.
jfernandezr 3y ago
Some of us use subtitles because we don't want to listen to the audio in situations where that would be disrespectful, like in the gym. In Spain there is a big dubbing industry with very good professionals that are able to transmit the acting emotions nearly as good as the original (but never the same). But listening to automated AI dubbing would never reach that level of acting, sounds awkward. Even listening to "latin" spanish dubbing seems out of phase to most of us. Anyway, YouTube should put some efforts on reviewing the (at least) iOS application localization in spanish, because it's impossible to see at a glance the age of a video in a video listing.
- Moldoteck 3y ago
  AI dubbing would never reach that level of acting, sounds awkward. - you should consider being more conservative in this direction)) I've said the same thing abt generating of images 5 years ago but the last progress of midjourney/dalle proved me wrong. We should not underestimate what hights human progress can achieve
  - danielbln 3y ago
    Op also clearly hasn't used Elevenlabs or similar tools. If you clone a professional narrator it already sounds incredibly good and effectively indistinguishable from a human. Giving acting directions to the model to steer the output (kind of like ControlNet does for Stable Diffusion) seems like a logical next step.
    - patall 3y ago
      But in this case, they want to avoid the human input. So, I guess, it would rather work by reading and copying the intonation of the source voice.
- hiccuphippo 3y ago
  Funny, for me the Latin Spanish acting sounds better and the Spain ones sound stiff. And for video games, the acting from Japanese voice actors usually feels better than the English ones. I guess it depends more on what you are used to than the actual acting.
haunter 3y ago
Not Youtube but I saw EZDubs on Twitter recently and honestly the results are not that bad Here is the Messi post match interview after his first game, mind you him and the interviewer not even speaking the same spanish dialect https://twitter.com/MLSes/status/1682579623089930241 https://twitter.com/MLSes/status/1682579623089930241 And this was the english dubbed version https://twitter.com/ezdubs_bot/status/1682612908604157952 https://twitter.com/ezdubs_bot/status/1682612908604157952 There are some mistakes like free throw instead of free kick but apart from that it's not that bad
butz 3y ago
Great, in a few years Youtube will be able to generate new videos with some sort of LLM for video equivalent and stop paying creators. And include as much sponsorships and ads as possible too.
- guerrilla 3y ago
  It is great because the demand for authentic content won't go away so this will create new networks which focus on promoting and ensuring that.
onetokeoverthe 3y ago
[dead]
tonnydourado 3y ago
I think I'd have a hard time coming up with a feature I wanted less than this even if I tried.
guerrilla 3y ago
I wonder if this could be used for language-learning. One of the biggest problem with learning certain small languages is that there's basically no interesting content produced in them (depending on your interests of course, but I ran into this problem multiple times.) For example, everything technical is done in English these days so you won't find too much advanced stuff on certain topics in Swedish, so you get stuck with repeating basics which gets boring after a while. There's also only certain countries which produce the kind of drama series or movies you want to watch. Sure, every country has some, but not always a lot, because again they import.
rcarmo 3y ago
As a native Continental Portuguese speaker, I find the Brazilian Portuguese translation jarring, and would never use it.
d3vmax 3y ago
While this may not be available to everyone right now, I am using a service from Finland https://www.hei.io/ https://www.hei.io/ to help me auto generate subtitles and "AI" voiceover. Upload youtube link/video, Generate eng subs. Fix english subs manually, auto generate subs into any of the popular languages (we have req. for Asian langs.). Manually fix the auto-generated asian subtitle from minor errors. Reupload it and get perfect true "AI" Voiceover. All AI voice over services will have errors in translation/meaning. If you want quick job, this does it. If you want accurate job, manually fix the subs created by such services. Anyways, what used to take us 2 weeks, takes us a day or two.
WirelessGigabit 3y ago
I want subs. I learned English through watching English TV with subtitles in my native language. And the generation after me started watching dubbed kids tv. Large decline in their English language skills.
hfhdjdks 3y ago
I grew up used to reading subtitles. Listening to the original voices and languages is super interesting to me. Besides some very specific cases (I can't look at the screen all the time) I don't have any use for dubbing.
mcpackieh 3y ago
If they allow this to work with English -> English, I bet it will be very useful for lecture videos where the speaker does the "uh uh uh um um umm" thing (since those utterances don't get added to the subtitles.) I'm looking forward to trying this. Also, I think some people are fixating on "forget subtitles" in the headline. Youtube obviously aren't going to take subtitles away. The article doesn't suggest that people stop using subtitles. It's just a meaningless figure of speech.
Fire-Dragon-DoL 3y ago
Subtitles are useful when you have children playing in the living room. Sometimes they will scream and you won't miss anything because of that

Anyone outside of English speaking countries should get ready to be detaulted to crappy AI voices instead of the original English without any option to disable it. Happened with subtitles as well. I don't think there's any way to disable subtitles once and for all on all devices. They come back even if you turn them off everywhere. I constantly have to disable them manually. If that crap starts with sound as well where I cannnot set a default, I'll just use a third party way to watch the videos.

superkuh 3y ago

It's cool and all but an awkward framing of the feature. Dubs don't really have the same functios as subtitle text. It's apples and oranges. As for the impact on people who want money to perform audio to audio translation services it sounds a lot like the luddites vs the power-looms. It's too bad for the people in that service industry but I'm 100% happy auto-translation is making everyone more able to communicate.

vouaobrasil 3y ago
I am wondering though: capitalism functions because of comparitive advantage. What happens when comparitive advantage vanishes in the realm of most people except the 0.1% in the tech companies. It's almost as if AI/technology is reaching a phase where it can reap every advantage, causing a sort of heat death analogous to the heat death in thermodynamics, where trade will eventually cease because no one has any meaningful contribution since AI can do it better. Maybe this will create a system where no one has to do any work and UBI will take care of everyone, but humans need a purpose and what happens to society when there is mass depression due to a widespread lack of purpose?
- supriyo-biswas 3y ago
  Thank you for this comment, sums up my thoughts perfectly. I fear a regression of society to the days of zero-sum games. As an example, even though folks may say "open source models" are the answer, the only way to run said models at scale is to acquire physical resources such as electricity and hardware, and physical resources, by their very nature, are subject to zero-sum games.
- falcor84 3y ago
  It's another case of "it is easier to imagine an end to the world than an end to capitalism". I for one can absolutely imagine having a "purpose" without a "job". Here are a few examples of activities that might give people a sense of purpose: art, traveling and exploring the world, lifelong learning, educating kids/adults about something you love, political activism, socializing in general, scientific inquiry, connecting with and helping animals, catching up on old literature/films/video games. And this is just me brainstorming for 2 minutes.
  - vouaobrasil 3y ago
    I am sure you can imagine such ways, and so can I. I personally hate working at a job and I don't even have a traditional full-time job anyway. I love doing a lot of things on your list as well and I feel like I have a purpose even without one.... but a lot of people are not so self-directed. And even if people are, a lot of them derive some satisfaction from taking their passion and sharing it with others. How are you going to share your passion in creating art when most people will be rather indifferent to art due to AI? And will most people even go to universities given that they don't need to go to get a job? Do you really think that if no one has any job whatsoever, they will all be self-directed enough to find true meaning in life? I could, and maybe you could to, but I really doubt if we are going to be transformed into a planet of traveling, science-teaching, animal caring, old video-game playing happy creatures any time soon.
    - falcor84 3y ago
      I don't have a full answer, and acknowledge that this will be a difficult transition, but humanity had lived for millennia before the modern concept of "a job", and many tribal societies still do, so we know it's possible. As for "happy", I don't think humans will ever be generally happy, but there's nothing new about that - our neural chemistry keeps us at a state of want as an adaptation; c'est la vie. As I see it, if you want something resembling lasting happiness, you need either drugs, or to relinquish "the self" via deep mindfulness, or to engineer ourselves beyond being human.
    - xboxnolifes 3y ago
- visarga 3y ago
  > AI/technology is reaching a phase where it can reap every advantage No, the one who can reap every advantage of the new skills and knowledge learned by AI is actually you. They are all just one prompt away. > no one has any meaningful contribution since AI can do it better It won't be human vs AI, but human+AI vs human+AI. Has anyone noticed a trend? every AI task is fragile, they all need human oversight in critical scenarios. Looks like AI is only going to scale as much as we can scale oversight. It will have to be a human+AI team effort.
  - ben_w 3y ago
    > It won't be human vs AI, but human+AI vs human+AI. I think the metaphor is "centaur"; in chess we started with humans dominating, then AI got better and (after DeepBlue) centaurs dominated, then AI got better and humans only held them back. > Has anyone noticed a trend? every AI task is fragile, they all need human oversight in critical scenarios. Well, that's one way to prevent misaligned AI paper-clipping us. (Pity some of us keep being lazy and letting the fragile AI break stuff…) Given we don't know how hard it is to make a human-level general AI, there's no way to know right now if we're going to be in a world of (for any skill not already automated) human dominance, centaur dominance, or AI dominance, nor when any transitions between states will happen.
  - vouaobrasil 3y ago
    Short-term thinking only. human+AI will give way to AI...of couse, I can reap the advantages of AI. THAT is the problem: thinking about what I can do. That is the problem that plagues us. I was not thinking of I. I could easily reap the advantages of AI now, but what about future generations?
- hfhdjdks 3y ago
  This is an old question in economics, but undoubtedly relevant now: "Is this new technology change different or similar to other technology changes?" In other technology changes, there are hundreds of jobs that disappear, but hundreds that appears...and, as far as I know, we haven't observed an increase in joblessness in the last, say 30-50 years. We went through the industrial revolution, the invention of personal computers and a big etc and it didn't happen. Is this one different? I don't have an answer, but I don't think the answer is obvious.

throwaway290 3y ago

If auto dub is as bad as auto generated subtitles, I am not holding my breath...

l3x4ur1n 3y ago
What are you on about? Youtube's auto generated subtitles are pretty good. granted, 5-10% of them are miss, but they usually convey at least enough to grasp the context. I would be lost without them (I'd have to be listening to really loud sound and/or focus more intensively to what is said)
- throwaway290 3y ago
  Your estimate is way too optimistic. But also what it doesn't get is usually key subject content and what it does get is low hanging fruit guessable from context. Sometimes I try subtitles on something I cannot understand (because English is my second language) and that has not helped ever, it's always gibberish. I tried going subtitles only a few times and gave up because I have to rewind and turn on sound every few seconds.
  - danielbln 3y ago
    For me YouTube captions/transcription have always been really bad, but whenever YouTube switched to some Whisper equivalent (or Whisper itself, ha) it has improved tremendously and rarely gets things wrong, in my own experience.
- Agentlien 3y ago
  This has really not been my experience... until just now. A few times before I've tried using auto generated subs and every time I turned them off within a minute because so much of it is nonsense. Last time was about a month ago. It seemed to get a lot of simple sentences right but every few it completely misparsed. Additionally, it seemed like every uncommon name or technical term became something else. In light of your comment I went to YouTube to evaluate it again and... it's a lot better than I remember! I watched a few minutes of a video analysing some music and everything, including music terms, was correct.

pasttense01 3y ago

Some of us who are English speaking still read subtitles on English speaking media because of lack of clarity of the voices or we are hard of hearing.

almostnormal 3y ago
If the dubbing machine can understand, it could produce a more readable version in the original language, not only a different one.
- hiccuphippo 3y ago
  If it's the same machine as the one that generates the captions, then it's not very reliable yet.
  - NavinF 3y ago
    Have you tried recently? YouTube captions have improved substantially over the last year. In my experience it now performs at human level which might not be good enough for every use case but it does make the "transcript" button and full text search practical

rolph 3y ago

subtitles can be anything you want if you have an editor with subtitle functionality. it could be the dialogue from the video; it could be "thought bubbles" for the characters, or just ascii strings of whatever code. in a "good enough" sense end user subtitled video has potential use for steganographic methods.

jfernandezr 3y ago

Some of us use subtitles because we don't want to listen to the audio in situations where that would be disrespectful, like in the gym. In Spain there is a big dubbing industry with very good professionals that are able to transmit the acting emotions nearly as good as the original (but never the same). But listening to automated AI dubbing would never reach that level of acting, sounds awkward. Even listening to "latin" spanish dubbing seems out of phase to most of us. Anyway, YouTube should put some efforts on reviewing the (at least) iOS application localization in spanish, because it's impossible to see at a glance the age of a video in a video listing.

Moldoteck 3y ago
AI dubbing would never reach that level of acting, sounds awkward. - you should consider being more conservative in this direction)) I've said the same thing abt generating of images 5 years ago but the last progress of midjourney/dalle proved me wrong. We should not underestimate what hights human progress can achieve
- danielbln 3y ago
  Op also clearly hasn't used Elevenlabs or similar tools. If you clone a professional narrator it already sounds incredibly good and effectively indistinguishable from a human. Giving acting directions to the model to steer the output (kind of like ControlNet does for Stable Diffusion) seems like a logical next step.
  - patall 3y ago
    But in this case, they want to avoid the human input. So, I guess, it would rather work by reading and copying the intonation of the source voice.
hiccuphippo 3y ago
Funny, for me the Latin Spanish acting sounds better and the Spain ones sound stiff. And for video games, the acting from Japanese voice actors usually feels better than the English ones. I guess it depends more on what you are used to than the actual acting.

haunter 3y ago

Not Youtube but I saw EZDubs on Twitter recently and honestly the results are not that bad Here is the Messi post match interview after his first game, mind you him and the interviewer not even speaking the same spanish dialect https://twitter.com/MLSes/status/1682579623089930241 https://twitter.com/MLSes/status/1682579623089930241 And this was the english dubbed version https://twitter.com/ezdubs_bot/status/1682612908604157952 https://twitter.com/ezdubs_bot/status/1682612908604157952 There are some mistakes like free throw instead of free kick but apart from that it's not that bad

butz 3y ago

Great, in a few years Youtube will be able to generate new videos with some sort of LLM for video equivalent and stop paying creators. And include as much sponsorships and ads as possible too.

guerrilla 3y ago
It is great because the demand for authentic content won't go away so this will create new networks which focus on promoting and ensuring that.

onetokeoverthe 3y ago

[dead]

tonnydourado 3y ago

I think I'd have a hard time coming up with a feature I wanted less than this even if I tried.

guerrilla 3y ago

I wonder if this could be used for language-learning. One of the biggest problem with learning certain small languages is that there's basically no interesting content produced in them (depending on your interests of course, but I ran into this problem multiple times.) For example, everything technical is done in English these days so you won't find too much advanced stuff on certain topics in Swedish, so you get stuck with repeating basics which gets boring after a while. There's also only certain countries which produce the kind of drama series or movies you want to watch. Sure, every country has some, but not always a lot, because again they import.

rcarmo 3y ago

As a native Continental Portuguese speaker, I find the Brazilian Portuguese translation jarring, and would never use it.

d3vmax 3y ago

While this may not be available to everyone right now, I am using a service from Finland https://www.hei.io/ https://www.hei.io/ to help me auto generate subtitles and "AI" voiceover. Upload youtube link/video, Generate eng subs. Fix english subs manually, auto generate subs into any of the popular languages (we have req. for Asian langs.). Manually fix the auto-generated asian subtitle from minor errors. Reupload it and get perfect true "AI" Voiceover. All AI voice over services will have errors in translation/meaning. If you want quick job, this does it. If you want accurate job, manually fix the subs created by such services. Anyways, what used to take us 2 weeks, takes us a day or two.

WirelessGigabit 3y ago

I want subs. I learned English through watching English TV with subtitles in my native language. And the generation after me started watching dubbed kids tv. Large decline in their English language skills.

hfhdjdks 3y ago

I grew up used to reading subtitles. Listening to the original voices and languages is super interesting to me. Besides some very specific cases (I can't look at the screen all the time) I don't have any use for dubbing.

mcpackieh 3y ago

If they allow this to work with English -> English, I bet it will be very useful for lecture videos where the speaker does the "uh uh uh um um umm" thing (since those utterances don't get added to the subtitles.) I'm looking forward to trying this. Also, I think some people are fixating on "forget subtitles" in the headline. Youtube obviously aren't going to take subtitles away. The article doesn't suggest that people stop using subtitles. It's just a meaningless figure of speech.

Fire-Dragon-DoL 3y ago

Subtitles are useful when you have children playing in the living room. Sometimes they will scream and you won't miss anything because of that