19 ms·

AI could end foreign-language subtitles

• chupchap 5y ago
  I'm prefer subtitles as opposed to poorly voiced and dubbed versions that are prevelant. Also how will marvel.ai add nuance that is missing in subtitles? The missing nuance is not always about tonality but the context that is missed by machine translation.
• iratewizard 5y ago
  About as likely as I am to sit through a blockbuster with my friends and family deepfaked onto the cast.
  • echelon 5y ago
    That sounds like a lot of fun. Especially if you can change/correct character position and posture.
• aeturnum 5y ago
  This would be a lovely option to have and I find it hilarious that anyone imagines that subtitles are some kind of problem we are trying to get out of. Adding text to movies has a huge array of advantages: it preserves the original performances, it makes the dialog accessible to people with hearing or auditory processing disabilities, they can be multiplexed in a way that audio cannot, etc. Obviously it would be wonderful to have an 'autodub' option! But the times where one would use it seem quite narrow.
  • hepaminondas 5y ago
    Yes. It also forces people to read and allows them to be exposed to foreign languages.
  • hiptobecubic 5y ago
    I don't know. I'm not sure it's really possible to overestimate how many people hate reading and view it as a chore. Even without that not-so-charitable interpretation, there is still the "TV is on but I'm focused on something else" use case. Subtitles don't work at all for that.
    • fomine3 5y ago
      It may getting work with AR grasses. I can't wait.
  • Swizec 5y ago
    Coming from a non english country I can assure you that dubbing is the worst. When we were kids we would actively avoid watching anything that was dubbed as soon as we could read. It’s actually funny seeing how frustrated with subtitles my American girlfriend gets when we watch anything foreign. “You mean this basic skill every 7 year old has is hard for you, an educated super smart 30 year old? Bwahahaha” I still prefer watching everything with subtitles. Actors mumble a lot.
    • samtheDamned 5y ago
      > I still prefer watching everything with subtitles. Actors mumble a lot. Yep I have no problems hearing but newer movies and shows have uneven mixing where it can get really hard to hear over the background noise and subtitles come in clutch constantly for that
    • Kuinox 5y ago
      It depends of the country. French have great dubs and most of the movie released are watched dubbed.
    • stormbrew 5y ago
      I have complicated feelings about this. I also watch most things with subtitles these days, because either (or probably both) my hearing is going a bit or sound design/editing in films and tv shows has gotten worse, but there is a non-trivial difference in how I perceive things with or without. With subtitles my eyes are drawn more often to a part of the screen where nothing visual is happening. They aren't glued there, but it does create some bias. I have occasionally rewatched a scene without subtitles and noticed a lot of subtlety in the performance I had missed the first time though. That said, a bad dub is at least as distracting. Probably much more so, if anything. I would rather watch a bad sub than a bad dub, though I have occasionally enjoyed dubbed things as well. I don't think it's an all or nothing thing, but "never watch dubs" is the hard line heuristic that makes sense to me if you're gonna go that way.
      • ubercow13 5y ago
        At least with subs the performance is there for you to see (and hear), even if you sometimes miss it. With dubs, half the performance is removed completely.
  • Bilal_io 5y ago
    I can confirm as an anime fan, when watching One Piece I prefer to watch it in Japanese as the dubbed version sounds odd. it was an advantage growing up watching Hollywood movies in English with Arabic subtitles, it helped me learn the language.
    • moffkalast 5y ago
      It also helps when what you're watching has atrocious audio mixing (looking at you Nolan), or you have shit speakers, or actors mumble or whisper, in which case it's helpful to have them as an aid when something's not clear.
    • Dracophoenix 5y ago
      I guess you were a fan of MBC 2 then?
      • Bilal_io 5y ago
        And MBC Action
        • Dracophoenix 5y ago
          Were you a fan of MBC 3?
          • Bilal_io 5y ago
            Ages ago. But the ads on MBC network... The meme was that you're watching ads and they hit you with a show.
            • Dracophoenix 5y ago
              Do you remember the name of a cartoon short involving a rotund orange cat walking around the city doing engaging in various hijinks before it arrived home unnoticed? It's been bugging me for ages. It played between shows in place of a commercials in-between shows circa 2006.
  • commandlinefan 5y ago
    > subtitles are some kind of problem Heck, they're not even limited to foreign languages. I put them on in English because I'm starting to lose my hearing now in my late 40's and if I turn up the volume loud enough that I can hear the dialogue ok, the sound effects would be so loud they'd set off the neighbor's car alarm.
    • aeturnum 5y ago
      I also use them regularly in my native language! Very helpful in noisy environments.
      • ncpa-cpl 5y ago
        And very helpful for slang too. I speak Spanish well, but I always add subtitles in English or other language in case I miss a word of two from another Spanish variety.
    • epolanski 5y ago
      many reasons to use them: - family is sleeping so you keep volume low - hard to understand accents or lingo - bad audio mixing or bad audio systems that muffle words and rest of the sound
      • wyldfire 5y ago
        > hard to understand accents or lingo It's easy to tell which transcription services have access to the original script as a reference and which don't. I have subs on most of the time to catch subtle or barely-audible dialogue (or to decipher accents). There's sometimes when I can tell "oh gee that's not right at all but I can see how you misheard it." And there's other times when dialogue is coming from offscreen or a TV/radio and it's too far in the background to understand unless you are reading along with the script.
      • maroonblazer 5y ago
        - You're a euro-, arab-, and/or, asia-phile (I'm in the U.S.) and the sound of the language further transports you to those parts of the world.
  • nkozyra 5y ago
    > I find it hilarious that anyone imagines that subtitles are some kind of problem we are trying to get out of. I think there's a lot of gatekeeping about this. People - perhaps rightfully so - believe subtitled versions to be more ... "pure" than dubbed versions. I don't watch any language dubbed movies, but I wouldn't chide anyone who did. Not everyone wants to spend the whole two hours glancing up and down across the screen, nor are they necessarily wanting to learn a language passively, etc. So I think there's absolutely a perceived problem, it just doesn't apply to all viewers.
    • jonpurdy 5y ago
      > I think there's a lot of gatekeeping about this. People - perhaps rightfully so - believe subtitled versions to be more ... "pure" than dubbed versions. Well, they are more pure than dubbed versions. There is so much nuance and tone in the original language that is either left out or changed in the dub. Not to mention word order (subject-verb-object in English/Chinese, subject-object-verb in Japanese/Korean) that further changes the timing of impactful words. I am not looking forward to having to watch the dub versions of Totoro and other Miyazaki films when my daughter is between 2-7 years old...
      • Bedon292 5y ago
        When I was a kid, my dad was stationed in Japan. We watched all kinds of stuff, including Totoro on Japanese TV. No subtitles or dubs. Not a clue what they were actually saying but we all enjoyed it anyways. So you could always try it out and see how it goes.
        • jonpurdy 5y ago
          I don't know why I didn't think of this before... thanks! I will definitely try that out, especially since at such a young age she won't be able to sit through a whole film anyway.
      • earthboundkid 5y ago
        I have been watching dubs of Ghibli with my kids. It’s fine. It would be better to watch the subs for me, but I’m not going to get bent out of shape about it. They usually have pretty good English voice casting. I kind of like Kirstin Dunst as Kiki.
  • jmckib 5y ago
    I love subtitles, but for me they tend to spoil the timing of the lines being delivered, especially with comedy. Unfortunately subtitles are necessary even when I'm able to play audio as loud as I want, because it's almost always hard to understand at least 10-20% of the words that actors are saying. Maybe it would be nice to have subtitles that appear a word at a time as they are spoken, but I imagine that's not very practical for various reasons.
    • aeturnum 5y ago
      I would also love to a more responsive subtitle format! I've seen some really wonderful custom jobs where people use different subtitle colors for different speakers, place subtitles near the speaker in the frame, etc. I think there is a ton of room for improving the expressiveness of the medium.
      • zeroimpl 5y ago
        A pet peeve of mine has been subtitles which flip the order of words, particularly when the sentence is broken into two lines. Like the spoken text in foreign language is “A murderer is... Joe” but it gets translated to “Joe is a... murderer”. The other day I happened to see live content with closed captions, and I thought I would love the option to see subtitles like this, where words are translated precisely when spoken.
        • shkkmo 5y ago
          But translation doesn't really work like that. There are significant tradeoffs between verbosity, cadence, meaning and subtext when translating. The different contraints are why the forieng dubbibg and foreign subtitles are usually quite different. You generally only see word for word matches with subtitles when the subtitles are in the same language as the original lines spoken by the actors.
          • lisper 5y ago
            And the GP's example is the perfect illustration. "A murderer is Joe" sounds really weird. No native English speaker would ever say that. They would say, "Joe is a murderer" or "The murderer is Joe" or even "One of the murderers is Joe." But never "A murderer is Joe." Maybe in a Victorian novel, but not in a movie.
  • Findecanor 5y ago
    In my country (Sweden) subtitles are the norm for imported content, with most of that content being in English. I believe that reading the Swedish subtitles and hearing the English words simultaneously have played a large role in helping Swedes to learn English. However, it does help a lot that the grammar of Swedish and English is relatively similar. I have been trying to use watching anime to get better at Japanese, and the different word-order and clause-order in the spoken Japanese to the subtitles have often tripped me up — especially when longer sentences have got broken up into multiple subtitles.
    • webmaven 5y ago
      > In my country (Sweden) subtitles are the norm for imported content, with most of that content being in English. That was the norm as I was growing up in Israel, with the exception being children's programming (usually animation) that was dubbed. The dubbing was usually quite good, and done by actors prominent in children's programming. OTOH, being bilingual myself, the poor translation of subtitles was constantly driving me up the wall. I just couldn't ignore the mismatch between the dialog in English and the subtitles in Hebrew, but the rest of my family wouldn't agree to let me cover the bottom fifth of the screen. Semantically, the translations were generally accurate except in infrequent cases where the translator obviously misheard something or translated a phrase overly literally (more often with non-RP British or Australian English than American), but stylistically, the translation was always bland and "correct" (eg. "sure" and "yeah" and "you bet" would all inevitably be translated as "yes").
  • fxtentacle 5y ago
    Agree. Dolby Atmos releases tend to have a lot of noisy background sounds all around you, and then people silently mumble on the center channel. My English is really good, but some movies are still difficult to watch without subtitles. For example, Hawkeye jumped in between "painfully loud effects" and "barely audible dialog".
    • ASalazarMX 5y ago
      Eating pop corn while watching a movie has become increasingly difficult with this kind of sound mixing.
    • kpozin 5y ago
      This is actually an improvement over stereo soundtracks, as you can just raise the relative volume of the center channel!
      • fxtentacle 5y ago
        My TV can't do that :(
  • tlbsofware 5y ago
    Personally I would like if someone could use AI to enhance the voices of the actors, I find a lot of movies like to use loud ominous music while the characters are casually talking in a soft voice. Really drives me insane, and placing subtitles helps this problem but it creates a new one, I tend to get distracted by the subtitles and find myself not watching the movie
    • acchow 5y ago
      This could be an issue with your audio mixing in your particular speaker setup. Often times trying to play audio meant for surround systems (5.1, 7.1, etc) on stereo channels results in a poor audio mix.
      • thfuran 5y ago
        And Netflix in particular seems to love defaulting to 5.1 tracks when it shouldn't.
        • ncpa-cpl 5y ago
          Diney Plus too
        • acchow 5y ago
          I don't even see an option to manually select a stereo track anymore. They seem to have removed it from my Smart TV's app.
          • thfuran 5y ago
            I don't think I've yet encountered something with an English 5.1 track that doesn't also have an English stereo track mixed in with whatever foreign language tracks are present but who knows what features any given app has.
    • robocat 5y ago
      a way to remove the mood music and leave the voices and foley. I really enjoyed the cinematography of a film the other day, but the music completely ruined it.
  • FredPret 5y ago
    It would be hilarious to see famous actors deepfaked into speaking foreign languages in a feature film. I suspect it simply won't work - translating accurately is really hard, but translating elegantly is frequently impossible. Some things are just easier/punchier/funnier to say in language A than B
    • spc476 5y ago
      Laurel and Hardy would refilm many of their shorts in different languages, where they were speaking the foreign language. I always thought that was pretty cool.
    • Dracophoenix 5y ago
      Already the case with Tom Cruise and Jack Nicholson speaking in German. https://www.dailymail.co.uk/sciencetech/article-9613749/AI-company-uses-deepfake-technology-seamlessly-dub-favorite-actors-dialogue.html https://www.dailymail.co.uk/sciencetech/article-9613749/AI-c...
  • dheera 5y ago
    Totally agree, I often much prefer subtitles to dubbing as I can hear the original voice artist's tone of expression as it was intended to be. Also, languages are bridges, not barriers, by exposing yourself more to another language you will learn it if you don't reject it.
    • throwaway675309 5y ago
      That's a cute sentiment but there is no evidence that merely being exposed to a foreign language necessarily allows you to learn that language. I think you're confusing the movie 13th Warrior for reality. I have friends who religiously watch subtitled anime only and they are no closer to learning Japanese in any real sense of the word than a person who would watch the dubs, except for picking up a stray random word here or there, but knowing that kawaii means "cute" doesn't give you even a superficial understanding of Japanese. For that, you need to at least pick up hiragana and katakana. Learning a language requires deliberate work. There is no shortcut.
      • ncpa-cpl 5y ago
        English and Japanese are very different. I speak Spanish. Subtitles have helped me a lot when languages are similar. I watched three seasons of a Catalan TV show subtitled in Spanish, and now I can follow conversations and read it. Before that I wasn't able to understand spoken Catalan. Some language pairs were this may work are: - Spanish/Catalan - Portuguese/Spanish - Dutch/German
      • Izkata 5y ago
        Subtitles won't really help with sentence structure or conjugations, but it does help a lot with building vocabulary once you have the basics of sentence structure.
      • dheera 5y ago
        > no evidence that merely being exposed to a foreign language necessarily allows you to learn that language Um, that's how babies learn, I think there is plenty of evidence in every baby on the planet. People stop learning this way when they start thinking of languages as barriers.
  • tompetry 5y ago
    I am very surprised more people watched the dubbed version of Squid Game over the subtitles. I saw a friend watching the dubbed version and to me it didn't "preserve the original performances" as you say. Surprised subtitles are a barrier, I enjoy them personally, but I guess they have the data to know. I have tested out services that translate your "own voice" to another language and it was pretty cool, although I couldn't tell if the translation was quality. Automated translation isn't a solved problem, and this seems more of an issue than the tone of voice.
    • creakingstairs 5y ago
      When I've moved to New Zealand, I was taken aback at how many people just could not stand subtitles. This and your mention about 'barrier' reminded me of Bong Joon Ho (director of Parasite) calling this out: "Once you overcome the 1-inch-tall barrier of subtitles, you will be introduced to so many more amazing films."
      • wumpus 5y ago
        ... and a well-deserved "Best Film" Oscar in the Academy awards, the first subtitled "foreign" film so awarded.
      • derekp7 5y ago
        The problem I have with subtitles is the same reason I could never enjoy comics / graphic novels. I can either read and process the text, or look at / process the action (pictures). For me it is the same thing as trying to pay attention to two conference calls at the same time with a phone in each ear (is it just me that can't do this?). And it isn't just the mental processing, the rest of the screen sort of disappears from my perception when reading subtitles. Now silent films aren't a problem, as the text is shown before / after a given scene. So there is no processing two things at the same time. And it isn't a reading problem in general, because I'm an avid book reader and can consume books quite fast (although I only enjoy them if I read them slower and contemplate each scene).
        • 5y ago
  • starburst 5y ago
    Another advantage is that you can eat your chips and still understand what's going on.
  • aaron695 5y ago
  • throwaway675309 5y ago
    Then you're simply not thinking about it deeply enough. There are plenty of shows that someone may want to watch but may not be able to invest 100% of my attention watching that show, therefore with a show dubbed in their native language, they can still have the show in the background while being able to listen to the audio. With subtitles one has to constantly be looking at the actual show itself. Furthermore if one makes use picture in picture such as on a mobile platform or a tablet device the subtitles are so correspondingly small as to be basically hieroglyphic at that point.
    • distances 5y ago
      This use case is totally foreign to me. If I can't concentrate, I stop the show and continue some other time. Someone wants to check their phone or visit the loo? Pause the show. Linear television is history after all. That said, other people can obviously have their own preferences. Just sounds so weird to me.
  • vagrantJin 5y ago
    I watch a lot of foreign films and dubbed version misses a lot of nuance - complexity of spoken language, inflection, depth. So maybe I watch the kind of stuff where that kind of performace is required. Maybe it has certain genres or mediums where dubs work well like Romcoms or Marvel movies.
  • jack_pp 5y ago
    It might seem to you but producers care about data and I do not find it hard to believe people prefer dubbing to subtitles. I believe most people prefer audio input to text because it requires far less of their attention. You would be surprised how many people watch / listen to shows while doing something else and thus not giving the show their full attention. I'm guilty of this with podcasts.
    • meheleventyone 5y ago
      Maybe a system of always pandering to the lowest common denominator isn’t all it’s cracked up to be?
      • jack_pp 5y ago
        That's the most profitable system. It certainly doesn't produce the most quality. Just like making a tool that lasts a lifetime is not as profitable as making something that breaks. Maybe crowdfunding media content is the way to go for quality but I can certainly understand how public companies make decisions in their search for infinite profits.
        • meheleventyone 5y ago
          > But I can certainly understand how public companies make decisions in their search for infinite profits. Well yes because it's not exactly novel or complex. My point isn't that there is no rationale it's whether we should just blindly go along with it.
          • jack_pp 5y ago
            Who is we? We the creators? We the consumers?
            • meheleventyone 5y ago
              We as wider society which is why I criticize the system rather than the players.
  • Someone 5y ago
    I’m in the “pro subtitles” camp, too, but they too have disadvantages. By changing part of the screen, they don’t preserve the original performance, either (but IMO, that’s a lesser problem than dubbing) and their timing often is off. You’ll read a joke before or after it’s made, may know somebody gets shot before it happens, etc. I think aster readers will have a larger problem there. They (obviously) also work less well for slower readers.
  • southerntofu 5y ago
    After a decade of training their model Youtube can't even produce meaningful subtitles for ordinary words in English (let alone my native language, french) and we're supposed to trust these fucked-up AI overlords are going to produce any better dubbing? I have no doubt these solutions will be popular, because many people have difficulty reading and will prefer audio when given the option. I'm just confident they won't reach any measurable quality and the translations will be full of non-sensical/anti-thetical statements.
  • pavelrub 5y ago
    In my experience people who dislike subtitles are mostly native English speakers living in English-speaking countries, who are used to the fact that almost all the media they consume is produced in their native language, and so they’ve never developed the ability to read subtitles comfortably. When the occasional rare case arises where they need to watch a foreign film/show, they find it straining to both read the subtitles and watch the film, and so they conclude that subtitles are bad or “not for them”. In practice this stems from lack of practice — the perceived conscious effort of having to read subtitles while watching a film disappears entirely after you have done it a few times with sufficient regularity, and becomes completely effortless, subconscious, and automatic. But this is a skill that needs to be developed. Saying that subtitles are bad because they distract from the film is like saying that riding a bicycle is bad because you fall all the time, when in reality you just haven’t learned to ride one.
• black_puppydog 5y ago
  Completely besides whether this is possible (I think it is; it's an offline learning problem with huge resources/economic potential and it's getting attention) I actually prefer subtitles... Coming from Germany, this wasn't always the case, but nowadays watching dubbed content feels so weird and impoverished, I don't even want to imagine what it'd be like knowing that the voice is fully synthetic.
• throw_away 5y ago
  I would much rather we use AI to improve subtitling. Better auto speech recognition, per-word millisecond synchronization, positioning based on speaker/screen content-- all these could improve the subtitled experience and would be way less creepy than deepfake voices.
  • xiii1408 5y ago
    Yes! My perennial problem is that most Chinese TV content has excellent subtitles, but they're burned in to the videos. So if I want to watch a Chinese show with a friend who doesn't read Chinese, there's no option even for auto-translated subtitles. I've often thought of writing a script to generate subtitle files using text recognition, but haven't gotten around to it.
    • xiii1408 5y ago
      There's also the problem of content which doesn't have subtitles to begin with---YouTube's automatic subtitle generation is great, but could be improved and expanded to languages other than English.
      • Jach 5y ago
        It is expanded, at least to Japanese. What's also fun is you can then have it auto translate the auto generated subs. The results are... not typically great, but the potential!
        • userbinator 5y ago
          A long time ago, I tried the "autosub" feature with anime and it replaced understanding with unintentional (dry) humour. The output had phrases that resembled a news transcript, which is probably what they were training the system on.
      • moffkalast 5y ago
        > YouTubes auto mathy subtitle gen ration is grate FTFY
• endisneigh 5y ago
  What’s really surprising to me is that synching subtitles isn’t a solved problem. Why can’t someone just loosely transcribe without time stamps and sync it to the video?
  • echelon 5y ago
    Have you turned on Google Meet subtitles recently? They're so good it's uncanny.
  • sp332 5y ago
    I think you can do this with YouTube.
  • mateo1 5y ago
    What's the market for this? Very small.
    • distances 5y ago
      Every movie, every show ever produced? Subtitles are required even for domestic markets for the hearing impaired, even if we disregard the audience that prefers to have subtitles.
    • pessimizer 5y ago
      Literally everyone who subtitles anything.
  • thaumasiotes 5y ago
    Because synching subtitles to the video isn't even a goal. Most people read more slowly than they can listen. Many other people can read faster than they can listen. But you need subtitles that stick around long enough for the first group, and really, you want a comfortable margin so that people aren't frantically trying to finish reading every subtitle before it disappears. I watched the show 大江大河 ("Like a Flowing River") on Viki, which has excellent community-generated subtitling. The second season isn't available on Viki - only YouTube. (Actually, Viki seems to have lost the license to the first season by now, too.) And the subtitles are abominably bad, bad enough to make me stop watching the show. But they're perfectly synched - every subtitle on Youtube is exactly matched, millisecond-to-millisecond, with a Chinese subtitle which it attempts to translate.[1] [1] Ignoring the timing, which is much too fast for the English subtitles, the fact that each subtitle is translated independently is another huge problem. It leads to nonsense when one sentence is split across multiple subtitles, because the English and Chinese do not naturally present the same information in the same order.
• fgdelcueto 5y ago
  I grew up consuming movies and TV shows with subtitles so it has never bothered me. My theory is that if you're not used to them, then they are really annoying. Most of my American friends do not like them. There is something about watching a movie/show in its original language that you cannot understand. I think its awesome to be immersed in a different world, getting the sounds, inflections, etc from a different language of yours. I will always prefer subtitles to dubbed versions if possible. I understand some people don't, but I think it's just a matter of being exposed to it. And probably the younger, the better.
  • sidibe 5y ago
    Yes I think it's being a matter of being used to watching things with subtitles. Some people find it very distracting, I hardly notice at all watching something with subtitles since it is how i grew up. The only time I notice is when I want to do something else at the same time as watching the movie/show, then I quickly realize oh I can't do that I dont understand the audio. Dubbed things suck it is always the same voice actors too so they always seem not quite right for each new character.
• thinkloop 5y ago
  Translation barely works with text, it's still too literal, doubt this can be human-equivalent for a long time.
  • aaronsdevera 5y ago
    heck, auto-generated English captions on YouTube still struggle to keep up most the time... https://knowyourmeme.com/memes/youtube-automatic-caption-fail https://knowyourmeme.com/memes/youtube-automatic-caption-fai...
• echelon 5y ago
  Absolutely coming. Goupon founder Andrew Mason's new startup Descript does a fantastic job editing podcasts from text. This same sort of tooling is absolutely going to make it into film. This sort of stuff is already trivially possible with zero effort: https://fakeyou.com/w2l/result/WR:t5m9xayqcwkjf9wng3nb4sxgzhy8p https://fakeyou.com/w2l/result/WR:t5m9xayqcwkjf9wng3nb4sxgzh... In ten years, we'll be able to reposition actors and fix flubbed takes. We won't have to visit set, use expensive lenses, or do many of the things that make Hollywood the domain of the rich and well funded. In thirty years, we'll just tell the machines what we want to watch.
  • cellis 5y ago
    The problem is it's not very good and there is a vast uncanny valley. Phoneme / voice conversion is an incredibly difficult problem and not likely to be solved with any techniques known today, least ways not cheaply. Transformers show some ability to get it right , but if you wanted to use e.g. a Seq2Seq model you'd be hard pressed to come up with enough data that would generalize.
• ahmed_ds 5y ago
  On a tangential note: As far as training data for text and speech driven machine learning models go, isn't srt files truly remarkable? They are time stamped hence annotated precisely, hash and file matching makes media content to subtitle matching exact and overall there are language variations.
  • cfcosta 5y ago
    They are very artisanal, though. I used to do timing for anime fansubs when I was a teenager, and there's a lot of subtlety going on, specially when you don't speak the language (you're guided mostly by sound and phonetics, as the translations come more or less untimed). Timing is much more automatized nowadays, but still doesn't feel as high quality as handicrafted timing does.
• dmitriid 5y ago
  This will not happen for many, many years to come. That is, companies will try it, there will be some bad and ridiculous results, there will be pusheback and the tech will largely be ignored for a decade or so. Main problems will be intonation and inflection. What is a question to an English speaker is a regular sentence to a Russian. And so on. "Petabytes of data from primarily English-speaking libraries" mean jack shit if you wnt to actually do voice translation into/from foreign languages
  • nostrademons 5y ago
    I'd encourage you to listen to some cutting edge speech synthesis like Google Duplex: https://www.youtube.com/watch?v=ijwHj2HaOT0 https://www.youtube.com/watch?v=ijwHj2HaOT0 This is live in production today - business owners are getting calls from robots, and you can trigger it yourself with Google Assistant. [Disclaimer: I work(ed) on Google Assistant - I'm in the process of transferring out - and have listened to some presentations about how Duplex works. It's pretty crazy technology.]
    • cellis 5y ago
      It's still not very good. The state of the art vocoders ( StyleTron et al ) are years away from getting intonation right. But then if you could get intonation correct, you still need to be able to get pausing and pitch correct, let alone timbre and a few other things I'm forgetting from my research.
    • mateo1 5y ago
      Although this is irrelevant, is Google Duplex publicly known to be used by any call center company in any capacity?
• nefitty 5y ago
  An adjacent concern: Ads bad, yeah, but I appreciate the growing trend of subtitling them. I'm surprised there isn't some existing regulation on that, in the same way that streaming services are required to include them. The lower the cost of accessibility features, the more inclusive the media we consume can become.
• Rodeoclash 5y ago
  I think it's a great option but their is an art to translations. This applies especially to books (I'm reading an English translation of https://en.wikipedia.org/wiki/Zorba_the_Greek https://en.wikipedia.org/wiki/Zorba_the_Greek which is very well done) and while you do have less dialog with movies, I wonder if some subtleties will be missed, particularly around humour.
  • henrikschroder 5y ago
    This is something that bothers the crap out of me watching recent shows on Netflix or similar that come with official subtitles, and that is that they're often crap. You're spending millions on actors and sets and directors and visual effects and editing and everything else around the production, but the subtitles are some kind of shitty after-thought doled out to the lowest bidding subcontractor? In Sweden, everything foreign is subtitled, and as a result of the vast quantity of subtitles produced, the quality is high, so I've grown up expecting good quality subtitles. And then I watch something on HBO or whatever, and the subtitles are just bad. Clearly not proofread, clearly done by someone who didn't watch the thing, and just translated the written material as is. And the art side of subtitles is often neglected or plain just not understood. Timing is really important, which people understand when it comes to for example dialogue timing to make a joke land. But then the subtitles are shit, and ruin the joke because the subtitle timing is off, when you could have had good subtitles that worked with the material, preserving the timing! Why aren't studios and networks investing as much in the subtitles as the rest of the production? Yes, 95% of subtitles are easy to make, there's a single unambiguous way to translate what's being said into your target language, and by syncing it to the audio it's gonna be good enough. But some things are hard to translate, because there simply doesn't exist a 1:1 mapping between every language, despite Google assuming the world works like that. But YouTube's auto-caption thing is garbage. It's enough to sort-of kinda make out what is being said, but it's often complete garble, and there's zero understanding of good timing or the art side of subtitling.
    • henrikschroder 5y ago
      Everyone understands that acting, directing, editing, and sound editing is Art, it's creative work, you need humans to do it. But somehow, subtitles are Not Art? It's somehow not creative work?
• donkarma 5y ago
  Yeah I forgot the only use of subtitles is for hearing people
• kingkawn 5y ago
  I want to watch the original language with text, not listen to english adaptation whether synthesized or acted.
• kashunstva 5y ago
  I feel the same way about this as I do about reading poetry in translation, or listening to the (fortunately rare) opera sung in translation. There's something integral about the native qualities of the original language that I can't imagine would survive the AI process.
• 6gvONxR4sf7o 5y ago
  > More Netflix subscribers watched dubbed versions of "Squid Game" than subtitled versions. The default of you load it is dubbed, right? That’s what it started for me. So this stat probably has more to do with the default than anything else. They’re making a weak case that this problem even needs solving.
  • nkozyra 5y ago
    > The default of you load it is dubbed, right? I don't think so, at least not for me.
    • creakingstairs 5y ago
      I wouldn't be surprised if there is probably some A/B testing going on for this setting.
      • devmor 5y ago
        It's based on previous user preference. My wife and I watch everything subbed and eventually netflix just defaulted to it.
• nackerhewz 5y ago
  Like those hard to follow auto translated generated captions on YouTube, but spoken... It's a good idea but I'm not sure how well that would work.
• testermelon 5y ago
  The notion that some people prefer dub over sub is not new, but it baffles me that it is to the level of urgent business concern. From what I see in other comments, is it an American thing? We should invest in making people’s ear accustomed/used to hear other languages. It has great intercultural benefits.
  • pessimizer 5y ago
    I'm pretty sure it's also a Russian thing. In the case of English, the orthography is difficult, a large proportion of people are functionally illiterate, and even more have a tough enough time reading that subtitles just go by too quickly.
    • kpozin 5y ago
      I have a difficult time imagining that Russian speakers prefer the dubbed content they're offered. Most of the Russian dubs I've seen have one male voice actor doing all the characters (or with an additional female voice actor doing all the female characters, if you're lucky) with no attempts at timing, emotion, or intonation. The dub track is played on top of all the other audio, instead of being properly mixed in to replace the original dialogue track.
• misiti3780 5y ago
  I hope not, subtitles are a great way to learn a new language.
• friendlydog 5y ago
  AI silent movie subtitles or AI generated comic book style captions with cell shaded actors would be awesome.
• bsenftner 5y ago
  Oh come on everybody, this is a PR pseudo-advertisement for Veritone. The piece neglects to acknowledge that auto-dubbing is already a thing. No longer research has highly accurate voice synthesis of any specific voice with very limited source material. Realistic voice synthesis is very actively researched, and is already well into auto deep fake territory.
• cardosof 5y ago
  My wife is a translator and based on our conversations and her experience, subtitles are needed because people want to have the original audio + text, foreign subtitles are needed for obvious reasons, and human translators are needed for the "tricky" parts like new words and expressions, cultural context expressions, stuff that can't be directly translated, among other things. That's why in many countries a translated piece counts as a new/original piece, and the translator has the rights to claim some of the books IP.
  • Delk 5y ago
    > subtitles are needed because people want to have the original audio + text I'm a non-native English speaker, and I like watching English-language content with closed captioning in the original language. There's often a lot lost in translation, especially on the screen (written content might have different issues), because it's often hard to get the rhythm and timing right without changing the contents in non-trivial ways when the source and target languages are completely different. Since I can understand English fairly fluently and nearly always get the nuance that would be lost in a translated version, I prefer the original language, even for the subtitles. Since I'm not 100 percent, though, having the subtitles or closed captioning usually lowers my cognitive load. It also means I don't have to worry about missing something because of an unfamiliar accent, a very quickly spoken line, or some other random thing.
• aaron695 5y ago
• tluyben2 5y ago
  Most people here said it already; that would be a very bad development imho. But one that will happen as, at least in my distributed developer team (all continents, many countries) everyone young hates reading; chat, email, books, subtitles; they all hate it and want to listen and talk, not read & write. So it makes sense. Hope there will be choice always; I grew up with English movies with Dutch subtitles which taught me English and saw, at the same time, how awful it was to watch Rambo saying 'Ich bin John Rambo' with a flinty German dubbed voice. The horror.
• deleted 5y ago
  [deleted]
• nopenopenopeno 5y ago
  I would never want AI generated dialogue in place of subtitles. That sounds like a massacre.
• superfamicom 5y ago
  The article seems more against manual dubbing and related difficulties than actual subtitles. The best subtitles I've seen included relevant context on cultural jokes I would have completely missed had there not been explanation text. One example would be recently while watching a Korean drama with dubs that didn't address cultural items and beauty standard, the content itself came off very differently than the intention seemed.
• orsenthil 5y ago
  I speak a minority language that is primarily in spoken form by native speakers, very limited in number. I wonder, how AI technologies can be utilized to enhance the content or auto generating speech. I find it hard. Plus, the economic incentives for doing this work is highly limiting because of a small / limited user base.
  • david_allison 5y ago
    Could you get in touch. Email is in my profile. I'm about to start experimenting with custom TTS generation, and I'd like to keep you in the loop if my efforts are successful.
• suthakamal 5y ago
  An idea only a single-language speaker could love.
• codefreakxff 5y ago
  Oh. This would be terrible. Not only from a loss of cultural and language preservation, but also losing the actor’ performance. I really dislike watching dubbed shows. If this somehow became a thing it would be sad
• civilized 5y ago
  Pass. Dubs are usually bad enough when humans do them.
• invictus_78 5y ago
  Only people from English speaking countries don't like subtitles, every other country in the world are used to watching movies with subtitles.
  • cheeze 5y ago
    Used to does not mean "prefers"
• rexreed 5y ago
  This article is a thinly veiled promotion for Veritone's MARVEL.ai which doesn't even do translation into foreign languages. It's a text-to-speech encoder that has a variety of voices trained for different languages. The actual translations and subtitles need to be done most likely still by human workers or human-supervised machine translation. The article never really makes the point of the title which is that foreign-language subtitles will be ending any time soon. There should be a disclaimer on this advertorial piece.
• purplepatrick 5y ago
  I second the importance of subtitles. I grew up in a country where dubbing is a huge industry and everything is dubbed. I’ve also lived in another country where almost nothing is dubbed. After moving to the US and seeing many of the films I had previously only known in their dubbed version, I can no longer stand dubbing. I can tell that something is dubbed without having to look at the screen. I’d never want to not have subtitles — I’d rather not watch at all… That being said, as I understand the article this marvel ai thing is supposed to replicate the original performance using the original voice in another language? Of course that would be fantastic a solution, but I doubt that the technology is anything near what it wants to do…
• joemelonyeah 5y ago
  AI can generate voices similar to the original actor, but the script will still have to be translated and proofread by expert translators. No AI can fully replace them.
• pavlov 5y ago
  I want the opposite: subtitles on everything, just not directly on the picture. Opera houses have subtitles on a separate text-only screen (or screens embedded in seats). Why can’t I have a device like that at home? A narrow subtitle display that sits below the TV, sort of like the Touch Bar on the previous-generation MacBooks. I’d buy that.
  • KeepFlying 5y ago
    Well now you've got me thinking. That has to be very feasible for any content you own the file for, but I wonder how steaming services currently handle subtitles. Are they embedded into the video stream or are they their own channel? Could a third party pull them out and pass them along to a separate device like that? Does anyone with more experience have any idea?
    • Morgawr 5y ago
      Definitely not embedded into the video stream, at least for Netflix or Youtube. There's actually extensions that take the subtitles and manipulate them (even put them outside of the video stream) so you can use them for various language-learning purposes. Look at stuff like Language Reactor (https://www.languagereactor.com/ https://www.languagereactor.com/), previously Language Learning with Netflix. There's specific browser extensions like subs2srs and yomichan (https://animecards.site/yomichansetup/ https://animecards.site/yomichansetup/) which are very common in Japanese language learning communities for people to use on subtitle lines to get direct translation/dictionary entries of certain words and be able to send them directly to Anki (SRS flashcard software).
  • jeroenhd 5y ago
    There's this: https://play.google.com/store/apps/details?id=com.google.audio.hearing.visualization.accessibility.scribe https://play.google.com/store/apps/details?id=com.google.aud... You can install it on your phone and it'll subtitle your life if you can find a good place to put it down. It's meant for the deaf and hard of hearing (hence features like visual alarms) but in my experience it'll transcribe stuff that's being said pretty well as long as the mic catches the audio you want to hear. Perhaps it would be even better with a quality (bluetooth? 3.5mm?) external microphone.
• zarmin 5y ago
  Why is this article written in bullet points?
• the_af 5y ago
  I like listening to the original audio of foreign movies and it's partly how I learned English, and how I pick up random words from languages I can't really speak. I also respect actors and actresses enough to appreciate their performance includes their original voices. I'm a fast reader and subtitles are like a second nature to me, to the point I don't "remember" having read them (likewise I sometimes don't remember if I read a novel in English or Spanish unless a particular turn of phrase caught my attention). About the only thing I agree with is that dubbing (barring some honorable exceptions) sucks. I did listen to a part of La Casa de Papel's English dub -- the horror! In short: do not want this.
  • chupchap 5y ago
    I dont understand Spanish or Portuguese, but I think a lot of emotion that is conveyed in the way the languages are spoken is lost in the dubbing. Reading the subtitles and just hearing the actors say the same in Spanish conveys it better for me.
  • TT-392 5y ago
    I have got pretty serious dyslexia, and thus I have problems with reading subtitles fast enough. As a child this basically meant no subs for me. So I definitely see the value in dubs. These days I have gotten a bit faster at reading and I got things like more readable fonts to help me (also, some subtitle systems are just terrible, I often torrent stuff that is on netflix despite having an account because their subtitle system sucks). Yet, I am from the Netherlands, and here, with basically nothing except media for children being dubbed, I have gotten better at reading them and used to it. And honestly, despite the effort I have to go through, I totally agree. It is totally worth the extra effort. I'll always prefer the original language, no matter the quality of the dub. Language is just an important part of the culture of the original content.
• bovermyer 5y ago
  A couple points here. 1. AI might be able to prevent bad dubbing/subbing, but it will not be able to compete with _good_ dubbing/subbing in our lifetimes. There's an artistry there that no AI can match. 2. I love subtitles and prefer original audio + English subtitles in anything foreign-language. Besides the voices matching up to the movement of the actors' lips and thus not breaking my immersion, I also adore hearing how familiar emotions are reflected in languages I do not know. It makes me feel more connected to humanity.
  • Gigachad 5y ago
    > There's an artistry there that no AI can match. This has been said a few times already and been proven false. The quality of GANs is moving at a staggering pace.
    • devmor 5y ago
      How can something that can't be proven true be proven false? "Artistry" has no concrete definition and is entirely based on the opinion of the viewer.
      • Gigachad 5y ago
        It's easy. Blind test humans and ask them which one they find more "artful". Once the average person can not tell between an AI and human creators, the AI is complete.
        • devmor 5y ago
          And what happens when public perception of "artful" changes in 10 years - now the AI is not complete anymore? You can't base a metric on something with no measurement.
    • bovermyer 5y ago
      Saying there's proof and there actually being proof are two very different things. Show me the proof.
• 29athrowaway 5y ago
  Why end at subtitles, you can also do it in videoconferences.
• LetThereBeLight 5y ago
  It is striking to me that this article doesn't discuss the negative impact such a technology will have on actors and translators. In fact, the author makes the surprising claim that there is a shortage of translators and voice actors, which is very hard to believe.
• Terry_Roll 5y ago
• deleted 5y ago
  [deleted]
• esjeon 5y ago
  I’m highly skeptical about AI-replacing-subtitle-as-a-whole. We all know what’s the problem: the quality. We already know that translation is difficult. We already know that acting is difficult. The only feasible outcome here is deepfaking the voice from the recordings from localized voice actors to the actual movie actors. Perhaps, in the future, a voice actor might be doing all of (1) translation (2) voice acting (3) deepfaking in one shot. One person might be able to dub the whole movie - alone. But, for sure, we won’t be able to remove human from the process.
• mcoliver 5y ago
  Couple thoughts: - The hard part about subtitles is not translation it's contextualization. Languages are not variable replacements and words have different meanings based on the context of the sentence. Languages have nuance, phrases, and slang unique to their cultures which further adds difficulty. - coupled with the above, permutations can get large with translated dubs. Which subtitle are you supposed to show? Most translated subs are done off the original audio track and the dubs are done separately. So for example with an English show, the English subs match the English track, but the translation will be off for the Spanish audio track. And the Spanish dubs will not match the Spanish subs either. - lip sync on dubs is a hard problem for many reasons ( full face and body emotion, how words are said in the native tounge, etc..) and will significantly improve the viewing experience. You can do creative things like have fewer full face shots to help with this. It's also important to get the audio mix right so it doesn't sound like an extra track. Keep in mind you now need more videos to cache and store and the ability to swap video streams when you swap audio streams. - animated content offers other approaches than leveraging nn/deepfakes because the assets are already rigged. The use of game engines offers some really interesting opportunities here.
• divbzero 5y ago
  The article would be far more convincing if it argued that AI could end foreign-language dubbing. Distaste for poor dubbing only partly explains the demand for subtitles. Film aficionados would scoff at the idea of abandoning the original audio, and even casual appreciators of film recognize the value of hearing the original tones and accents.
• zxcvbn4038 5y ago
  This would be interesting if it worked - I recently re-watched The Expanse in German with subtitles on but it wasn’t very helpful as a lot of the time they simply gave up and used the English terms. Some are a bear to translate - try saying “donkey balls” in German, “eseleier”? The captions reverted to English and I didn’t really pick up what the voice track said but wasn’t even close. I had much better luck with Star Trek TOS, very little deviation between the English and German dialog, and the subtitles matched up well with the spoken dialog. Much better for learning - though “er ist tot Jim” doesn’t quite have the same punch IMO. https://www.youtube.com/watch?v=BOtx4Jk-OaQ https://www.youtube.com/watch?v=BOtx4Jk-OaQ
• estebarb 5y ago
  As a programmer I think that this is awesome. But... I'm also thinking that we may be missing opportunities. For example, I prefer to see subtitled movies in their original language (I'm a Spanish native speaker). But there are some dubs that are awesome, even better than the original: for example, Shrek. I'm concerned that eventually studios may just throw the script through Google translate and call it a day.
• Zenst 5y ago
  Whilst this is good, as long as it is an option and the user still has the choice to have the original language with subtitles then great. Many enjoy films in other languages they don't fully understand for many reasons, learning the language, nuance of the language and tonality, many things. After all, a Shakespeare love scene in Klingon is just not going to cut the ambience like many other languages. But one thing that bugs me, why is it that even today - we have to have programs with a hard-overlay of sign language interpreter in a you have it if you want it or not and the selection is limited. For those who depend upon such sign language, wouldn't it be great if they could watch anything and some AI would do the sign language for them. You would of thought that such a task would be easy, and the market is there but yet, it seems to elude us. So my hope is that whilst this is useful, it may well open up avenues like the above in which these tools can be leveraged in more practical ways for a group of users who in effect have what they can and can not enjoy, somewhat overly limited as they are at the mercy and whims of the broadcasters who in many instances, only do them to quota tick some regulation put in place to at least not totally shut out a group of people who can't hear at all.
• j7ake 5y ago
  Even as a native English speaker, I prefer watching English speaking shows with English subtitles. I find I don’t need the volume as loud and I get retain more of the dialogue.
• jillesvangurp 5y ago
  AI is also able to create sub titles. So, people will have a choice. Many people prefer subtitles. Dubbing movies is common in big countries where the locals speak nothing else than their own language. Here in Germany, you have to pay attention if you go to the cinema but they do have original language movies as well. Lots of Germans go to see those instead of the badly dubbed versions. I tossed out my TV ages ago because I don't speak German very well and don't like watching dubbed content. Especially if that content is originally in English which I actually can understand. One annoyance with the local netflix here is that while they have most content with the original sound track, they often swap out a lot of the text in these movies with German text. E.g. movie fragments showing a news paper or somebody's text messages, or the famous rolling text in star wars. And of course a lot of American movies or series sometimes feature people not speaking English. Netflix solves this by offering German sub titles. And sometimes those are actually baked into the movie. Par of the issue is that there's a big dubbing industry that is incentivized to keep on dubbing everything whether people like it or not. AI will decimate how profitable that business is.
  • distances 5y ago
    > Netflix solves this by offering German sub titles. This is my pet peeve about streaming services. Why would they have only German subtitles in Germany? The licensing costs of all major languages can't be prohibitive. I only watch content that has English subtitles -- Netflix has thankfully improved on including them, but otherwise you're mostly out of luck. Even Blu-rays need to be imported if you want to have English subtitles on English movies!
    • Mindwipe 5y ago
      > The licensing costs of all major languages can't be prohibitive. It's not the cost, it's that their removal is a disincentive for people to use VPNs to hop outside of their usual residential territory.
      • distances 5y ago
        Netflix doesn't ship all the languages even with their own content, so it can't be the only reason. But good point, never thought of that, and truly sad if access restrictions again is the reason for legal services delivering less than piracy.
• errcorrectcode 5y ago
  Deep fake audio plus alternate video tracks to lip/face synchronize too? Why not just deep fake the entire movie without using any live actors or sets too?