5 ms·

There's an art to subtitling that goes beyond mere speech-to-text processing. Sometimes it's better to paraphrase dialog to reduce the amount of text that needs

by jl6 2y ago

There's an art to subtitling that goes beyond mere speech-to-text processing. Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read. Sometimes you need to name a voice as unknown, to avoid spoilers. Sometimes the positioning on the screen matters. I hope the model can be made to understand all this.

entropie 2y ago
> Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read. Questionable. It drives me crazy to have subtitles that are paraphrase in a way that changes the meaning of statements.
lm28469 2y ago
As long as they're synced properly I don't care much, some movies/shows have really bad sound mix and it's not always possible to find good subs in the first place
eviks 2y ago
> better to paraphrase dialog to reduce the amount of text that needs to be read. That's just bad destructive art, especially for a foreign language that you partially know. > Sometimes you need to name a voice as unknown, to avoid spoilers. Don't name any, that's what your own eye-ear voice recognition/matching and positioning are for (also reduces the amount of text) > Sometimes the positioning on the screen matters. This is rather valuable art indeed! Though unlikely fit to be modelled well
- lxgr 2y ago
  > Don't name any, that's what your own eye-ear voice recognition/matching and positioning are for That’s tricky when one or more speakers aren’t visible.
  - eviks 2y ago
    They'd also have to start speaking at the same time and have similar voices to make it tricky
    - lxgr 2y ago
      You’re assuming that the subtitle user can hear the voice track here.
      - eviks 2y ago
        Yes I'm, the other option is the "hearing impaired" variant, let that one have all the extra info
        lxgr 2y ago
        I've first hand encountered several situations with subtitles where it would have been ambiguous who was speaking without speaker annotations, despite the voices being distinctive, me being able to hear them clearly etc. Just think of a rapid exchange with neither speaker on-screen for more than two or three sentences and replies. You can probably get 99% there without that for a lot of content, but I'd challenge the notion that this is somehow only important for hearing impaired viewers (or people just watching without clearly audible sound for other reasons).
        soco 2y ago
        I guess you can have that in a real life situation as well, where you don't have subtitles at all (hello AR) and still can handle. Do you want your subtitles full of metadata the whole movie for every movie and every day, for those several situations when the image director made a mess? You can always play that confusing scene again.
        lxgr 2y ago
        I definitely prefer subtitles to be as helpful as possible, yes. That includes having situationally appropriate metadata (which is different from "all the metadata, all the time"). I don't think that's an unrealistic goal to have for AIs; they're already extremely good at semantic scene description after all. By looking at the image in addition to just the audio track they probably also get a lot more metadata, which a refined world model will eventually be able to use just like a human subtitle editor can today.
- MindSpunk 2y ago
  Subtitles aren't just for foreign viewers though, they're also for native speakers who are now hearing impaired.
  - eviks 2y ago
    Sure, that's what HI version is for
diggan 2y ago
> Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read Please no. Some subtitle companies do think like this, and it's really weird, like when they try to "convert" cultural jokes, and then add in a bunch of more assumptions regarding what cultures you're aware of depending on the subtitle language, making it even harder to understand... Just because I want my subtitles in English, doesn't mean I want all typical discussed Spanish food names to be replaced by "kind of the same" British names, yet something like that is something I've come across before. Horrible.
- flippyhead 2y ago
  I totally get this. When I'm watching videos for the purpose of learning a language, I want all the actual words in the subtitles. But if I'm watching just ot enjoy, say in a language I don't care to learn, I don't mind someone creatively changing the dialog to how it probably would have been written in English. This happens with translations of novels all the time. People even seek out specific translators who they feel are especially talented at this kind of thing.
- scarface_74 2y ago
  I know a little Spanish and even I get annoyed when the English subtitles don’t match what they said in Spanish. Of course I expect grammatically correct Spanish to be translated into grammatically correct English.
- lifthrasiir 2y ago
  More precisely speaking, there are two kinds of subtly different subtitles with different audiences: those with auditory imparements and those with less understanding of given language. The former will benefit from paraphrasing while the latter will be actively disadvantaged due to the mismatch.
- close04 2y ago
  > Spanish food names to be replaced by "kind of the same" British names The purpose of a translation is after all to convey the meaning of what was said. So for example you'd want the English "so so" to be translated in Spanish as "más o menos" instead of repeating the translation of "so" twice. You don't want to just translate word for word, venir infierno o alta agua. A lot of dialog needs language specific context, many expressions don't lend themselves to literal translation, or the translation in that language is long and cumbersome so paraphrasing is an improvement. Like with anything else, the secret is using it sparingly, only when it adds value.
Aurornis 2y ago
AI subtitles are just text representation of the sound track. There is no need for artistic interpretation, substituting words, or hiding information. If it’s in the audio, there’s no reason to keep it out of the subtitle. An AI subtitle generator that takes artistic license with the conversion is not what anyone wants.
- Hard_Space 2y ago
  That doesn't work for idioms, certainly in Italian, which has multiple colorful metaphors which would be mystifying if translated directly.
  - saint_yossarian 2y ago
    I don't think anybody's talking about translations.
lidavidm 2y ago
I suppose this feature should have been termed closed captioning and not subtitling. It seems you're not going to get much sympathy for human translation here.
llm_nerd 2y ago
There is the art of subtitling, and then there is the technical reality that sometimes you have some content with no subtitles and just want a solution now, but the content didn't come with an SRT or better yet VTT and OpenSubtitles has no match. They're using Whisper for speech to text, and some other small model for basic translation where necessary. It will not do speaker identification (diarization), and certainly isn't going to probe into narrative plot points to figure out if naming a character is a reveal. It isn't going to place text on the screen according to the speaker's frame place, nor for least intrusion. It's just going to have a fixed area where a best effort at speech to text is performed, as a last resort where the alternative is nothing. Obviously it would be preferred to have carefully crafted subtitles from the content creator, translating if the desired language isn't available but still using all the cues and positions. Secondly to have some carefully crafted community subtitles from opensubtitles or the like, maybe where someone used "AI" and then hand positioned/corrected/updated. Failing all that, you fall to this.
raincole 2y ago
> Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read I really hope people to stop doing that.
latexr 2y ago
> There's an art to subtitling that goes beyond mere speech-to-text processing. Agreed. > Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read. Hard no. If it’s the same language, the text you read should match the text you listen to. Having those not match makes parsing confusing and slow. > Sometimes you need to name a voice as unknown, to avoid spoilers. Subtitles don’t usually mention who’s talking, because you can see that. Taking the source of a voice is uncommon and not something I expect these system to get right anyway.
mouse_ 2y ago
> Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read. Pretty sure this is a violation of the Americans with Disabilities Act, so illegal in the U.S. at least. Being Deaf doesn't mean you need "reduced" dialogue.
the_clarence 2y ago
The best subtitles I've had were fantrads
- latexr 2y ago
  Amazon agrees. https://www.reddit.com/r/amazonprime/comments/h922rg/primevideo_now_using_pirated_subtitles/ https://www.reddit.com/r/amazonprime/comments/h922rg/primevi...
thiht 2y ago
> Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read NO! I speak and understand 90% of English but I still use subtitles because sometimes I don't understand a word, or the sound sucks, or the actor thought speaking in a very low voice was a good idea. When the subtitles don't match what's being said, it's a terrible experience.
mohamez 2y ago
This is horrible for people who learn languages using TV Shows and Movies. One of the most frustrating things I've encountered while learning German is the "paraphrase" thing, it makes practicing listening very hard, because my purpose wasn't to understand what was being said, but rather familiarizing my ear with spoken German. So, knowing exactly the words being said is of utter importance.

by jl6 2y ago

entropie 2y ago

> Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read. Questionable. It drives me crazy to have subtitles that are paraphrase in a way that changes the meaning of statements.

lm28469 2y ago

As long as they're synced properly I don't care much, some movies/shows have really bad sound mix and it's not always possible to find good subs in the first place

eviks 2y ago

> better to paraphrase dialog to reduce the amount of text that needs to be read. That's just bad destructive art, especially for a foreign language that you partially know. > Sometimes you need to name a voice as unknown, to avoid spoilers. Don't name any, that's what your own eye-ear voice recognition/matching and positioning are for (also reduces the amount of text) > Sometimes the positioning on the screen matters. This is rather valuable art indeed! Though unlikely fit to be modelled well

lxgr 2y ago
> Don't name any, that's what your own eye-ear voice recognition/matching and positioning are for That’s tricky when one or more speakers aren’t visible.
- eviks 2y ago
  They'd also have to start speaking at the same time and have similar voices to make it tricky
  - lxgr 2y ago
    You’re assuming that the subtitle user can hear the voice track here.
    - eviks 2y ago
      Yes I'm, the other option is the "hearing impaired" variant, let that one have all the extra info
      - lxgr 2y ago
        I've first hand encountered several situations with subtitles where it would have been ambiguous who was speaking without speaker annotations, despite the voices being distinctive, me being able to hear them clearly etc. Just think of a rapid exchange with neither speaker on-screen for more than two or three sentences and replies. You can probably get 99% there without that for a lot of content, but I'd challenge the notion that this is somehow only important for hearing impaired viewers (or people just watching without clearly audible sound for other reasons).
        soco 2y ago
        I guess you can have that in a real life situation as well, where you don't have subtitles at all (hello AR) and still can handle. Do you want your subtitles full of metadata the whole movie for every movie and every day, for those several situations when the image director made a mess? You can always play that confusing scene again.
        lxgr 2y ago
        I definitely prefer subtitles to be as helpful as possible, yes. That includes having situationally appropriate metadata (which is different from "all the metadata, all the time"). I don't think that's an unrealistic goal to have for AIs; they're already extremely good at semantic scene description after all. By looking at the image in addition to just the audio track they probably also get a lot more metadata, which a refined world model will eventually be able to use just like a human subtitle editor can today.
MindSpunk 2y ago
Subtitles aren't just for foreign viewers though, they're also for native speakers who are now hearing impaired.
- eviks 2y ago
  Sure, that's what HI version is for

diggan 2y ago

> Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read Please no. Some subtitle companies do think like this, and it's really weird, like when they try to "convert" cultural jokes, and then add in a bunch of more assumptions regarding what cultures you're aware of depending on the subtitle language, making it even harder to understand... Just because I want my subtitles in English, doesn't mean I want all typical discussed Spanish food names to be replaced by "kind of the same" British names, yet something like that is something I've come across before. Horrible.

flippyhead 2y ago
I totally get this. When I'm watching videos for the purpose of learning a language, I want all the actual words in the subtitles. But if I'm watching just ot enjoy, say in a language I don't care to learn, I don't mind someone creatively changing the dialog to how it probably would have been written in English. This happens with translations of novels all the time. People even seek out specific translators who they feel are especially talented at this kind of thing.
scarface_74 2y ago
I know a little Spanish and even I get annoyed when the English subtitles don’t match what they said in Spanish. Of course I expect grammatically correct Spanish to be translated into grammatically correct English.
lifthrasiir 2y ago
More precisely speaking, there are two kinds of subtly different subtitles with different audiences: those with auditory imparements and those with less understanding of given language. The former will benefit from paraphrasing while the latter will be actively disadvantaged due to the mismatch.
close04 2y ago
> Spanish food names to be replaced by "kind of the same" British names The purpose of a translation is after all to convey the meaning of what was said. So for example you'd want the English "so so" to be translated in Spanish as "más o menos" instead of repeating the translation of "so" twice. You don't want to just translate word for word, venir infierno o alta agua. A lot of dialog needs language specific context, many expressions don't lend themselves to literal translation, or the translation in that language is long and cumbersome so paraphrasing is an improvement. Like with anything else, the secret is using it sparingly, only when it adds value.

Aurornis 2y ago

AI subtitles are just text representation of the sound track. There is no need for artistic interpretation, substituting words, or hiding information. If it’s in the audio, there’s no reason to keep it out of the subtitle. An AI subtitle generator that takes artistic license with the conversion is not what anyone wants.

Hard_Space 2y ago
That doesn't work for idioms, certainly in Italian, which has multiple colorful metaphors which would be mystifying if translated directly.
- saint_yossarian 2y ago
  I don't think anybody's talking about translations.

lidavidm 2y ago

I suppose this feature should have been termed closed captioning and not subtitling. It seems you're not going to get much sympathy for human translation here.

llm_nerd 2y ago

There is the art of subtitling, and then there is the technical reality that sometimes you have some content with no subtitles and just want a solution now, but the content didn't come with an SRT or better yet VTT and OpenSubtitles has no match. They're using Whisper for speech to text, and some other small model for basic translation where necessary. It will not do speaker identification (diarization), and certainly isn't going to probe into narrative plot points to figure out if naming a character is a reveal. It isn't going to place text on the screen according to the speaker's frame place, nor for least intrusion. It's just going to have a fixed area where a best effort at speech to text is performed, as a last resort where the alternative is nothing. Obviously it would be preferred to have carefully crafted subtitles from the content creator, translating if the desired language isn't available but still using all the cues and positions. Secondly to have some carefully crafted community subtitles from opensubtitles or the like, maybe where someone used "AI" and then hand positioned/corrected/updated. Failing all that, you fall to this.

raincole 2y ago

> Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read I really hope people to stop doing that.

latexr 2y ago

> There's an art to subtitling that goes beyond mere speech-to-text processing. Agreed. > Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read. Hard no. If it’s the same language, the text you read should match the text you listen to. Having those not match makes parsing confusing and slow. > Sometimes you need to name a voice as unknown, to avoid spoilers. Subtitles don’t usually mention who’s talking, because you can see that. Taking the source of a voice is uncommon and not something I expect these system to get right anyway.

mouse_ 2y ago

> Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read. Pretty sure this is a violation of the Americans with Disabilities Act, so illegal in the U.S. at least. Being Deaf doesn't mean you need "reduced" dialogue.

the_clarence 2y ago

The best subtitles I've had were fantrads

latexr 2y ago
Amazon agrees. https://www.reddit.com/r/amazonprime/comments/h922rg/primevideo_now_using_pirated_subtitles/ https://www.reddit.com/r/amazonprime/comments/h922rg/primevi...

thiht 2y ago

> Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read NO! I speak and understand 90% of English but I still use subtitles because sometimes I don't understand a word, or the sound sucks, or the actor thought speaking in a very low voice was a good idea. When the subtitles don't match what's being said, it's a terrible experience.

mohamez 2y ago

This is horrible for people who learn languages using TV Shows and Movies. One of the most frustrating things I've encountered while learning German is the "paraphrase" thing, it makes practicing listening very hard, because my purpose wasn't to understand what was being said, but rather familiarizing my ear with spoken German. So, knowing exactly the words being said is of utter importance.