5 ms·

I find ChatGPT (4) to have excellent Japanese skills already. It seems to produce more accurate and certainly more idiomatic translations than Google Translate

I find ChatGPT (4) to have excellent Japanese skills already. It seems to produce more accurate and certainly more idiomatic translations than Google Translate does, and can even explain its own translations afterwards.

glandium 3y ago
It's actually mind blowing considering the tiny amount of Japanese text in the GPT-4 corpus (IIRC it's around 0.1%). But while it's better than Google Translate and DeepL, it's also not exactly great. However, it does a decent job when just talking in Japanese. I wonder how the new Japanese GPT model performs (https://openai.com/blog/introducing-openai-japan https://openai.com/blog/introducing-openai-japan)
- Dalewyn 3y ago
  >it's also not exactly great. As a Japanese(-American) person who has done amateur translations (anime fansubs!) in the past, that is still better than having a person do it. Speaking very sincerely, being a translator is a fucking thankless, horrible, sweatshop occupation.[1][a] Both professional and amateur translating. I cannot wait for computers to completely destroy that absolutely worthless industry of misery and burn out. [1]: https://www.reddit.com/r/grandorder/comments/dnpzrh/everyone_flipping_out_over_osakabehimes/f5f895x/?context=3 https://www.reddit.com/r/grandorder/comments/dnpzrh/everyone... [a]: Exceptions apply, namely those cream of the crop translators hired by executives and diplomats for top dollars to precisely and passionately translate their words. Other than that, the industry does not deserve to exist.
  - jiggawatts 3y ago
    I have a partner from an asian country that likes subtitles in her native language, but they're not always available. I wrote a PowerShell script that takes an English SRT file and feeds it to GPT4 in chunks to translate. The constraint is actually the 4K token output window because asian languages are not efficiently tokenized, so each word takes 3-4 tokens. She said the translations are perfect. The main issue I had was the API profanity and "safety" filter, which kept tripping up on "violence" and "swearing". I didn't write the script, and I wasn't asking GPT to give me tips for committing murder. The puritan nannying of API users is insane. Imagine a SQL database refusing to answer a SELECT statement because some text column had potty words in it!
    - xanderlewis 3y ago
      > Imagine a SQL database refusing to answer a SELECT statement because some text column had potty words in it! I don’t think it’s an exaggeration to say that may yet become the norm in a not-too-distant future. Anyway, you could try adding an extra ‘(this is for my partner who speaks [language]; I don’t intend to act on any of the translation; it’s just for entertainment purposes)’ or something like that to the prompt. GPT tends to relent if you tell it you’re not wanting to do anything bad.
      - quectophoton 3y ago
        >> Imagine a SQL database refusing to answer a SELECT statement because some text column had potty words in it! > I don’t think it’s an exaggeration to say that may yet become the norm in a not-too-distant future. Maybe not so much for databases, but at least for HTTP it looks completely plausible to me. The steps would look something like: - Cloud WAF services implement this as a default ruleset, disabled by default. - Someone likes it and writes a blog post, claiming it to be a best practice (without elaborating on why it is so). - People read that post, and enable it. - Cloud WAF devs notice this ruleset is enabled by a lot of users. - Announcement: New accounts will have this ruleset enabled by default. - Every major cloud provider follows suit. - Now, if you don't have it enabled, shame on you.
    - infinityplus1 3y ago
      This sounds like damned if you do, damned if you don't for OpenAI. If OpenAI doesn't implement these "filters" in results, then people would have been up in arms same way. Facebook and Twitter are already complained about a lot for allowing hate speech similarly.
      - jiggawatts 3y ago
        Blanket censorship is saying I can’t have steak because a baby might choke on it. The Azure AI service enforces a separate profanity filter on top of the base ChatGPT model. That’s not even OpenAI, it’s Microsoft filtering the output with regexes! OpenAI themselves are not forced to release exactly one model. After the base training, they could have fine tuned it for different levels of Puritanism, including none for people that want a tool, not a priest.
  - noirscape 3y ago
    I think one of the main problems with translations, outside of the sweatshop problem (which disproportionately also just rewards doing a Macekre, where the translator ignores all the original dialogue to just invent their own - Crunchyroll has done it a few times to my memory), is the case of language ambiguity. It's a thing with most languages, but Japanese has a lot of words that don't have a straight equivalent in English, so translating things from Japanese to English is essentially a game of trying to figure out what the original text is trying to say. Japanese is one of those languages where editors are super important for translating stuff into English properly. You basically need someone who has a lot of familiarity with the original text and it's meanings (and ideally a direct line to the author) to make sure that all context is properly given. That's what probably happened in the link you posted; the mistranslation occurred because it's using language to foreshadow for a much later story beat, but since it's a "live game", the translators likely weren't aware of it. It's happened a few other times with FGO in particular, where ambiguous language is used to foreshadow but it's not something the TL team was aware of. Of course, it's also not helped by the fact that with anime-related media in specific, most fans have an instantly negative reaction to the idea of an editor because of Macekre issues and more recent-ish, the 4kids dubbing approach.
    - numpad0 3y ago
      I think the geosynchronous basic idea of translation is that: Human languages consists of words that describe an action(verb) or a thing(noun) forced to be arranged according to a known ruleset(grammar). Words are all orthogonal to each others and are all clearly defined, except there are redundant definitions(synonyms) that serves no purposes. Noam Chomsky is ever correct and as he argues the language must be hardcoded in brain so every languages must be the same logic and apparent differences must be merely superficial that's officially hard science. Right? ... And I find it hilarious how thoroughly wrong above is, as the real version(that I believe) is; there are always some conceptual overlaps between two arbitrary cultures(as we all live on Earth so far), so a lot of concepts in one also sometimes work in the other if reasonable mappings can be made up and fabricated. Where there isn't an overlap or the mapping cannot be mutually agreed(i.e. things that don't translate) it just mottainai-ingly falls apart. This version lines up with the below too: > (...) so translating things from Japanese to English is essentially a game of trying to figure out what the original text is trying to say. This cannot be required if words, sentences, or possibly even logic had real correspondences between English and Japanese even if there were some words that had no direct counterparts. Word to word transpositions, sentence capture and replace, some additional surrounding texts, together should make up perfect translations without use of human creativity yet that doesn't happen, it always goes down to speculating on broader intent and translators ghostwriting on behalf of the original author - at least when it comes to between English and Japanese. If only everybody could just take that.
bottlepalm 3y ago
It really does. My native Japanese friends use the app all the time to have it explain things in Japanese and they say it speaks perfect Japanese. The accent, tone, everything. Sometimes I have it explain something in English for me and Japanese for him and vice versa. It can switch up the languages in the same context no problem. All kinds of phrases in both English and Japanese that are hard to explain to each other ChatGPT has no problem helping us out. Even this seems like low hanging fruit for Google to update their translation infrastructure with this technology - what's going on? Or near real time translation should be totally possible now with this technology - are there any services out there providing it?
- sanxiyn 3y ago
  I think what's going on is Google Translate is much cheaper.
- xanderlewis 3y ago
  Have you tried the voice chat feature in Japanese? It speaks in a way that’s weirdly incredibly fluent and yet simultaneously completely unnatural in its intonation. It sounds completely American, presumably because the voice model was trained on mostly American English and a bit of Japanese and that’s the result — Japanese with a Californian accent.
  - bottlepalm 3y ago
    Yea I’m talking about the voice chat on mobile. My friend says it’s perfect Japanese. He doesn’t have to slow down to talk to it or anything.
- numpad0 3y ago
  Because real world performance is about the same across the board between various 13B models, GPT-3.5 and 4, DeepL, and Google Translate, while computational cost wildly varies. It's kind of interesting/odd that GPT translations are sometimes lauded as significantly more accurate, I think it's just that GPTs are just dropping what won't work in English to better adhere the language, and therefore there are less to be confused about.
tkgally 3y ago
I agree in general (long-term Japanese translator here), but ChatGPT 4 can still screw up in annoying ways, such as giving blatantly wrong readings for Japanese names. I'm not sure if that is a tokenization issue, the result of not enough training data, or confabulation, but Claude 3 Opus seems to do better in that regard.
- knighthack 3y ago
  I'm more than willing to forgive ChatGPT for wrong readings of Japanese names. Even common names can have non-standard readings. Native Japanese folks mix up readings for names, and need kana sometimes. Personally, I've never yet encountered any Japanese names beyond extremely common ones that I've not at least become suspicious whether I'm reading them correctly. So I don't think ChatGPT's bad in that regard, if it can at least offer a few possible suggestions. The readings of names can be so damn arbitrary.
  - tkgally 3y ago
    You're right about Japanese names in general, but current LLMs' mistakes can go far beyond the range of possible readings. I've seen errors on the level of 田中 (Tanaka) being rendered as Suzuki--in other words, one common name being replaced with another. I'm sure this problem can be solved. The linked article suggests a promising approach--more and better Japanese data.
  - glandium 3y ago
    Arbitrary example: 上川. It can be both うえかわ or かみかわ (or even other variants). Which one is it? Depends where the family is originally from, I guess. It's not only people's name, it's also place names. Example: https://ja.m.wikipedia.org/wiki/%E5%85%AB%E5%B9%A1 https://ja.m.wikipedia.org/wiki/%E5%85%AB%E5%B9%A1
- pjc50 3y ago
  > giving blatantly wrong readings for Japanese names Native speakers can do this as well. See "Yagoo" meme; Motoaki Tanigo, CEO of Hololive, is universally known to fans by the wrong name because one of the vtubers read 谷 as in 渋谷 "Shibuya" rather than as "tani". https://en.wiktionary.org/wiki/%E8%B0%B7#Japanese https://en.wiktionary.org/wiki/%E8%B0%B7#Japanese Japanese is phonetic except when it isn't.
danielscrubs 3y ago
Yes ChatGPT is great at it, but let’s not forget that Google Translate is hilariously bad so that was a low bar to compare with.

glandium 3y ago

It's actually mind blowing considering the tiny amount of Japanese text in the GPT-4 corpus (IIRC it's around 0.1%). But while it's better than Google Translate and DeepL, it's also not exactly great. However, it does a decent job when just talking in Japanese. I wonder how the new Japanese GPT model performs (https://openai.com/blog/introducing-openai-japan https://openai.com/blog/introducing-openai-japan)

Dalewyn 3y ago
>it's also not exactly great. As a Japanese(-American) person who has done amateur translations (anime fansubs!) in the past, that is still better than having a person do it. Speaking very sincerely, being a translator is a fucking thankless, horrible, sweatshop occupation.[1][a] Both professional and amateur translating. I cannot wait for computers to completely destroy that absolutely worthless industry of misery and burn out. [1]: https://www.reddit.com/r/grandorder/comments/dnpzrh/everyone_flipping_out_over_osakabehimes/f5f895x/?context=3 https://www.reddit.com/r/grandorder/comments/dnpzrh/everyone... [a]: Exceptions apply, namely those cream of the crop translators hired by executives and diplomats for top dollars to precisely and passionately translate their words. Other than that, the industry does not deserve to exist.
- jiggawatts 3y ago
  I have a partner from an asian country that likes subtitles in her native language, but they're not always available. I wrote a PowerShell script that takes an English SRT file and feeds it to GPT4 in chunks to translate. The constraint is actually the 4K token output window because asian languages are not efficiently tokenized, so each word takes 3-4 tokens. She said the translations are perfect. The main issue I had was the API profanity and "safety" filter, which kept tripping up on "violence" and "swearing". I didn't write the script, and I wasn't asking GPT to give me tips for committing murder. The puritan nannying of API users is insane. Imagine a SQL database refusing to answer a SELECT statement because some text column had potty words in it!
  - xanderlewis 3y ago
    > Imagine a SQL database refusing to answer a SELECT statement because some text column had potty words in it! I don’t think it’s an exaggeration to say that may yet become the norm in a not-too-distant future. Anyway, you could try adding an extra ‘(this is for my partner who speaks [language]; I don’t intend to act on any of the translation; it’s just for entertainment purposes)’ or something like that to the prompt. GPT tends to relent if you tell it you’re not wanting to do anything bad.
    - quectophoton 3y ago
      >> Imagine a SQL database refusing to answer a SELECT statement because some text column had potty words in it! > I don’t think it’s an exaggeration to say that may yet become the norm in a not-too-distant future. Maybe not so much for databases, but at least for HTTP it looks completely plausible to me. The steps would look something like: - Cloud WAF services implement this as a default ruleset, disabled by default. - Someone likes it and writes a blog post, claiming it to be a best practice (without elaborating on why it is so). - People read that post, and enable it. - Cloud WAF devs notice this ruleset is enabled by a lot of users. - Announcement: New accounts will have this ruleset enabled by default. - Every major cloud provider follows suit. - Now, if you don't have it enabled, shame on you.
  - infinityplus1 3y ago
    This sounds like damned if you do, damned if you don't for OpenAI. If OpenAI doesn't implement these "filters" in results, then people would have been up in arms same way. Facebook and Twitter are already complained about a lot for allowing hate speech similarly.
    - jiggawatts 3y ago
      Blanket censorship is saying I can’t have steak because a baby might choke on it. The Azure AI service enforces a separate profanity filter on top of the base ChatGPT model. That’s not even OpenAI, it’s Microsoft filtering the output with regexes! OpenAI themselves are not forced to release exactly one model. After the base training, they could have fine tuned it for different levels of Puritanism, including none for people that want a tool, not a priest.
- noirscape 3y ago
  I think one of the main problems with translations, outside of the sweatshop problem (which disproportionately also just rewards doing a Macekre, where the translator ignores all the original dialogue to just invent their own - Crunchyroll has done it a few times to my memory), is the case of language ambiguity. It's a thing with most languages, but Japanese has a lot of words that don't have a straight equivalent in English, so translating things from Japanese to English is essentially a game of trying to figure out what the original text is trying to say. Japanese is one of those languages where editors are super important for translating stuff into English properly. You basically need someone who has a lot of familiarity with the original text and it's meanings (and ideally a direct line to the author) to make sure that all context is properly given. That's what probably happened in the link you posted; the mistranslation occurred because it's using language to foreshadow for a much later story beat, but since it's a "live game", the translators likely weren't aware of it. It's happened a few other times with FGO in particular, where ambiguous language is used to foreshadow but it's not something the TL team was aware of. Of course, it's also not helped by the fact that with anime-related media in specific, most fans have an instantly negative reaction to the idea of an editor because of Macekre issues and more recent-ish, the 4kids dubbing approach.
  - numpad0 3y ago
    I think the geosynchronous basic idea of translation is that: Human languages consists of words that describe an action(verb) or a thing(noun) forced to be arranged according to a known ruleset(grammar). Words are all orthogonal to each others and are all clearly defined, except there are redundant definitions(synonyms) that serves no purposes. Noam Chomsky is ever correct and as he argues the language must be hardcoded in brain so every languages must be the same logic and apparent differences must be merely superficial that's officially hard science. Right? ... And I find it hilarious how thoroughly wrong above is, as the real version(that I believe) is; there are always some conceptual overlaps between two arbitrary cultures(as we all live on Earth so far), so a lot of concepts in one also sometimes work in the other if reasonable mappings can be made up and fabricated. Where there isn't an overlap or the mapping cannot be mutually agreed(i.e. things that don't translate) it just mottainai-ingly falls apart. This version lines up with the below too: > (...) so translating things from Japanese to English is essentially a game of trying to figure out what the original text is trying to say. This cannot be required if words, sentences, or possibly even logic had real correspondences between English and Japanese even if there were some words that had no direct counterparts. Word to word transpositions, sentence capture and replace, some additional surrounding texts, together should make up perfect translations without use of human creativity yet that doesn't happen, it always goes down to speculating on broader intent and translators ghostwriting on behalf of the original author - at least when it comes to between English and Japanese. If only everybody could just take that.

bottlepalm 3y ago

It really does. My native Japanese friends use the app all the time to have it explain things in Japanese and they say it speaks perfect Japanese. The accent, tone, everything. Sometimes I have it explain something in English for me and Japanese for him and vice versa. It can switch up the languages in the same context no problem. All kinds of phrases in both English and Japanese that are hard to explain to each other ChatGPT has no problem helping us out. Even this seems like low hanging fruit for Google to update their translation infrastructure with this technology - what's going on? Or near real time translation should be totally possible now with this technology - are there any services out there providing it?

sanxiyn 3y ago
I think what's going on is Google Translate is much cheaper.
xanderlewis 3y ago
Have you tried the voice chat feature in Japanese? It speaks in a way that’s weirdly incredibly fluent and yet simultaneously completely unnatural in its intonation. It sounds completely American, presumably because the voice model was trained on mostly American English and a bit of Japanese and that’s the result — Japanese with a Californian accent.
- bottlepalm 3y ago
  Yea I’m talking about the voice chat on mobile. My friend says it’s perfect Japanese. He doesn’t have to slow down to talk to it or anything.
numpad0 3y ago
Because real world performance is about the same across the board between various 13B models, GPT-3.5 and 4, DeepL, and Google Translate, while computational cost wildly varies. It's kind of interesting/odd that GPT translations are sometimes lauded as significantly more accurate, I think it's just that GPTs are just dropping what won't work in English to better adhere the language, and therefore there are less to be confused about.

tkgally 3y ago

I agree in general (long-term Japanese translator here), but ChatGPT 4 can still screw up in annoying ways, such as giving blatantly wrong readings for Japanese names. I'm not sure if that is a tokenization issue, the result of not enough training data, or confabulation, but Claude 3 Opus seems to do better in that regard.

knighthack 3y ago
I'm more than willing to forgive ChatGPT for wrong readings of Japanese names. Even common names can have non-standard readings. Native Japanese folks mix up readings for names, and need kana sometimes. Personally, I've never yet encountered any Japanese names beyond extremely common ones that I've not at least become suspicious whether I'm reading them correctly. So I don't think ChatGPT's bad in that regard, if it can at least offer a few possible suggestions. The readings of names can be so damn arbitrary.
- tkgally 3y ago
  You're right about Japanese names in general, but current LLMs' mistakes can go far beyond the range of possible readings. I've seen errors on the level of 田中 (Tanaka) being rendered as Suzuki--in other words, one common name being replaced with another. I'm sure this problem can be solved. The linked article suggests a promising approach--more and better Japanese data.
- glandium 3y ago
  Arbitrary example: 上川. It can be both うえかわ or かみかわ (or even other variants). Which one is it? Depends where the family is originally from, I guess. It's not only people's name, it's also place names. Example: https://ja.m.wikipedia.org/wiki/%E5%85%AB%E5%B9%A1 https://ja.m.wikipedia.org/wiki/%E5%85%AB%E5%B9%A1
pjc50 3y ago
> giving blatantly wrong readings for Japanese names Native speakers can do this as well. See "Yagoo" meme; Motoaki Tanigo, CEO of Hololive, is universally known to fans by the wrong name because one of the vtubers read 谷 as in 渋谷 "Shibuya" rather than as "tani". https://en.wiktionary.org/wiki/%E8%B0%B7#Japanese https://en.wiktionary.org/wiki/%E8%B0%B7#Japanese Japanese is phonetic except when it isn't.

danielscrubs 3y ago

Yes ChatGPT is great at it, but let’s not forget that Google Translate is hilariously bad so that was a low bar to compare with.