8 ms·

It's weirdly flown under the radar but GPT style models as translators are a lot better than state of the art Machine Translators (Deepl, NLLB, Google etc). Li

It's weirdly flown under the radar but GPT style models as translators are a lot better than state of the art Machine Translators (Deepl, NLLB, Google etc). Like above, you can already see the difference with close language pairs where Google etc are already very good. For pairs like English and Japanese etc, google et al will happily devolve into half gibberish so the difference is even more stark. I did a number of examples a couple months back with English/Chinese before 4 was released. Even then you can see it's a lot better and 4 is as usual a lot better than 3.5. https://github.com/ogkalu2/Human-parity-on-machine-translations https://github.com/ogkalu2/Human-parity-on-machine-translati...

luxpir 3y ago
It's not under the radar because that's sadly not the case, although it looks like it should work better on the surface. Neural machine translation is just more consistent, doesn't hallucinate and can be easily and cheaply trained over time. They have some benefits, but as a lot of LLM research has found, they are not production ready. Yet.
- deleted 3y ago
  [deleted]
- famouswaffles 3y ago
  >It's not under the radar because that's sadly not the case It is the case. Haven't seen anyone who uses both who thinks otherwise. also https://arxiv.org/abs/2301.13294 https://arxiv.org/abs/2301.13294. This benchmarks 3.5 which is quite a bit worse than 4 as it is against Google, NLLB, Deepl here https://arxiv.org/pdf/2304.02210.pdf https://arxiv.org/pdf/2304.02210.pdf, GPT wins overwhelmingly with human evaluations. Seems like the typically evaluation models aren't really cutting it anymore especially BLEU >Neural machine translation is just more consistent, doesn't hallucinate It's not more consistent. The 2nd bit is just wrong lol. One of the biggest complaints of Deepl is the tendency to make stuff up to make translations seem more natural. Summarization and Translation are the tasks GPT models hallucinate the least.
  - cj 3y ago
    I wrote a much longer reply but looks like you deleted the downvoted comment and reposted it. > It's not more consistent. I think the parent was probably saying for a given input, Google MT provides the same output. What is the value of temperature/variability in a LLM powered MT model? I’d assume given the same inputs, you should only be given the best output.
    - famouswaffles 3y ago
      >I think the parent was probably saying for a given input, Google MT provides the same output. I don't care about being given the exact same output (you're not getting deterministic translations from people either). I care about quality translations. Variability for GPT style translations is much more about word choice and style than wrong or extremely different translations. and if i really wanted to, i could guide both (word choice, style) either with instructions or examples.
      - naniwaduni 3y ago
        Good news! Per your second link, GPT-4 is a stunning improvement up to "borderline passes quality control"!
  - yorwba 3y ago
    In my experience, ChatGPT tends to produce more fluent output, but is less likely to closely follow the input. For some high-resource language pairs, complete mistranslations are rare, but for other languages, not so much. Of the ones I can evaluate, Burmese is particularly error-prone: ChatGPT translates ငါ့ကို ဒုတိယ အခွင့်ရေး တစ်ခု ပေးခဲ့တယ်။ as "I have received a second warning.", which is incorrect. အခွင့် https://en.wiktionary.org/wiki/%E1%80%A1%E1%80%81%E1%80%BD%E1%80%84%E1%80%B7%E1%80%BA https://en.wiktionary.org/wiki/%E1%80%A1%E1%80%81%E1%80%BD%E... does not mean "warning", even though that is a likely completion of "I have received a second " in English. Google Translate gives me "Gave me a second chance.", which closely matches the Burmese sentence down to dropping the subject (common in Burmese, rare in English) which makes the translation sound weird. So any claim that ChatGPT is better/worse at translating really needs to specify the languages involved and what your goal for the translation is. (E.g. the benchmark paper you link seems to focus on the ability to steer the translation by providing additional context.)
- benbreen 3y ago
  I've tried using ChatGPT to translate Latin texts from the Renaissance. I know enough Latin (intermediate, but can figure it out with a dictionary) to check it, and it was very, very impressive. What blew me away was not just the fluency of the translation but the fact that I could drop in highly imperfect OCR'd text from Google Books, and it didn't have any trouble making sense of garbled passages. This ability makes it a really distinct advance on Google Translate and the like, at least for my purposes. Also, another underrated feature: I asked it to summarize each page in a single sentence, while also picking out the passages most relevant to my research question. It did a great job.
  - luxpir 3y ago
    That is a truly exceptional use case, and I'm impressed too. I should have specified above that I'm referring to the practicalities of professional translation workflows as they currently exist for things like high volume flows in dozens of formats, and translation memory leveraging etc.
- mcswell 3y ago
  "Neural machine translation... doesn't hallucinate..." I've looked at neural MT output from Arabic and Russian, and there was lots of hallucination.
jug 3y ago
Yeah, GPT-4 is so good Iceland is using it for language preservation of Icelandic as it translates text so well, letting them enrich the language with new works. Like you, I’ve definitely often though “Come ON!” about the increasingly archaic Google Translate in light of DeepL etc. It has really stagnated over the years.

luxpir 3y ago

It's not under the radar because that's sadly not the case, although it looks like it should work better on the surface. Neural machine translation is just more consistent, doesn't hallucinate and can be easily and cheaply trained over time. They have some benefits, but as a lot of LLM research has found, they are not production ready. Yet.

deleted 3y ago
[deleted]
famouswaffles 3y ago
>It's not under the radar because that's sadly not the case It is the case. Haven't seen anyone who uses both who thinks otherwise. also https://arxiv.org/abs/2301.13294 https://arxiv.org/abs/2301.13294. This benchmarks 3.5 which is quite a bit worse than 4 as it is against Google, NLLB, Deepl here https://arxiv.org/pdf/2304.02210.pdf https://arxiv.org/pdf/2304.02210.pdf, GPT wins overwhelmingly with human evaluations. Seems like the typically evaluation models aren't really cutting it anymore especially BLEU >Neural machine translation is just more consistent, doesn't hallucinate It's not more consistent. The 2nd bit is just wrong lol. One of the biggest complaints of Deepl is the tendency to make stuff up to make translations seem more natural. Summarization and Translation are the tasks GPT models hallucinate the least.
- cj 3y ago
  I wrote a much longer reply but looks like you deleted the downvoted comment and reposted it. > It's not more consistent. I think the parent was probably saying for a given input, Google MT provides the same output. What is the value of temperature/variability in a LLM powered MT model? I’d assume given the same inputs, you should only be given the best output.
  - famouswaffles 3y ago
    >I think the parent was probably saying for a given input, Google MT provides the same output. I don't care about being given the exact same output (you're not getting deterministic translations from people either). I care about quality translations. Variability for GPT style translations is much more about word choice and style than wrong or extremely different translations. and if i really wanted to, i could guide both (word choice, style) either with instructions or examples.
    - naniwaduni 3y ago
      Good news! Per your second link, GPT-4 is a stunning improvement up to "borderline passes quality control"!
- yorwba 3y ago
  In my experience, ChatGPT tends to produce more fluent output, but is less likely to closely follow the input. For some high-resource language pairs, complete mistranslations are rare, but for other languages, not so much. Of the ones I can evaluate, Burmese is particularly error-prone: ChatGPT translates ငါ့ကို ဒုတိယ အခွင့်ရေး တစ်ခု ပေးခဲ့တယ်။ as "I have received a second warning.", which is incorrect. အခွင့် https://en.wiktionary.org/wiki/%E1%80%A1%E1%80%81%E1%80%BD%E1%80%84%E1%80%B7%E1%80%BA https://en.wiktionary.org/wiki/%E1%80%A1%E1%80%81%E1%80%BD%E... does not mean "warning", even though that is a likely completion of "I have received a second " in English. Google Translate gives me "Gave me a second chance.", which closely matches the Burmese sentence down to dropping the subject (common in Burmese, rare in English) which makes the translation sound weird. So any claim that ChatGPT is better/worse at translating really needs to specify the languages involved and what your goal for the translation is. (E.g. the benchmark paper you link seems to focus on the ability to steer the translation by providing additional context.)
benbreen 3y ago
I've tried using ChatGPT to translate Latin texts from the Renaissance. I know enough Latin (intermediate, but can figure it out with a dictionary) to check it, and it was very, very impressive. What blew me away was not just the fluency of the translation but the fact that I could drop in highly imperfect OCR'd text from Google Books, and it didn't have any trouble making sense of garbled passages. This ability makes it a really distinct advance on Google Translate and the like, at least for my purposes. Also, another underrated feature: I asked it to summarize each page in a single sentence, while also picking out the passages most relevant to my research question. It did a great job.
- luxpir 3y ago
  That is a truly exceptional use case, and I'm impressed too. I should have specified above that I'm referring to the practicalities of professional translation workflows as they currently exist for things like high volume flows in dozens of formats, and translation memory leveraging etc.
mcswell 3y ago
"Neural machine translation... doesn't hallucinate..." I've looked at neural MT output from Arabic and Russian, and there was lots of hallucination.

jug 3y ago

Yeah, GPT-4 is so good Iceland is using it for language preservation of Icelandic as it translates text so well, letting them enrich the language with new works. Like you, I’ve definitely often though “Come ON!” about the increasingly archaic Google Translate in light of DeepL etc. It has really stagnated over the years.