10 ms·

It's not under the radar because that's sadly not the case, although it looks like it should work better on the surface. Neural machine translation is just more

by luxpir 3y ago

It's not under the radar because that's sadly not the case, although it looks like it should work better on the surface. Neural machine translation is just more consistent, doesn't hallucinate and can be easily and cheaply trained over time. They have some benefits, but as a lot of LLM research has found, they are not production ready. Yet.

deleted 3y ago
[deleted]
famouswaffles 3y ago
>It's not under the radar because that's sadly not the case It is the case. Haven't seen anyone who uses both who thinks otherwise. also https://arxiv.org/abs/2301.13294 https://arxiv.org/abs/2301.13294. This benchmarks 3.5 which is quite a bit worse than 4 as it is against Google, NLLB, Deepl here https://arxiv.org/pdf/2304.02210.pdf https://arxiv.org/pdf/2304.02210.pdf, GPT wins overwhelmingly with human evaluations. Seems like the typically evaluation models aren't really cutting it anymore especially BLEU >Neural machine translation is just more consistent, doesn't hallucinate It's not more consistent. The 2nd bit is just wrong lol. One of the biggest complaints of Deepl is the tendency to make stuff up to make translations seem more natural. Summarization and Translation are the tasks GPT models hallucinate the least.
- cj 3y ago
  I wrote a much longer reply but looks like you deleted the downvoted comment and reposted it. > It's not more consistent. I think the parent was probably saying for a given input, Google MT provides the same output. What is the value of temperature/variability in a LLM powered MT model? I’d assume given the same inputs, you should only be given the best output.
  - famouswaffles 3y ago
    >I think the parent was probably saying for a given input, Google MT provides the same output. I don't care about being given the exact same output (you're not getting deterministic translations from people either). I care about quality translations. Variability for GPT style translations is much more about word choice and style than wrong or extremely different translations. and if i really wanted to, i could guide both (word choice, style) either with instructions or examples.
    - naniwaduni 3y ago
      Good news! Per your second link, GPT-4 is a stunning improvement up to "borderline passes quality control"!
- yorwba 3y ago
  In my experience, ChatGPT tends to produce more fluent output, but is less likely to closely follow the input. For some high-resource language pairs, complete mistranslations are rare, but for other languages, not so much. Of the ones I can evaluate, Burmese is particularly error-prone: ChatGPT translates ငါ့ကို ဒုတိယ အခွင့်ရေး တစ်ခု ပေးခဲ့တယ်။ as "I have received a second warning.", which is incorrect. အခွင့် https://en.wiktionary.org/wiki/%E1%80%A1%E1%80%81%E1%80%BD%E1%80%84%E1%80%B7%E1%80%BA https://en.wiktionary.org/wiki/%E1%80%A1%E1%80%81%E1%80%BD%E... does not mean "warning", even though that is a likely completion of "I have received a second " in English. Google Translate gives me "Gave me a second chance.", which closely matches the Burmese sentence down to dropping the subject (common in Burmese, rare in English) which makes the translation sound weird. So any claim that ChatGPT is better/worse at translating really needs to specify the languages involved and what your goal for the translation is. (E.g. the benchmark paper you link seems to focus on the ability to steer the translation by providing additional context.)
  - famouswaffles 3y ago
    I’m not making a claim for chatGPT so much as I am making a claim for GPT style models. It’s not really a question of high resource vs low resource languages so much as what languages ended up in the training corpus. 1. There’s a lot of transfer learning going on with predict the next token LLMs. A model trained on 500B tokens of English and 50b tokens of French will speak French far better than if it was trained on only 50b tokens of French. 2. You don’t need parallel corpora for every single pair you want to translate to. This means that GPT LLMs only need single text data for the vast majority of languages. Training most NMT models you would need Burmese/English parallel data. Both of the above combine to mean that not only is quality demonstrably better, amount of data needed is lower too. GPT's burmese isn't worse because it's low resource. It's because open ai made no specific attempt to include burmese text. They're not even trying. GPT-3 was 93% English with the 2nd biggest holder less than 2%
    - YeGoblynQueenne 3y ago
      >> 2. You don’t need parallel corpora for every single pair you want to translate to. Neural Machine Translation models don't need those either. That's been the case at least since 2016. That was one of the big claims by Google, that Google Translate had created its own "interlingua" thanks to which it could translate between languages for which it doesn't have parallel corpora: Following several months of testing, the researchers behind the AI have seen it be able to blindly translate languages even if it's never studied one of the languages involved in the translation. "An example of this would be translations between Korean and Japanese where Korean⇄Japanese examples were not shown to the system," the Mike Schuster, from Google Brain wrote in a blogpost. https://www.wired.co.uk/article/google-ai-language-create https://www.wired.co.uk/article/google-ai-language-create It was obvious already back then that this "interlingua" was no other than English, e.g., from the same article: The team said the system was able to make "reasonable" translations of the languages it had not been taught to translate. In one instance, a research paper published alongside the blog, says the AI was taught Portuguese→English and English→Spanish. It was then able to make translations between Portuguese→Spanish. But, hey, hype.
- luxpir 3y ago
  Your reply is not passing my sniff test. Your hype and bias are showing. It may present more fluent text, but if it doesn't know it's strayed from the source text and you can't tell either (because you don't understand source language) then you'll end up with error-laden pseudo translations. At least with NMT you know the errors are consistent. I don't know who you know who thinks GPT is ahead, but nobody in the very well funded translation industry has a GPT powered translation engine for the key reason that it's not ready for production. For a human to post edit MT, we're mainly talking fixing broken vocab. You'd never present raw MT to a client. It needs editing. Heavily. To think LLM translation doesn't need editing is either coming from someone not in the industry, or blinded by hype. And the kind of editing issues are more insidious, like those found in voice dictated texts. Homophones aren't flagged by QA software because they are real words. Just like LLMs make real sentences, except when they don't, but good luck detecting that, and editing out the additional meaning it has decided to inject. Have you tried to run the GPT-4 API on a segmented xliff at all? If the segmentation is bad, and full of tags, GPT4 breaks completely. It tries to close sentences that run across segments, it can't handle tags in-line (the chatgpt interface can, but you can't use that at scale). It can do some impressive work, don't get me wrong, but I'm not sure how hands-on you've really been if you think it's a solved problem. Production translation is a non-trivial output. The entire industry hasn't released an LLM solution yet for translation (excepting the rewording mini features). What makes you think you know more than those on the ground? Or have you developed something that's still in stealth? EDIT: Oh wow, all of your 107 submissions to HN from the 6 months your account has existed, have been about AI and LLMs. I guess I got the hype part right. As for industry knowledge the jury is still out, but this could well be the classic HN "I understand tech so obviously I understand everything" play. Keep us posted!
  - naniwaduni 3y ago
    > It may present more fluent text, but if it doesn't know it's strayed from the source text and you can't tell either (because you don't understand source language) then you'll end up with error-laden pseudo translations. At least with NMT you know the errors are consistent. To be fair, this is an infamous failure mode of neural mtl too, and a big part of what makes the discourse around GPT so ... evocative of the discourse in 2017.
benbreen 3y ago
I've tried using ChatGPT to translate Latin texts from the Renaissance. I know enough Latin (intermediate, but can figure it out with a dictionary) to check it, and it was very, very impressive. What blew me away was not just the fluency of the translation but the fact that I could drop in highly imperfect OCR'd text from Google Books, and it didn't have any trouble making sense of garbled passages. This ability makes it a really distinct advance on Google Translate and the like, at least for my purposes. Also, another underrated feature: I asked it to summarize each page in a single sentence, while also picking out the passages most relevant to my research question. It did a great job.
- luxpir 3y ago
  That is a truly exceptional use case, and I'm impressed too. I should have specified above that I'm referring to the practicalities of professional translation workflows as they currently exist for things like high volume flows in dozens of formats, and translation memory leveraging etc.
mcswell 3y ago
"Neural machine translation... doesn't hallucinate..." I've looked at neural MT output from Arabic and Russian, and there was lots of hallucination.

by luxpir 3y ago

deleted 3y ago

[deleted]

famouswaffles 3y ago

>It's not under the radar because that's sadly not the case It is the case. Haven't seen anyone who uses both who thinks otherwise. also https://arxiv.org/abs/2301.13294 https://arxiv.org/abs/2301.13294. This benchmarks 3.5 which is quite a bit worse than 4 as it is against Google, NLLB, Deepl here https://arxiv.org/pdf/2304.02210.pdf https://arxiv.org/pdf/2304.02210.pdf, GPT wins overwhelmingly with human evaluations. Seems like the typically evaluation models aren't really cutting it anymore especially BLEU >Neural machine translation is just more consistent, doesn't hallucinate It's not more consistent. The 2nd bit is just wrong lol. One of the biggest complaints of Deepl is the tendency to make stuff up to make translations seem more natural. Summarization and Translation are the tasks GPT models hallucinate the least.

cj 3y ago
I wrote a much longer reply but looks like you deleted the downvoted comment and reposted it. > It's not more consistent. I think the parent was probably saying for a given input, Google MT provides the same output. What is the value of temperature/variability in a LLM powered MT model? I’d assume given the same inputs, you should only be given the best output.
- famouswaffles 3y ago
  >I think the parent was probably saying for a given input, Google MT provides the same output. I don't care about being given the exact same output (you're not getting deterministic translations from people either). I care about quality translations. Variability for GPT style translations is much more about word choice and style than wrong or extremely different translations. and if i really wanted to, i could guide both (word choice, style) either with instructions or examples.
  - naniwaduni 3y ago
    Good news! Per your second link, GPT-4 is a stunning improvement up to "borderline passes quality control"!
yorwba 3y ago
In my experience, ChatGPT tends to produce more fluent output, but is less likely to closely follow the input. For some high-resource language pairs, complete mistranslations are rare, but for other languages, not so much. Of the ones I can evaluate, Burmese is particularly error-prone: ChatGPT translates ငါ့ကို ဒုတိယ အခွင့်ရေး တစ်ခု ပေးခဲ့တယ်။ as "I have received a second warning.", which is incorrect. အခွင့် https://en.wiktionary.org/wiki/%E1%80%A1%E1%80%81%E1%80%BD%E1%80%84%E1%80%B7%E1%80%BA https://en.wiktionary.org/wiki/%E1%80%A1%E1%80%81%E1%80%BD%E... does not mean "warning", even though that is a likely completion of "I have received a second " in English. Google Translate gives me "Gave me a second chance.", which closely matches the Burmese sentence down to dropping the subject (common in Burmese, rare in English) which makes the translation sound weird. So any claim that ChatGPT is better/worse at translating really needs to specify the languages involved and what your goal for the translation is. (E.g. the benchmark paper you link seems to focus on the ability to steer the translation by providing additional context.)
- famouswaffles 3y ago
  I’m not making a claim for chatGPT so much as I am making a claim for GPT style models. It’s not really a question of high resource vs low resource languages so much as what languages ended up in the training corpus. 1. There’s a lot of transfer learning going on with predict the next token LLMs. A model trained on 500B tokens of English and 50b tokens of French will speak French far better than if it was trained on only 50b tokens of French. 2. You don’t need parallel corpora for every single pair you want to translate to. This means that GPT LLMs only need single text data for the vast majority of languages. Training most NMT models you would need Burmese/English parallel data. Both of the above combine to mean that not only is quality demonstrably better, amount of data needed is lower too. GPT's burmese isn't worse because it's low resource. It's because open ai made no specific attempt to include burmese text. They're not even trying. GPT-3 was 93% English with the 2nd biggest holder less than 2%
  - YeGoblynQueenne 3y ago
    >> 2. You don’t need parallel corpora for every single pair you want to translate to. Neural Machine Translation models don't need those either. That's been the case at least since 2016. That was one of the big claims by Google, that Google Translate had created its own "interlingua" thanks to which it could translate between languages for which it doesn't have parallel corpora: Following several months of testing, the researchers behind the AI have seen it be able to blindly translate languages even if it's never studied one of the languages involved in the translation. "An example of this would be translations between Korean and Japanese where Korean⇄Japanese examples were not shown to the system," the Mike Schuster, from Google Brain wrote in a blogpost. https://www.wired.co.uk/article/google-ai-language-create https://www.wired.co.uk/article/google-ai-language-create It was obvious already back then that this "interlingua" was no other than English, e.g., from the same article: The team said the system was able to make "reasonable" translations of the languages it had not been taught to translate. In one instance, a research paper published alongside the blog, says the AI was taught Portuguese→English and English→Spanish. It was then able to make translations between Portuguese→Spanish. But, hey, hype.
luxpir 3y ago
Your reply is not passing my sniff test. Your hype and bias are showing. It may present more fluent text, but if it doesn't know it's strayed from the source text and you can't tell either (because you don't understand source language) then you'll end up with error-laden pseudo translations. At least with NMT you know the errors are consistent. I don't know who you know who thinks GPT is ahead, but nobody in the very well funded translation industry has a GPT powered translation engine for the key reason that it's not ready for production. For a human to post edit MT, we're mainly talking fixing broken vocab. You'd never present raw MT to a client. It needs editing. Heavily. To think LLM translation doesn't need editing is either coming from someone not in the industry, or blinded by hype. And the kind of editing issues are more insidious, like those found in voice dictated texts. Homophones aren't flagged by QA software because they are real words. Just like LLMs make real sentences, except when they don't, but good luck detecting that, and editing out the additional meaning it has decided to inject. Have you tried to run the GPT-4 API on a segmented xliff at all? If the segmentation is bad, and full of tags, GPT4 breaks completely. It tries to close sentences that run across segments, it can't handle tags in-line (the chatgpt interface can, but you can't use that at scale). It can do some impressive work, don't get me wrong, but I'm not sure how hands-on you've really been if you think it's a solved problem. Production translation is a non-trivial output. The entire industry hasn't released an LLM solution yet for translation (excepting the rewording mini features). What makes you think you know more than those on the ground? Or have you developed something that's still in stealth? EDIT: Oh wow, all of your 107 submissions to HN from the 6 months your account has existed, have been about AI and LLMs. I guess I got the hype part right. As for industry knowledge the jury is still out, but this could well be the classic HN "I understand tech so obviously I understand everything" play. Keep us posted!
- naniwaduni 3y ago
  > It may present more fluent text, but if it doesn't know it's strayed from the source text and you can't tell either (because you don't understand source language) then you'll end up with error-laden pseudo translations. At least with NMT you know the errors are consistent. To be fair, this is an infamous failure mode of neural mtl too, and a big part of what makes the discourse around GPT so ... evocative of the discourse in 2017.

benbreen 3y ago

I've tried using ChatGPT to translate Latin texts from the Renaissance. I know enough Latin (intermediate, but can figure it out with a dictionary) to check it, and it was very, very impressive. What blew me away was not just the fluency of the translation but the fact that I could drop in highly imperfect OCR'd text from Google Books, and it didn't have any trouble making sense of garbled passages. This ability makes it a really distinct advance on Google Translate and the like, at least for my purposes. Also, another underrated feature: I asked it to summarize each page in a single sentence, while also picking out the passages most relevant to my research question. It did a great job.

luxpir 3y ago
That is a truly exceptional use case, and I'm impressed too. I should have specified above that I'm referring to the practicalities of professional translation workflows as they currently exist for things like high volume flows in dozens of formats, and translation memory leveraging etc.

mcswell 3y ago

"Neural machine translation... doesn't hallucinate..." I've looked at neural MT output from Arabic and Russian, and there was lots of hallucination.