4 ms·
The transformer architecture was originally developed for translation tasks, but massive, overfit generative models are awful at translation. (Seriously, part-o
by wizzwizz4 1y ago
The transformer architecture was originally developed for translation tasks, but massive, overfit generative models are awful at translation. (Seriously, part-of-speech classification + dictionary lookup + grammar mapping – an incredibly simplistic system with performance measurable in chapters per second – does a better job, and you even get confidence intervals.) If you want a translation tool, use something like Project Bergamot (https://browser.mt/ https://browser.mt/), not generative AI.
> Unfortunately, so much of the humanities in high school is treated as an exercise rather than real discovery.
I, too, find this exceptionally annoying.
- bonoboTP 1y agoThat's an absurd claim. Show me the translation benchmark where you saw such a result.
- wizzwizz4 1y agoDoes "every time I've challenged people about this on Hacker News" count as a benchmark? See e.g. https://news.ycombinator.com/item?id=35531558 https://news.ycombinator.com/item?id=35531558.
- fc417fc802 1y agoNo, it doesn't. I don't find the linked thread remotely convincing regarding your (frankly preposterous) claim that preexisting non-deep-learning solutions are better at translation in the general case. Don't get me wrong. I'm sure that once fine tuned by a human for a specific language pair that such systems are better at performing literal translation. But the value proposition of deep learning here is that you don't need a large team of experts to laboriously train a given language pair, that the entire training process is largely unsupervised, and that the translations aren't hopelessly literal. The ML algorithms can pick up on idioms given a sufficiently large dataset.
- wizzwizz4 1y agoFound the other good post in that thread: https://news.ycombinator.com/item?id=35532941 https://news.ycombinator.com/item?id=35532941. Google Translate and DeepL were capable of mostly-accurate Japanese translation two (or even 5) years ago (as seen in my previous link). Multiple people ran this text through GPT-4, and none of its translations were any good, as assessed by this Japanese-speaker. There's a reason we needed the Rosetta stone to give as an in for translating Egyptian hieroglyphics. You can't just throw Big Data at the problem and expect it to give accurate results. Dedicated translation setups can get good results, but ChatGPT's ilk doesn't. Convincing translations are not the same thing as accurate translations. RLHF optimises for convincing, not accurate.
- fc417fc802 1y agoYou originally made a broad claim about the technology as a class though. It's not particularly surprising that chatgpt (from 2 years ago no less) isn't state of the art at translations given that it isn't optimized for that (at least that I'm aware). The same approaches that are used to construct LLMs can be applied to construct a model intended for machine translation. That is where the value proposition lies.
- wizzwizz4 1y agoI'm not sure whether you're missing the point by a mile, or whether I am. The transformer architecture, later modified to create GPT models, was originally designed for translation. The modifications to make it do predictive text in a chatbot style make it much, much worse at translating: one of the biggest issues is that generative systems fail silently. Using the tech appropriately gives you things like Project Bergamot: https://browser.mt/ https://browser.mt/.
- dr_dshiv 1y agoI get your point about the surprising news that gpt4 does so poorly at translation. I didn’t know that! However, I think the idea is that LLM technologies have improved considerably since then. Do you still feel that Claude or ChatGPT perform worse than DeepL? It would be really nice to have an objective benchmark for comparison
- dr_dshiv 1y agohttps://lokalise.com/blog/what-is-the-best-llm-for-translation/ https://lokalise.com/blog/what-is-the-best-llm-for-translati... Claims Sonnet 3.5 wins