8 ms·
Claude 3 beats Google Translate
- itake 2y ago[flagged]
- Tistron 2y agoChatGPT 3 was already far beyond google translate in my attempts, mostly using it between English, Danish and Swedish. Even being able to ask it to translate song lyrics while retaining rhyming structure was sometimes not too bad :)
- moffkalast 2y agoCan confirm for Slovenian too, Google Translate is so bad it doesn't even compare.
- SamBam 2y agoAlso being able to ask it to use the formal vs. informal is very helpful for romance languages. I could never get Google translate to do that (although sometimes adding ", sir" to the end of a sentence would trick it into using the formal).
- hnfong 2y ago[Nit] It's either GPT-3 or ChatGPT (which was also called GPT-3.5-turbo) I do agree GPT-3 probably did perform better than Google Translate though.
- sunaookami 2y agoDeepL already beat Google Translate years ago.
- frabcus 2y agoIn tests I did with linguists a year ago, GPT-3.5 was better than DeepL also. Google Translate largely caught up with DeepL a while ago! Specifically, LLMs have a context window larger than a sentence. So can eg infer gender throughout documents. Neither DeepL or Google Translate did that.
- casey2 2y agoDeepL is also very bad, it ties with google in plenty of very basic sentences. Try out: 私は毎週本を読む。友達は漫画だ。 Meta's gets it despite a message claiming that it doesn't know non-English languages very well (which is true).
- sunaookami 2y agoI didn't say that DeepL is better than LLMs only that DeepL is miles better than Google ;). And yeah, I also noticed DeepL sometimes misses some words when translating Japanese (it just ignores them). Google is so much worse though, because it translates Japanese through English (if the target language is not English). DeepL gets better when you have a lot of text. Point is that Google Translate hasn't been #1 since DeepL launched. But now LLMs are (obviously) much better with the added bonus that they can also break down sentences for you.
- Hakkin 2y agoDeepL has its moments but it also has many many failure cases. It particularly is very bad with long text blocks, it commonly completely misses sections of text or repeats the same block of text multiple times.
- KwanEsq 2y agoYeah this is my experience with DeepL as well. It might (sometimes) do better than Google Translate with lone sentences, but give it a few paragraphs and it'll entirely ignore some, and other times it starts repetitively rambling about stuff not even in the original text, quite frequently cars/houses/money.
- hwbunny 2y agoWell, it became worse in the last 6 months.
- snapcaster 2y agoYeah this is old news. When i was in China last year google translate seemed to literally never work (looks of confusion trying to do basic interactions in stores). GPT-4 worked perfectly every time, I think google translate might be another soft abandoned project from google
- ImHereToVote 2y agoGoogle Translate simply doesn't use a GPU for translation. It's as easy as that. There is a huge jump in cost associated with using the GPU.
- dartos 2y agoWhat are you even talking about? You don’t just “use a gpu.” Software doesnt get better by magically throwing it at a gpu. Moreover you can’t just run any old software on a gpu, it has to be built for it. Even then, the gpu is just a speed increase, not a magical make better box. That’s like saying any random text editor would be a better translator if they ran on GPUs. EDIT: google also literally builds its own acceleration hardware, suggesting that they can’t afford GPUs (which they already own for GCP) for google translate is weird.
- ImHereToVote 2y agoNo... I mean LLM's use a GPU. Hence why the cost is different. Changing Google Translate to use an LLM would become much more expensive for Google. This is why you see a different cost structure for using "AI". LLM's can't realistically use the CPU for anything serious.
- trashtester 2y agoYou have it backwards. For any given amount of compute needed, GPU's are a lot cheaper than CPU's. But GPU's can't run all the code you can run on a CPU. The reason models that use GPU's cost more to run, is that they tend to be A LOT more compute intensive. However, if you run the inference for the same models on CPU, they will be both much more expensive than on GPU's (or on specialized tensor silicon) and also slower.
- spacebanana7 2y agoI still find it amazing how LLM translation capabilities are an almost accidental feature. Yet they still managed to leapfrog decades of research and billions of investment dollars in traditional machine translation.
- dartos 2y agoTranslation is the reason transformer models exist. Every other use case is basically an accidental feature
- jampekka 2y agoThe original Transformer, indeed for translation, was an encoder-decoder architecture. Most current LLMs, like Claude 3, are "decoder only". So in a sense the decoder-only translation is kinda accidental feature too.
- _giorgio_ 2y agoSo anything except encoder decoder translation is accidental? Crazy way to define research.
- jampekka 2y agoResearch, especially in academia, isn't usually that interested in the single concrete task that is under study. Translation is just one case where you need to find some good functions to map sequences to other sequences. The "Attention Is All You Need" paper frames the problem and contribution as: "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely." Translation is "just an experiment" for the general architecture and they study other tasks in the paper too. In a sense, all applications of basic research are "accidental".
- poszlem 2y agoNot only that, but it also beats all the Grammarly et al, and on top of that it understands all my commands - "make it less stiff", "please make it less casual", "be ironic" etc. ChatGPT is still better due to it being less castrated and less likely to actually refuse my commands, but Claude works very well with text, when it works.
- rwmj 2y agoGoogle Translate is just terrible anyway. The other day I was using it to read some difficult Kanji and found that it didn't recognize は as being "wa" in certain places, which is something you learn in like your second or third Japanese lesson. And of course it was useless for the Kanji too.
- famahar 2y agoI think it works well enough. I'm glad Google didn't try to perfect the product before releasing. It still feels like magic to me and helps me get the gist of most translations when I take a photo.
- hnfong 2y agoIt probably depends on the language. In my experience, which aligns with GP's, their performance on Asian languages (to/from English) is notoriously bad. I wouldn't trust it at all. My impression is that it's better among European languages, but then I only know English.
- maven29 2y agoPeople trust Google Translate to not go off-topic or "hallucinate" and there is still a lot of value in something semi-deterministic like that. (Edit: To clarify, maintaining semantics is held sacrosanct in classical methods of machine translation)
- famouswaffles 2y ago>People trust Google Translate to not go off-topic or "hallucinate" Then "people" have no idea what they're doing. Google Translate and Deepl "hallucinate" way more than the likes of GPT-4 and Claude 3 for Translation.
- endisneigh 2y agoI’m fascinated by this since I use translate regularly. Do you have an example of google translate doing this?
- famouswaffles 2y agohttps://i.imgur.com/e1aeLej.png https://i.imgur.com/e1aeLej.png
- endisneigh 2y agoWhere is the hallucination? It seems in line with the others.
- famouswaffles 2y agoYou think this: "Swarming like a swarm of bees. He was carried among the people, hanging from the handle. No matter how good you think about the situation you're in, it's disgusting. Where are you now?" is a comparable translation to this: "No matter how you couch it, riding the subway feels disgusting: you dangle like ripe fruit from a hanging vine, squeezed in among humans swarming like bees." Or this ?: "Being crammed among a swarm of humans, dangling from a strap as I'm carried along, is frankly disgusting, no matter how you look at it"
- akumetsu 2y ago"We find that Claude has remarkable resource efficiency – the degree to which the quality of the translation model depends on a language pair’s resource level.” It looks like the main takeaway is that Claude 3 (Opus) is a lot better at low resource language pairs compared with other LLMs and Google Translate. While this is not as important for the typical english usage of Google Translate it promises way better usage of LLMs as simple translators for languages with fewer speakers or just less data available.
- stillathing 2y agoGoogle translate was always bad, at least from and into Russian, German and English. I've used it on and off from 2007 - 2020 just to see how it improved. It didn't. Don't benchmark against something that makes a sophomore sigh. Get 4 years of multicultural students @ universities that focus on language studies. Feed LLM's the translation exercises. Record the sessions with the teachers and professors. Let the students revise. Repeat. Let LLM's learn the nuances and perspectives by following students through their evolution. Build LLMs that want to learn from students and not the other way around. Let the LLMs discuss it all with each other at human speeds and let the crowd moderate the conversations.
- bjord 2y agoyandex translate has been better than google translate for as long as I can remember, at least as far as languages of the former ussr I'm curious how it stacks up against claude
- deepvibrations 2y agoPretty much all LLMs easily beat Google Translate, this is not news... One of the best yet unplanned features of them imo.
- eptcyka 2y agoBut it does. It offers "Rēdze tapa" as one of the alternative translations of "wrist" when translating from English to Latvian. There is no such concept as a "rēdze tapa" in Latvian. It might be a byproduct of translating English first to russian and then to Latvian.
- paxys 2y agoIt's not unplanned. The transformer architecture was literally created for better language translation.
- dontreact 2y agoIm surprised they didn’t try out Gemini. It’s a lot better than Google translate and in my experience this is one use case where Gemini outshines even other frontier models like ChatGPT4
- Tistron 2y agoI've also found gemini to be better than chatGPT (3.5, I'm using both of them for free). The results have felt much more like natural language to me.
- EasyMark 2y agoMaybe the backend uses more energy compared to Google translate?
- dontreact 2y agoMaybe but it just seems silly to not briefly try out a few of the options as a teacher model before settling on one (Claude)
- famouswaffles 2y agoIt's not just Google translate. There no contest between the likes of Claude 3, GPT-4 and traditionally trained models so Google but also Deepl, Papago etc
- ojosilva 2y agoSlightly (un)related, but there's an interesting recent paper on how LLMs perform on code translation: https://arxiv.org/pdf/2308.03109.pdf https://arxiv.org/pdf/2308.03109.pdf
- helsinkiandrew 2y agoOne thing it has going for it is speed though - just about instant after you type each word. Even though I use ChatGPT4 for translating and explaining most Finnish, I still use Google Translate for a quick translation.
- Symmetry 2y agoLLMs also have a wider range of targets than you'd expect. QNTAL is known for putting medieval poetry to modern beats and when I put the old Galician lyrics to Vedes Amigo into Google translate it chokes since it isn't trained in that. But GPT-3 and Claud can both do great, I assume mostly by being fluent in both Portugues and Spanish and being able to interpolate. https://genius.com/Qntal-vedes-amigo-lyrics https://genius.com/Qntal-vedes-amigo-lyrics
- techn00 2y agoI wonder's what's the best quality/price translation service
- smusamashah 2y agoIf LLMs are fed two very different languages with zero connections between each other will they still translate? There won't be any predictable tokens across those languages. Will LLMs still generalize the concept from one language to another or the translation will fail?
- flohofwoe 2y agoIs Google Translate considered "the benchmark to beat"? At least between German and English (both not exactly "fringe languages"), Google Translate quality is still very hit and miss.
- jan_Inkepa 2y agoDeepl has the best rep for European languages - though it now supports many others. I can say it's German translations are pretty good (as a non-native German speaker). But ChatGPT blows it out of the water if you have a specific context and don't need something exactly translated (e.g. formal letter/e-mail writing).
- deleted 2y ago[deleted]
- mark_l_watson 2y agoSorry in advance for a rant against the proliferation of LLM benchmarks: I wrote a book on using LLMs in applications 14 months ago, I am sort of a fan, within reason. I don’t like all the mind-space taken up on comparisons between LLMs on standard test suites because I personally think most instruction tuned models are specifically tuned for the standard tests. I like to see which models people are choosing to use for their projects, and of course, tools like Ollama make it easy to try many models and get a least a subjective feel for what they can do. For model comparison I have my own standard little tests that are my own and private, and I can be sure models are not tuned specifically for. I do the same for commercial LLM APIs and products surfacing models. For example, I have a few short tests I run in Bard to test integration with Google Workspace data, something I am interested in, and it is interesting to track at least subjectively the slow improvement.
- deleted 2y ago[deleted]
- aragonite 2y agoOne area in which Claude seems to unambiguously beat Google Translate is classical Chinese. I recently tried both on a paragraph from a book written during Qing (paragraph 13 of [1]), and it wasn't even close. Some examples: > (GT:) Ten years after Yongzheng's reign, the platform experienced many turmoils, but there was never any attempt to mobilize troops to suppress the troops. > (Claude:) After the tenth year of the Yongzheng reign, although there were frequent disturbances in Taiwan, there were no instances of dispatching troops to suppress the harm caused by the indigenous people, indicating their weakened state (note GT's mistranslation of 臺地 (Taiwan) as 'platform', and the crucial chronological difference between 'After the tenth year of the Yongzheng reign' (correct translation of 雍正十年以後) and 'Ten years after Yongzheng's reign' (incorrect)) --- > (GT:) Since the establishment of trade in the Western Kingdom, his ships have often traveled behind mountains and landed on reefs in the wind. Many people have seen that their appearance and clothing are different, and they cannot understand the language, so their lives may not be saved. In the future, provocations may inevitably arise from various sources! Why conquer it? > (Claude:) Since the opening of trade with Western countries, their ships often sailed behind the mountains. If they encountered storms or reefs and landed, the indigenous people, upon seeing their strange appearance and clothing and being unable to communicate with them, might not spare their lives. Future border conflicts may inevitably begin with these indigenous tribes! How can we deal with this? --- > (GT:) In the sixth year of Tongzhi's reign, an American Roman merchant ship was caught in a storm and ran aground at Guizaijiao, south of Langqiao, under the jurisdiction of Fengshan County. > (Claude:) In the sixth year of the Tongzhi reign (1867), an American merchant ship, the Rover, encountered a storm and ran aground on Guizai Cape south of Langqiao, Fengshan County, breaking the ship. The captain and several sailors swam ashore but were killed by the indigenous people, who also injured a military officer (note how GT simply silently dropped a whole sentence (船主與數水手鳧水近岸，被番所殺，續又傷其兵官一人) about what happened to the the captain & sailors!) --- > (GT:) In the spring of the seventh year, the Prime Minister and the Minister of Foreign Affairs Wang wrote to the governor of Fujian, saying that although Shengfan was not legally bound, the land belonged to China. > (Claude:) In the spring of the seventh year (1868), Prince Gong, the Minister in charge of foreign affairs, sent a letter to the Governor of Fujian, stating that although the indigenous people could not be restrained by law, their land still belonged to China [1] https://ctext.org/wiki.pl?if=en&chapter=149754#:~:text=%E4%B8%83%E5%B9%B4%E6%98%A5,%E8%A1%8C%E7%B1%8C%E5%8F%8A%E3%80%82 https://ctext.org/wiki.pl?if=en&chapter=149754#:~:text=%E4%B... Direct link to Google Translate version: https://tinyurl.com/6juv4nkd https://tinyurl.com/6juv4nkd (what you see may differ slightly from mine)
- jamesponddotco 2y agoThis doesn't surprise me, Claude 3 beats pretty much anything I use it for. Still, my wife and I translate erotica at times, and both Claude and GPT-4 refuse to translate a shit ton of content, while Google Translate and DeepL will happily translate anything we throw at them. I wonder if an uncensored version of Llama 3 would perform better. It's supposed to be on GPT-4 level in certain languages, after all.
- jklinger410 2y agoYes, Claude may make advances here and there in it's tech. But it's main feature is that it is censored (or "safe). That will forever be Claude's main selling point: censorship. Any technological gains it makes will just be marketing fodder to get their pre-captured AI into more people's stack. They are lucky they haven't tried images yet.
- ogrisel 2y agoI assume that Google Translate has a much larger usage volume than any of the free-to-use LLMs. I don't know the average energy/hardware*time usage per query on google translate vs competing LLMs such as Claude 3 Opus but I wouldn't be surprised that a large LLM such as Claude 3 Opus would be much too expensive to be used as the backend model for a free service like Google Translate. The paper authors do acknowledge this concern and run experiments on smaller models with knowledge distillation. However, as far as I know we cannot know if their distilled networks can compete with the current Google Translate system in terms of energy / hardware usage efficiency.
- staticman2 2y agoNeither Google translate or DeepL seem to be good at translating Japanese to English, so this is hardly a surprise to me. In the case of Japanese, only an LLM seems to be capable of tracking the gender of a fictional character, and since gender is rarely indicated in the original japanese language, Google Translate will alternate every sentence indicating whether "He" or "she" took action when referring to the same character. This is just the tip of the iceberg in the problems that come up when trying to translate Japanese. LLM's on the other hand have awareness of the "content" -- they understand what is happening in the original story and it auds their translation choices -- and LLMs tend to be superior at novel translation in general. I imagine the non LLM tools work much better translating between similar languages.
- clauderoux 2y agoI read the article and I found it quite lacking. Why on Earth would you force your LLM to translate sentence by sentence? It ruins the whole interest of LLM, which is to use large contexts to drive your generation. I used Deepl a lot in the past and I had a recurrent problem when translating from French into English, computer related texts. In French, a "chaine" in the context of computer science is mostly translated as "string", however, when translating with Deepl (or Google translate) since the model would not take previous sentences into account, the system would loose the computer context and translate "chaine" into "chain", which of course was usually wrong. But the funniest part was when I wanted to translate "jeûner" in Greek. "jeûner" in French means "to fast", in the sense of not eating. However, Google translated "jeûner" into "gregoria" in Greek, which means fast in the sense of speed... It went through English to translate "jeûner" into "fast" then "fast" into "gregoria"...
- hackmaxim 2y agoI'm one of the authors on the paper. Actually, sentence-by-sentence translation is important in a machine translation system because in many cases users will only provide single sentences. We also test document-level translation in Section 5, and find large improvements (but it isn't the focus of our paper).
- nunez 2y agoBut how does it compare to DeepL (which is AI-assisted IIRC)? It's well known that Translate isn't as great for everyday usage.
- CCmorgan2 2y ago[flagged]
- CCmorgan2 2y ago[flagged]