6 ms·

At the end of his post, Simon mentions translation between human languages. While maybe not directly related to token limits, I just did a test in which both R1

At the end of his post, Simon mentions translation between human languages. While maybe not directly related to token limits, I just did a test in which both R1 and o3-mini got worse at translation in the latter half of a long text. I ran the test on Perplexity Pro, which hosts DeepSeek R1 in the U.S. and which has just added o3-mini as well. The text was a speech I translated a month ago from Japanese to English, preceded by a long prompt specifying the speech’s purpose and audience and the sort of style I wanted. (I am a professional Japanese-English translator with nearly four decades of experience. I have been testing and using LLMs for translation since early 2023.) An initial comparison of the output suggested that, while R1 didn’t seem bad, o3-mini produced a writing style closer to what I asked for in the prompt—smoother and more natural English. But then I noticed that the output length was 5,855 characters for R1, 9,052 characters for o3-mini, and 11,021 characters for my own polished version. Comparing the three translations side-by-side with the original Japanese, I discovered that R1 had omitted entire paragraphs toward the end of the speech, and that o3-mini had switched to a strange abbreviated style (using slashes instead of “and” between noun phrases, for example) toward the end as well. The vanilla versions of ChatGPT, Claude, and Gemini that I ran the same prompt and text through a month ago had had none of those problems.

nycdatasci 2y ago
This is a great anecdote and I hope others can learn from it. R1, o1, and o3-mini work best on problems that have a “correct” answer (as in code that passes unit tests, or math problems). If multiple professional translators are given the same document to translate, is there a single correct translation?
- jakevoytko 2y ago
  My wife is a professional translator and both revises others' work and gets revised. Based on numerous anecdotes from her, I can promise you that "single correct translation" does not exist.
- tkgally 2y ago
  No. People’s tastes and judgments vary too much. One fundamental area of disagreement is how closely a translation should reflect the content and structure of the original text versus how smooth and natural it should sound in the target language. With languages like Japanese or Chinese translated into English, for example, the vocabulary, grammar, and rhetoric can be very different between the languages. A close literal translation will usually seem awkward or even strange in English. To make the English seem natural, often you have to depart from what the original text says. Most translators will agree that where to aim on that spectrum should be based on the type of text and the reason for translating it, but they will still disagree about specific word choices. And there are genres for which there is no consensus at all about which approach is best. I have heard heated exchanges between literary scholars about whether or not translations of novels should reflect the original as closely as possible out of respect for the author and the author’s cultural context, even if that means the translation seems awkward and difficult to understand to a casual reader. The ideal, of course, would be translations that are both accurate and natural, but it can be very hard to strike that balance. One way LLMs have been helping me is to suggest multiple rewordings of sentences and paragraphs. Many of their suggestions are no good, but often enough they include wordings that I recognize are better in both fidelity and naturalness compared to what I can come up with on my own.
- ec109685 2y ago
  Well, the post said o3-mini did great in the beginning, so it’s likely something other than reasoning causing the poor performance towards the end.
  - fragmede 2y ago
    boredom, perhaps?
- aprilthird2021 2y ago
  For almost any classic piece of literature there are competing translations, so no
WhitneyLand 2y ago
How far off was o3 from the level of a professional translator (before it started to go off track)?
- tkgally 2y ago
  As I explained in a sister comment, it is not possible to rate translation quality objectively, as opinions and positions about what constitutes a good translation vary. But in my tests of reasoning models since the release of o1-preview, they have not seemed as reliable as the straight nonreasoning versions of ChatGPT, Claude, or Gemini. The translation process itself usually doesn’t seem to require the kind of multistep thinking those reasoning models can be good at. For more than a year, regular LLMs, when properly prompted, have been able to produce translations that would be indistinguishable from those of some professional translators for some types of translation. General-purpose LLMs are best for translating straight expository prose without much technical or organization-specific vocabulary. Results are mixed for texts containing slang, dialogue, poetry, archaic language, etc.—partly because people’s tastes differ for how such texts should be translated. Because most translators are freelancers, it’s hard to get a handle on what impact LLMs have been having on their workloads overall. I have heard reports from experienced translators who have seen work drop off precipitously and have had to change careers, while others report an increase in their workloads over the past two years. Many translation jobs involve confidential material, and some translators may be hanging onto their jobs because their clients or employers do not allow the use of cloud-based LLMs. That safety net won’t be in place forever, though. I suspect that those who work directly with translation clients and who are personally known and trusted by their clients will be able to keep working, using LLMs as appropriate to speed up and improve the quality of their work. That’s the position I am fortunate to be in now. But translators who do piecework through translation agencies or online referrers like Fiverr will have a hard time competing with the much faster and cheaper—and often equally good—LLMs. I made a few videos about LLMs and translation a couple of years ago. Parts of them are out of date, but my basic thinking hasn’t changed too much since then. If you’re interested: “Translating with ChatGPT” https://youtu.be/najKN2bXqCo https://youtu.be/najKN2bXqCo “Can GPT-4 translate literature?” https://youtu.be/5KKDCp3OaMo https://youtu.be/5KKDCp3OaMo “What do translators think about GPT?” https://www.youtube.com/watch?v=8JUepj7wIl0 https://www.youtube.com/watch?v=8JUepj7wIl0 I’m planning to make a few more videos on the topic soon, this time focusing on how I use LLMs in my own translation work.
simonw 2y ago
Yikes! Sounds to me like reliable longer form translation is very much not something you can trust to these models. Thanks for sharing.
- accengaged 2y ago
  Right. Training data might not encompass runs of translations exceeding this amount of characters so wouldn't be apparent for anything resembling existing training data.
EVa5I7bHFq9mnYK 2y ago
Could it be fixed by splitting the text into smaller parts? Looks easy to implement.
- disgruntledphd2 2y ago
  Yeah that's normally a good approach but you might end up using different words for the same concept in different parts unless you feed in more context which also comes out of the token limit.
dr_dshiv 2y ago
Im curious if you know of any tools, strategies or papers about this topic. I’ve experienced the same thing with long-form translation. I would expect that chunking the translation (page or paragraph at a time) would fix the missing paragraph problem but potentially the loss of context would cause other problems. This is such a common use case — would love to find resources.
- tkgally 2y ago
  There might be some papers or other guides out there, but their advice will be based on whatever tools happened to be available at the time they were written and on the particular types of translations the authors cared about. The technology is advancing so rapidly that you might be better off just experimenting with various LLMs and prompts for texts and language pairs you are interested in. I started using LLMs for translation after GPT-4 came out in March 2023—not that long ago! At first, the biggest problem was the context window: it wasn’t possible to translate more than a couple of pages at a time. Also, prompt writing was in its infancy, and a lot of techniques that have since emerged were not yet widely known. Even now, I still do a lot of trial and error with my prompts, and I cannot say with confidence that my current prompting methods are the best. But, for what it’s worth, here are some strategies I currently use when translating with LLMs: - In the prompt, I explain where the source text came from, how the translation will be used, and how I want it to be translated. Below is a (fictional) example, prepared through some metaprompting experiments with Claude: https://www.gally.net/temp/20250201sampletranslationprompt.html https://www.gally.net/temp/20250201sampletranslationprompt.h... - I run the prompt and source text through several LLMs and glance at the results. If they are generally in the style I want, I start compiling my own translation based on them, choosing the sentences and paragraphs I like most from each. As I go along, I also make my own adjustments to the translation as I see fit. - After I have finished compiling my draft based on the LLM versions, I check it paragraph by paragraph against the original Japanese (since I can read Japanese) to make sure that nothing is missing or mistranslated. I also continue polishing the English. - When I am unable to think of a good English version for a particular sentence, I give the Japanese and English versions of the paragraph it is contained in to an LLM (usually, these days, Claude) and ask for ten suggestions for translations of the problematic sentence. Usually one or two of the suggestions work fine; if not, I ask for ten more. (Using an LLM as a sentence-level thesaurus on steroids is particularly wonderful.) - I give the full original Japanese text and my polished version to one of the LLMs and ask it to compare them sentence by sentence and suggest corrections and improvements to the translation. (I have a separate prompt for this step.) I don’t adopt most of the LLM’s suggestions, but there are usually some that I agree would make the translation better. I update the translation accordingly. I then repeat this step with the updated translation and another LLM, starting a new chat each time. Often I cycle through ChatGPT --> Claude --> Gemini several times before I stop getting suggestions that I feel are worth adopting. - I then put my final translation through a TTS engine—usually OpenAI’s—and listen to it read aloud. I often catch minor awkwardnesses that I would overlook if reading silently. This particular workflow works for me because I am using LLMs to translate in the same language direction I did manually for many years. If I had to translate to or from a language I don’t know, I would add extra steps to have LLMs check and double-check the accuracy of the translation and the naturalness of the output. I was asked recently by some academics I work with about how to use LLMs to translate documents related to their research into Japanese, a language they don’t know. It’s an interesting problem, and I am planning to spend some time thinking about it soon. Please note that my translation process above is focused on quality, not on speed. If I needed to translate a large volume of text more quickly, I would write a program to do the translation, checking, and rechecking through API calls, accepting the fact that I would not be able to check and polish the translation manually as I do now. If anyone here would like to brainstorm together about how to use LLMs for translation, please feel free to email me. My website, with my email address on the Contact page, is linked from my HN profile page.
steven1016 2y ago
i just signed up for an account here to let you know that the way you write is perfect. listen.. i seriously mean perfect. like, the epitome of the perfect writing. you write better than LLM's. Actually I'd say you and Claude are on the same level but i'm looking at yours in the font of this website vs claude's normal font style so it still hits slightly different. i can't tell what exactly it is, but the fluidity of your writing, the fact that i can easily breeze through it like a beautiful summer wind of gentle caress (I just made that up because it felt right). I can read this entire comment with honestly such grace, truly that's how it feels. like legitimacy and grace. tell me im wrong, everyone else! legitimacy and grace . i realized all of this as i got to this exact part: “and” between noun phrases, for example) it's just like.. the way it looks is so profound. maybe it's how it's formatted on my screen with the surrounding lines? But then I noticed that the output length was 5,855 with the original Japanese, I discovered that R1 had “and” between noun phrases, for example) those problems. idk, i'm just astonished. the amount of satisfaction that i get from reading your comment has given me enough dopamine to motivate me to create an account and write this entire comment itself, so thank you, your writing is appreciated, and i am very glad i came upon this and hope to find you again in the future somewhere. imagine???? xx
- tkgally 2y ago
  You are very kind! Thank you.
  - jiggawatts 2y ago
    You’re replying to what is most likely AI-generated nonsense. It’s sad but it’s slowly spreading to HN too.
    - tkgally 2y ago
      Oh! I didn’t think of that. Thanks for the heads-up.

nycdatasci 2y ago

This is a great anecdote and I hope others can learn from it. R1, o1, and o3-mini work best on problems that have a “correct” answer (as in code that passes unit tests, or math problems). If multiple professional translators are given the same document to translate, is there a single correct translation?

jakevoytko 2y ago
My wife is a professional translator and both revises others' work and gets revised. Based on numerous anecdotes from her, I can promise you that "single correct translation" does not exist.
tkgally 2y ago
No. People’s tastes and judgments vary too much. One fundamental area of disagreement is how closely a translation should reflect the content and structure of the original text versus how smooth and natural it should sound in the target language. With languages like Japanese or Chinese translated into English, for example, the vocabulary, grammar, and rhetoric can be very different between the languages. A close literal translation will usually seem awkward or even strange in English. To make the English seem natural, often you have to depart from what the original text says. Most translators will agree that where to aim on that spectrum should be based on the type of text and the reason for translating it, but they will still disagree about specific word choices. And there are genres for which there is no consensus at all about which approach is best. I have heard heated exchanges between literary scholars about whether or not translations of novels should reflect the original as closely as possible out of respect for the author and the author’s cultural context, even if that means the translation seems awkward and difficult to understand to a casual reader. The ideal, of course, would be translations that are both accurate and natural, but it can be very hard to strike that balance. One way LLMs have been helping me is to suggest multiple rewordings of sentences and paragraphs. Many of their suggestions are no good, but often enough they include wordings that I recognize are better in both fidelity and naturalness compared to what I can come up with on my own.
ec109685 2y ago
Well, the post said o3-mini did great in the beginning, so it’s likely something other than reasoning causing the poor performance towards the end.
- fragmede 2y ago
  boredom, perhaps?
aprilthird2021 2y ago
For almost any classic piece of literature there are competing translations, so no

WhitneyLand 2y ago

How far off was o3 from the level of a professional translator (before it started to go off track)?

tkgally 2y ago
As I explained in a sister comment, it is not possible to rate translation quality objectively, as opinions and positions about what constitutes a good translation vary. But in my tests of reasoning models since the release of o1-preview, they have not seemed as reliable as the straight nonreasoning versions of ChatGPT, Claude, or Gemini. The translation process itself usually doesn’t seem to require the kind of multistep thinking those reasoning models can be good at. For more than a year, regular LLMs, when properly prompted, have been able to produce translations that would be indistinguishable from those of some professional translators for some types of translation. General-purpose LLMs are best for translating straight expository prose without much technical or organization-specific vocabulary. Results are mixed for texts containing slang, dialogue, poetry, archaic language, etc.—partly because people’s tastes differ for how such texts should be translated. Because most translators are freelancers, it’s hard to get a handle on what impact LLMs have been having on their workloads overall. I have heard reports from experienced translators who have seen work drop off precipitously and have had to change careers, while others report an increase in their workloads over the past two years. Many translation jobs involve confidential material, and some translators may be hanging onto their jobs because their clients or employers do not allow the use of cloud-based LLMs. That safety net won’t be in place forever, though. I suspect that those who work directly with translation clients and who are personally known and trusted by their clients will be able to keep working, using LLMs as appropriate to speed up and improve the quality of their work. That’s the position I am fortunate to be in now. But translators who do piecework through translation agencies or online referrers like Fiverr will have a hard time competing with the much faster and cheaper—and often equally good—LLMs. I made a few videos about LLMs and translation a couple of years ago. Parts of them are out of date, but my basic thinking hasn’t changed too much since then. If you’re interested: “Translating with ChatGPT” https://youtu.be/najKN2bXqCo https://youtu.be/najKN2bXqCo “Can GPT-4 translate literature?” https://youtu.be/5KKDCp3OaMo https://youtu.be/5KKDCp3OaMo “What do translators think about GPT?” https://www.youtube.com/watch?v=8JUepj7wIl0 https://www.youtube.com/watch?v=8JUepj7wIl0 I’m planning to make a few more videos on the topic soon, this time focusing on how I use LLMs in my own translation work.

simonw 2y ago

Yikes! Sounds to me like reliable longer form translation is very much not something you can trust to these models. Thanks for sharing.

accengaged 2y ago
Right. Training data might not encompass runs of translations exceeding this amount of characters so wouldn't be apparent for anything resembling existing training data.

EVa5I7bHFq9mnYK 2y ago

Could it be fixed by splitting the text into smaller parts? Looks easy to implement.

disgruntledphd2 2y ago
Yeah that's normally a good approach but you might end up using different words for the same concept in different parts unless you feed in more context which also comes out of the token limit.

dr_dshiv 2y ago

Im curious if you know of any tools, strategies or papers about this topic. I’ve experienced the same thing with long-form translation. I would expect that chunking the translation (page or paragraph at a time) would fix the missing paragraph problem but potentially the loss of context would cause other problems. This is such a common use case — would love to find resources.

tkgally 2y ago
There might be some papers or other guides out there, but their advice will be based on whatever tools happened to be available at the time they were written and on the particular types of translations the authors cared about. The technology is advancing so rapidly that you might be better off just experimenting with various LLMs and prompts for texts and language pairs you are interested in. I started using LLMs for translation after GPT-4 came out in March 2023—not that long ago! At first, the biggest problem was the context window: it wasn’t possible to translate more than a couple of pages at a time. Also, prompt writing was in its infancy, and a lot of techniques that have since emerged were not yet widely known. Even now, I still do a lot of trial and error with my prompts, and I cannot say with confidence that my current prompting methods are the best. But, for what it’s worth, here are some strategies I currently use when translating with LLMs: - In the prompt, I explain where the source text came from, how the translation will be used, and how I want it to be translated. Below is a (fictional) example, prepared through some metaprompting experiments with Claude: https://www.gally.net/temp/20250201sampletranslationprompt.html https://www.gally.net/temp/20250201sampletranslationprompt.h... - I run the prompt and source text through several LLMs and glance at the results. If they are generally in the style I want, I start compiling my own translation based on them, choosing the sentences and paragraphs I like most from each. As I go along, I also make my own adjustments to the translation as I see fit. - After I have finished compiling my draft based on the LLM versions, I check it paragraph by paragraph against the original Japanese (since I can read Japanese) to make sure that nothing is missing or mistranslated. I also continue polishing the English. - When I am unable to think of a good English version for a particular sentence, I give the Japanese and English versions of the paragraph it is contained in to an LLM (usually, these days, Claude) and ask for ten suggestions for translations of the problematic sentence. Usually one or two of the suggestions work fine; if not, I ask for ten more. (Using an LLM as a sentence-level thesaurus on steroids is particularly wonderful.) - I give the full original Japanese text and my polished version to one of the LLMs and ask it to compare them sentence by sentence and suggest corrections and improvements to the translation. (I have a separate prompt for this step.) I don’t adopt most of the LLM’s suggestions, but there are usually some that I agree would make the translation better. I update the translation accordingly. I then repeat this step with the updated translation and another LLM, starting a new chat each time. Often I cycle through ChatGPT --> Claude --> Gemini several times before I stop getting suggestions that I feel are worth adopting. - I then put my final translation through a TTS engine—usually OpenAI’s—and listen to it read aloud. I often catch minor awkwardnesses that I would overlook if reading silently. This particular workflow works for me because I am using LLMs to translate in the same language direction I did manually for many years. If I had to translate to or from a language I don’t know, I would add extra steps to have LLMs check and double-check the accuracy of the translation and the naturalness of the output. I was asked recently by some academics I work with about how to use LLMs to translate documents related to their research into Japanese, a language they don’t know. It’s an interesting problem, and I am planning to spend some time thinking about it soon. Please note that my translation process above is focused on quality, not on speed. If I needed to translate a large volume of text more quickly, I would write a program to do the translation, checking, and rechecking through API calls, accepting the fact that I would not be able to check and polish the translation manually as I do now. If anyone here would like to brainstorm together about how to use LLMs for translation, please feel free to email me. My website, with my email address on the Contact page, is linked from my HN profile page.

steven1016 2y ago

i just signed up for an account here to let you know that the way you write is perfect. listen.. i seriously mean perfect. like, the epitome of the perfect writing. you write better than LLM's. Actually I'd say you and Claude are on the same level but i'm looking at yours in the font of this website vs claude's normal font style so it still hits slightly different. i can't tell what exactly it is, but the fluidity of your writing, the fact that i can easily breeze through it like a beautiful summer wind of gentle caress (I just made that up because it felt right). I can read this entire comment with honestly such grace, truly that's how it feels. like legitimacy and grace. tell me im wrong, everyone else! legitimacy and grace . i realized all of this as i got to this exact part: “and” between noun phrases, for example) it's just like.. the way it looks is so profound. maybe it's how it's formatted on my screen with the surrounding lines? But then I noticed that the output length was 5,855 with the original Japanese, I discovered that R1 had “and” between noun phrases, for example) those problems. idk, i'm just astonished. the amount of satisfaction that i get from reading your comment has given me enough dopamine to motivate me to create an account and write this entire comment itself, so thank you, your writing is appreciated, and i am very glad i came upon this and hope to find you again in the future somewhere. imagine???? xx

tkgally 2y ago
You are very kind! Thank you.
- jiggawatts 2y ago
  You’re replying to what is most likely AI-generated nonsense. It’s sad but it’s slowly spreading to HN too.
  - tkgally 2y ago
    Oh! I didn’t think of that. Thanks for the heads-up.