3 ms·

It is absolutely revolting to see LLM translation being sold as "better than professionals". Yeah, yeah, I saw you attached a caveat to it. No. You don't need t

It is absolutely revolting to see LLM translation being sold as "better than professionals". Yeah, yeah, I saw you attached a caveat to it. No. You don't need to be dishonest. LLM translation is shit but it's borderline free, and you want the benefit of translation without having to pay for it. Who can argue with the economics of "free", after all? I would leave it at that, but then I consider what you're actually using it for... > two-person founding team running a language-learning app in dozens of languages, so the copy is read by people who are unusually sensitive to it. They aren't "unusually sensitive to it". Language learners don't fucking know what native speech in the target language sounds like, they're learning it! Nor does your two person team. How could you possibly claim to objectively evaluate how good translations are in dozens of languages you don't speak? 58, according to your site! This is beyond scummy even for the usual AI boosters, and will deceive people trying to learn, who don't know better, with low-grade slop. Disgusting.

thaumasiotes 3mo ago
> LLM translation is shit but it's borderline free I don't think this is true. LLM translation will generally satisfy the core goals of (1) communicating the information that's present in the source document; and (2) not communicating information that isn't present in the source document. There are some style issues -- I have a comment sidethread complaining about them -- but those are very much second-order effects compared to "can I get a foreigner to understand what my document says?"
- applfanboysbgon 3mo ago
  Possibly, if it's a strictly formulated medical, business, or legal document (but if it's a medical, business, or legal document, being right 99% of the time isn't good enough!). Google Translate could already do that much, incidentally. If it's not a formal document - if it's anything vaguely resembling casual speech or literary writing - an LLM will butcher it in all kinds of ways no matter how much context it has.
tetromino_ 3mo ago
> LLM translation is shit Compared to who? Compared to a professional with a language degree, with a feel and love for both languages and an understanding of the subject area, LLM translation is shit, no question. But such professionals don't get hired by startups to localize their websites, too expensive. When comparing to paid- by-the-word contractors who do website localization, I am not so sure that LLM is worse - because in practice too many biologically-human-localized websites and apps are translated horrifically.
- applfanboysbgon 3mo ago
  My experience with dirt cheap contractors you're talking about is that they simply throw what you put into Google Translate / an LLM themselves. So yes, strictly speaking, doing that yourself is better than paying someone else to do it for you. But it's not better than hiring a real professional, so let's not pretend it is. It is what it is: cheap. Be up front about that. And I think using machine translation, and being honest about it, is completely understandable for some tiny startup that just wants their website localized. I think it is completely and totally unacceptable for a language learning application, though.
  - stuess 3mo ago
    The point I'm making isn't (in life or in this blog post) that LLMs do a better job than humans. My point is that LLMs can get pretty close with the right context, and in some circumstances they can build context better than the typical human translator would. Most CAT tools do vet for ML/LLM translations these days, so "cheap" human translation is mostly ML-assisted and human-edited, but my entire point is that they're not to blame, the majority of translation fails happens because of the lack of context, not because the human didn't try. It's about setting the site and l10n infrastructure out in a way that gives them the freedom to translate in the way that makes most sense for their language without being forced into "English in a costume". It is unfortunately a fact that one can be both things: a startup and a language learning app. But my entire point wasn't about LLMs vs. humans, it was about laying out the architecture in a way that enables great l10n, whether we do that with LLMs today or humans tomorrow once we can afford it.
hidalgoerrant 3mo ago
Language learners need to be able to use the site in their native language. The whole world doesn’t feel comfortable using English as a second language to learn another language. None of my students use English to learn Spanish or Mandarin Chinese. They use their mother tongues so they don’t have to add to their cognitive load by processing another language on top of the one they're already learning. The localized sites are exactly what they can judge as native speakers. They’re judging copy in their own language, not the language they’re learning. We noticed that a good share of our learners were already using Google Translate built into their browsers to access the site in their native languages. The quality is atrocious, so what we’re offering is a step change from that. We offer 58 languages to learners. Only 16 have been localized. That means the chrome of the website and apps -- the learner’s native language -- not the video content they watch in the target language they’re learning. Between my co-founder and me plus people who have kindly offered to review the site we can judge seven of those localizations well enough to see that they’re orders of magnitude better than what Google Translate would concoct for them.
stuess 3mo ago
So our approach here to vet these: Between the two of us, we do speak 7 languages well enough to judge a localized site, and we have people around us for review on some more. We run our workflow for a new localized version of say, German, then judge the quality and add rules to our prompt, guidance and voice guide for issues we spot, never hand-editing the actual strings. We then re-run it until it's in a place where we feel this feels fairly native (feels transcreated, not calqued) and serves its purpose well). Once we had that done in a good amount of locales, we started running it in languages we can't judge ourselves. Here we do depend on people on the app/website to spot and help us if we got it wrong somewhere, so it's a living process that evolves. Here's an example of how that guidance and prompt evolved over time, in case you're interested: https://gist.github.com/tobyurff/2b461c259c34dfc6758a39329822dab8 https://gist.github.com/tobyurff/2b461c259c34dfc6758a3932982...

thaumasiotes 3mo ago

> LLM translation is shit but it's borderline free I don't think this is true. LLM translation will generally satisfy the core goals of (1) communicating the information that's present in the source document; and (2) not communicating information that isn't present in the source document. There are some style issues -- I have a comment sidethread complaining about them -- but those are very much second-order effects compared to "can I get a foreigner to understand what my document says?"

applfanboysbgon 3mo ago
Possibly, if it's a strictly formulated medical, business, or legal document (but if it's a medical, business, or legal document, being right 99% of the time isn't good enough!). Google Translate could already do that much, incidentally. If it's not a formal document - if it's anything vaguely resembling casual speech or literary writing - an LLM will butcher it in all kinds of ways no matter how much context it has.

tetromino_ 3mo ago

> LLM translation is shit Compared to who? Compared to a professional with a language degree, with a feel and love for both languages and an understanding of the subject area, LLM translation is shit, no question. But such professionals don't get hired by startups to localize their websites, too expensive. When comparing to paid- by-the-word contractors who do website localization, I am not so sure that LLM is worse - because in practice too many biologically-human-localized websites and apps are translated horrifically.

applfanboysbgon 3mo ago
My experience with dirt cheap contractors you're talking about is that they simply throw what you put into Google Translate / an LLM themselves. So yes, strictly speaking, doing that yourself is better than paying someone else to do it for you. But it's not better than hiring a real professional, so let's not pretend it is. It is what it is: cheap. Be up front about that. And I think using machine translation, and being honest about it, is completely understandable for some tiny startup that just wants their website localized. I think it is completely and totally unacceptable for a language learning application, though.
- stuess 3mo ago
  The point I'm making isn't (in life or in this blog post) that LLMs do a better job than humans. My point is that LLMs can get pretty close with the right context, and in some circumstances they can build context better than the typical human translator would. Most CAT tools do vet for ML/LLM translations these days, so "cheap" human translation is mostly ML-assisted and human-edited, but my entire point is that they're not to blame, the majority of translation fails happens because of the lack of context, not because the human didn't try. It's about setting the site and l10n infrastructure out in a way that gives them the freedom to translate in the way that makes most sense for their language without being forced into "English in a costume". It is unfortunately a fact that one can be both things: a startup and a language learning app. But my entire point wasn't about LLMs vs. humans, it was about laying out the architecture in a way that enables great l10n, whether we do that with LLMs today or humans tomorrow once we can afford it.

hidalgoerrant 3mo ago

Language learners need to be able to use the site in their native language. The whole world doesn’t feel comfortable using English as a second language to learn another language. None of my students use English to learn Spanish or Mandarin Chinese. They use their mother tongues so they don’t have to add to their cognitive load by processing another language on top of the one they're already learning. The localized sites are exactly what they can judge as native speakers. They’re judging copy in their own language, not the language they’re learning. We noticed that a good share of our learners were already using Google Translate built into their browsers to access the site in their native languages. The quality is atrocious, so what we’re offering is a step change from that. We offer 58 languages to learners. Only 16 have been localized. That means the chrome of the website and apps -- the learner’s native language -- not the video content they watch in the target language they’re learning. Between my co-founder and me plus people who have kindly offered to review the site we can judge seven of those localizations well enough to see that they’re orders of magnitude better than what Google Translate would concoct for them.

stuess 3mo ago

So our approach here to vet these: Between the two of us, we do speak 7 languages well enough to judge a localized site, and we have people around us for review on some more. We run our workflow for a new localized version of say, German, then judge the quality and add rules to our prompt, guidance and voice guide for issues we spot, never hand-editing the actual strings. We then re-run it until it's in a place where we feel this feels fairly native (feels transcreated, not calqued) and serves its purpose well). Once we had that done in a good amount of locales, we started running it in languages we can't judge ourselves. Here we do depend on people on the app/website to spot and help us if we got it wrong somewhere, so it's a living process that evolves. Here's an example of how that guidance and prompt evolved over time, in case you're interested: https://gist.github.com/tobyurff/2b461c259c34dfc6758a39329822dab8 https://gist.github.com/tobyurff/2b461c259c34dfc6758a3932982...