4 ms·

How would we know that the AI didn't hallucinate when it translated a tablet? In other words, how could one trust the translation? A human translator can still

How would we know that the AI didn't hallucinate when it translated a tablet? In other words, how could one trust the translation? A human translator can still provide some justification for their interpretation. Also, "acquired" could simply mean that the museum paid money to a third party to, well, acquire the piece(s) in question while not inquiring too deeply into how said third party came into possession of the objects that they sold.

bruce511 2y ago
At the very least one could see from the AI if the tablet is interesting or not. That could triage tablets for human translation.
- mangamadaiyan 2y ago
  Sure, as long as one has some means of predicting the probability of false positives as well as false negatives. Until then, colour me unconvinced of the (f)utility of this approach :)
  - timschmidt 2y ago
    LLMs do a pretty darn good job of translating other languages, even preserving inflection and tone and rhyme in some cases. Same when translating programming languages. If the training pool is large enough, they should be quite good at it. No problem marking them as machine translated and keeping track of which have been spot-checked by experts either.
    - Keysh 2y ago
      I think "if the training pool is large enough" is a real issue here. We're not talking about living languages with large, properly attested and annotated corpuses. Indeed, one of the thing you'd probably like the translators to do is identify rare or unique words that can be added to our existing knowledge of these languages.
      - timschmidt 2y ago
        > I think "if the training pool is large enough" is a real issue here. It would be really neat to set up something like a wiki populated with the existing translations and machine translations done via LLM, and to periodically re-train the LLM on all the newly manually verified translations and automatically re-run the machine translations after. The whole thing could move incrementally toward high quality output.
- Groxx 2y ago
  This is the same thing as trusting its translation though. Or worse, since there's more "interpretation" happening (as it's basically a summary rather than a whole text, and you can't trust either of those steps).
  - flir 2y ago
    It's like using a classifier to sort biopsies from most-likely to least-likely to be Really Bad News. A human will get to them all eventually, but you can at least fast-track the ones that are almost certainly Really Bad News. (I think I just said "triage" in a lot more words, to be honest).
wood_spirit 2y ago
I remember a time team episode where they found some Ogham script on a stone in an early Christian church now on a fairway on the isle of Mann. They got photos to the leading professor in Ogham who translated it for them. I happened to mention this on HN a few months ago in relation to something or other, and a commenter replied to me to explain that that translation was no longer sound, and the current understanding was that the tablet said something completely different instead! (Found the link they gave! https://www.babelstone.co.uk/Blog/2008/05/throng-of-fifty-warriors-routed-by.html https://www.babelstone.co.uk/Blog/2008/05/throng-of-fifty-wa...)
- YeGoblynQueenne 2y ago
  If it wasn't an LLM that found the professor's mistake then the point I think you're trying to make is, well, missing the point: a human made a mistake, and humans found out and corrected it. The question is what happens when LLMs make mistakes. Will people still be careful enough to catch and correct them, as often as we find and correct the mistakes we make ourselves? Or will our ability to do so be overwhelmed by the extreme rate at which LLMs can generate text?
  - wood_spirit 2y ago
    My point was more meta - the experts don’t have a fixed opinion on the translations of rare ancient texts in the first place, ergo there is nothing to train the llma on.
    - YeGoblynQueenne 2y ago
      Thanks for clarifying. But note that the same applies to modern translations. There is no agreement between experts (translators) what it means for a translation to be "good" or "bad", or anything in between. So the done thing in translation is to choose some existing translation as a "gold standard" and use that one, without any assumptions of how good or bad it is. Sometimes there's an attempt to rate translations by polling humans but that can not be easily done at scale and certainly not at web-scale, as in LLMs (the modern de facto standard for automatic translation as far as I can tell; I haven't been watching the field closely lately). The same logic is applied to metrics like BLEU and ROUGE scores, used to measure the goodness of a translation. The general idea is that you choose a translation to be the "gold standard" and then compare the n-grams in the gold standard, and the automatic translation, for overlap. It's a very crude and imperfect measure and it's one reason why, in practice, we have no idea how good automatic translations are especially now that automatic translation systems are deployed and working every day with millions of input and output texts that nobody can reasonably be expected to evaluate. In any case just because there's no fixed opinion on what is a good translation, doesn't mean an LLM can't be used to produce one. It probably will. Assuming I understand your point better now, I agree that this is going to cause trouble down the line.

bruce511 2y ago

At the very least one could see from the AI if the tablet is interesting or not. That could triage tablets for human translation.

mangamadaiyan 2y ago
Sure, as long as one has some means of predicting the probability of false positives as well as false negatives. Until then, colour me unconvinced of the (f)utility of this approach :)
- timschmidt 2y ago
  LLMs do a pretty darn good job of translating other languages, even preserving inflection and tone and rhyme in some cases. Same when translating programming languages. If the training pool is large enough, they should be quite good at it. No problem marking them as machine translated and keeping track of which have been spot-checked by experts either.
  - Keysh 2y ago
    I think "if the training pool is large enough" is a real issue here. We're not talking about living languages with large, properly attested and annotated corpuses. Indeed, one of the thing you'd probably like the translators to do is identify rare or unique words that can be added to our existing knowledge of these languages.
    - timschmidt 2y ago
      > I think "if the training pool is large enough" is a real issue here. It would be really neat to set up something like a wiki populated with the existing translations and machine translations done via LLM, and to periodically re-train the LLM on all the newly manually verified translations and automatically re-run the machine translations after. The whole thing could move incrementally toward high quality output.
Groxx 2y ago
This is the same thing as trusting its translation though. Or worse, since there's more "interpretation" happening (as it's basically a summary rather than a whole text, and you can't trust either of those steps).
- flir 2y ago
  It's like using a classifier to sort biopsies from most-likely to least-likely to be Really Bad News. A human will get to them all eventually, but you can at least fast-track the ones that are almost certainly Really Bad News. (I think I just said "triage" in a lot more words, to be honest).

wood_spirit 2y ago

I remember a time team episode where they found some Ogham script on a stone in an early Christian church now on a fairway on the isle of Mann. They got photos to the leading professor in Ogham who translated it for them. I happened to mention this on HN a few months ago in relation to something or other, and a commenter replied to me to explain that that translation was no longer sound, and the current understanding was that the tablet said something completely different instead! (Found the link they gave! https://www.babelstone.co.uk/Blog/2008/05/throng-of-fifty-warriors-routed-by.html https://www.babelstone.co.uk/Blog/2008/05/throng-of-fifty-wa...)

YeGoblynQueenne 2y ago
If it wasn't an LLM that found the professor's mistake then the point I think you're trying to make is, well, missing the point: a human made a mistake, and humans found out and corrected it. The question is what happens when LLMs make mistakes. Will people still be careful enough to catch and correct them, as often as we find and correct the mistakes we make ourselves? Or will our ability to do so be overwhelmed by the extreme rate at which LLMs can generate text?
- wood_spirit 2y ago
  My point was more meta - the experts don’t have a fixed opinion on the translations of rare ancient texts in the first place, ergo there is nothing to train the llma on.
  - YeGoblynQueenne 2y ago
    Thanks for clarifying. But note that the same applies to modern translations. There is no agreement between experts (translators) what it means for a translation to be "good" or "bad", or anything in between. So the done thing in translation is to choose some existing translation as a "gold standard" and use that one, without any assumptions of how good or bad it is. Sometimes there's an attempt to rate translations by polling humans but that can not be easily done at scale and certainly not at web-scale, as in LLMs (the modern de facto standard for automatic translation as far as I can tell; I haven't been watching the field closely lately). The same logic is applied to metrics like BLEU and ROUGE scores, used to measure the goodness of a translation. The general idea is that you choose a translation to be the "gold standard" and then compare the n-grams in the gold standard, and the automatic translation, for overlap. It's a very crude and imperfect measure and it's one reason why, in practice, we have no idea how good automatic translations are especially now that automatic translation systems are deployed and working every day with millions of input and output texts that nobody can reasonably be expected to evaluate. In any case just because there's no fixed opinion on what is a good translation, doesn't mean an LLM can't be used to produce one. It probably will. Assuming I understand your point better now, I agree that this is going to cause trouble down the line.