4 ms·

I remember a time team episode where they found some Ogham script on a stone in an early Christian church now on a fairway on the isle of Mann. They got photos

I remember a time team episode where they found some Ogham script on a stone in an early Christian church now on a fairway on the isle of Mann. They got photos to the leading professor in Ogham who translated it for them. I happened to mention this on HN a few months ago in relation to something or other, and a commenter replied to me to explain that that translation was no longer sound, and the current understanding was that the tablet said something completely different instead! (Found the link they gave! https://www.babelstone.co.uk/Blog/2008/05/throng-of-fifty-warriors-routed-by.html https://www.babelstone.co.uk/Blog/2008/05/throng-of-fifty-wa...)

YeGoblynQueenne 2y ago
If it wasn't an LLM that found the professor's mistake then the point I think you're trying to make is, well, missing the point: a human made a mistake, and humans found out and corrected it. The question is what happens when LLMs make mistakes. Will people still be careful enough to catch and correct them, as often as we find and correct the mistakes we make ourselves? Or will our ability to do so be overwhelmed by the extreme rate at which LLMs can generate text?
- wood_spirit 2y ago
  My point was more meta - the experts don’t have a fixed opinion on the translations of rare ancient texts in the first place, ergo there is nothing to train the llma on.
  - YeGoblynQueenne 2y ago
    Thanks for clarifying. But note that the same applies to modern translations. There is no agreement between experts (translators) what it means for a translation to be "good" or "bad", or anything in between. So the done thing in translation is to choose some existing translation as a "gold standard" and use that one, without any assumptions of how good or bad it is. Sometimes there's an attempt to rate translations by polling humans but that can not be easily done at scale and certainly not at web-scale, as in LLMs (the modern de facto standard for automatic translation as far as I can tell; I haven't been watching the field closely lately). The same logic is applied to metrics like BLEU and ROUGE scores, used to measure the goodness of a translation. The general idea is that you choose a translation to be the "gold standard" and then compare the n-grams in the gold standard, and the automatic translation, for overlap. It's a very crude and imperfect measure and it's one reason why, in practice, we have no idea how good automatic translations are especially now that automatic translation systems are deployed and working every day with millions of input and output texts that nobody can reasonably be expected to evaluate. In any case just because there's no fixed opinion on what is a good translation, doesn't mean an LLM can't be used to produce one. It probably will. Assuming I understand your point better now, I agree that this is going to cause trouble down the line.

YeGoblynQueenne 2y ago

If it wasn't an LLM that found the professor's mistake then the point I think you're trying to make is, well, missing the point: a human made a mistake, and humans found out and corrected it. The question is what happens when LLMs make mistakes. Will people still be careful enough to catch and correct them, as often as we find and correct the mistakes we make ourselves? Or will our ability to do so be overwhelmed by the extreme rate at which LLMs can generate text?

wood_spirit 2y ago
My point was more meta - the experts don’t have a fixed opinion on the translations of rare ancient texts in the first place, ergo there is nothing to train the llma on.
- YeGoblynQueenne 2y ago
  Thanks for clarifying. But note that the same applies to modern translations. There is no agreement between experts (translators) what it means for a translation to be "good" or "bad", or anything in between. So the done thing in translation is to choose some existing translation as a "gold standard" and use that one, without any assumptions of how good or bad it is. Sometimes there's an attempt to rate translations by polling humans but that can not be easily done at scale and certainly not at web-scale, as in LLMs (the modern de facto standard for automatic translation as far as I can tell; I haven't been watching the field closely lately). The same logic is applied to metrics like BLEU and ROUGE scores, used to measure the goodness of a translation. The general idea is that you choose a translation to be the "gold standard" and then compare the n-grams in the gold standard, and the automatic translation, for overlap. It's a very crude and imperfect measure and it's one reason why, in practice, we have no idea how good automatic translations are especially now that automatic translation systems are deployed and working every day with millions of input and output texts that nobody can reasonably be expected to evaluate. In any case just because there's no fixed opinion on what is a good translation, doesn't mean an LLM can't be used to produce one. It probably will. Assuming I understand your point better now, I agree that this is going to cause trouble down the line.