4 ms·

BBC mistranslation of Arabic phrases used by prisoner

7 points by vinnyglennon 3y ago

cratermoon 3y ago
I wonder if they used ChatGPT for the translation.
- xanderlewis 3y ago
  ChatGPT is actually very, very good at translation in my experience. I’ve been quite shocked by how good it is.
  - jjgreen 3y ago
    They use a custom version trained on BBC output.
  - cratermoon 3y ago
    Emily Bender wrote, "If their output seems to make sense, it's because we make sense of it" <https://dair-community.social/@emilymbender/111464013178920266 https://dair-community.social/@emilymbender/1114640131789202...>
    - xanderlewis 3y ago
      I agree in many cases (particularly when it comes to asking it mathematical questions that require real understanding, for example), but for translation I think it’s genuinely hugely impressive. I’ve tried translating between Japanese and English for years using various tools and nothing else comes close. It may fail to understand almost everything else about the ‘real’ world, but it gets language and clearly has learnt very good mappings between languages. The mappings are also impressively stable under all sorts of meaning-preserving transformations and accurately preserve the tone / level of politeness / etc. Contrary to what many seem to believe, I don’t believe it can write that well. It’s perfectly grammatical and hard to stump, but it’s very bland and even after a lot of prompting to do better it still writes in a stilted ChatGPTish kind of way. (Caveat: I’m not sure how much of this is down to limitations of the architecture/data or just the amount of so-called ‘lobotomising’ OpenAI has done to it after initial training.) …but if you start with good source material and ask it to manipulate it, it’s amazing. For individual sentences with not much context, it’s already far better than any human. No particular surprise — it’s a language model after all.
      - cratermoon 3y ago
        > It’s perfectly grammatical and hard to stump, but it’s very bland and even after a lot of prompting to do better it still writes in a stilted ChatGPTish kind of way. That matches the experience of others who have experimented with it. From this thread, https://mastodon.world/@jfballenger/110686620108242354 https://mastodon.world/@jfballenger/110686620108242354 students noted the "relatively superficial and simplistic arguments that ChatGPT made"
ggm 3y ago
This is the kind of thing which has longterm repercussions. Aside from the specific contextual impact on perception of this story, The exposure of this goes to the trust in the public broadcaster. If the BBC minimises this, it's indicative of a lack of wider concern to their role. This worries me. I don't want a "we've fixed it" one liner, I want to understand how it happened, and how they intend preventing it happening in the future.

7 points by vinnyglennon 3y ago

cratermoon 3y ago

I wonder if they used ChatGPT for the translation.

xanderlewis 3y ago
ChatGPT is actually very, very good at translation in my experience. I’ve been quite shocked by how good it is.
- jjgreen 3y ago
  They use a custom version trained on BBC output.
- cratermoon 3y ago
  Emily Bender wrote, "If their output seems to make sense, it's because we make sense of it" <https://dair-community.social/@emilymbender/111464013178920266 https://dair-community.social/@emilymbender/1114640131789202...>
  - xanderlewis 3y ago
    I agree in many cases (particularly when it comes to asking it mathematical questions that require real understanding, for example), but for translation I think it’s genuinely hugely impressive. I’ve tried translating between Japanese and English for years using various tools and nothing else comes close. It may fail to understand almost everything else about the ‘real’ world, but it gets language and clearly has learnt very good mappings between languages. The mappings are also impressively stable under all sorts of meaning-preserving transformations and accurately preserve the tone / level of politeness / etc. Contrary to what many seem to believe, I don’t believe it can write that well. It’s perfectly grammatical and hard to stump, but it’s very bland and even after a lot of prompting to do better it still writes in a stilted ChatGPTish kind of way. (Caveat: I’m not sure how much of this is down to limitations of the architecture/data or just the amount of so-called ‘lobotomising’ OpenAI has done to it after initial training.) …but if you start with good source material and ask it to manipulate it, it’s amazing. For individual sentences with not much context, it’s already far better than any human. No particular surprise — it’s a language model after all.
    - cratermoon 3y ago
      > It’s perfectly grammatical and hard to stump, but it’s very bland and even after a lot of prompting to do better it still writes in a stilted ChatGPTish kind of way. That matches the experience of others who have experimented with it. From this thread, https://mastodon.world/@jfballenger/110686620108242354 https://mastodon.world/@jfballenger/110686620108242354 students noted the "relatively superficial and simplistic arguments that ChatGPT made"

ggm 3y ago

This is the kind of thing which has longterm repercussions. Aside from the specific contextual impact on perception of this story, The exposure of this goes to the trust in the public broadcaster. If the BBC minimises this, it's indicative of a lack of wider concern to their role. This worries me. I don't want a "we've fixed it" one liner, I want to understand how it happened, and how they intend preventing it happening in the future.