I agree in many cases (particularly when it comes to asking it mathematical questions that require real understanding, for example), but for translation I think

I agree in many cases (particularly when it comes to asking it mathematical questions that require real understanding, for example), but for translation I think it’s genuinely hugely impressive. I’ve tried translating between Japanese and English for years using various tools and nothing else comes close. It may fail to understand almost everything else about the ‘real’ world, but it gets language and clearly has learnt very good mappings between languages. The mappings are also impressively stable under all sorts of meaning-preserving transformations and accurately preserve the tone / level of politeness / etc. Contrary to what many seem to believe, I don’t believe it can write that well. It’s perfectly grammatical and hard to stump, but it’s very bland and even after a lot of prompting to do better it still writes in a stilted ChatGPTish kind of way. (Caveat: I’m not sure how much of this is down to limitations of the architecture/data or just the amount of so-called ‘lobotomising’ OpenAI has done to it after initial training.) …but if you start with good source material and ask it to manipulate it, it’s amazing. For individual sentences with not much context, it’s already far better than any human. No particular surprise — it’s a language model after all.