I think this was the right decision for the long run though. What we have now is the right situation where the strings we work with in the language are 'magic'

I think this was the right decision for the long run though. What we have now is the right situation where the strings we work with in the language are 'magic' and we don't need to think about representation or encoding or whatever. And then at the periphery we encode/decode based on what the outside world is working on. It has become clear that this is the 'correct' approach for programming languages and is almost universally the approach in new languages. E.g. Rust does something similar (they don't hide that the internal encoding is UTF-8, but it's consistent). I'm glad that this is what we have for Python now, despite the pain it caused. We'll be using Python for another few decades probably, so we'll reap the benefits.