2 ms·

In truth I think these are mostly arguments for letting apps/users pick their encoding. If GBK works for your app and user base, go for it. If UTF-8 ends up bei

In truth I think these are mostly arguments for letting apps/users pick their encoding. If GBK works for your app and user base, go for it. If UTF-8 ends up being more efficient, have at it. We can sit around hypothesizing about a book reading app or a grammar analyzing app or blah blah blah, but I'm pretty sure UTF-8 won't be the best encoding for every circumstance. Python at least used to let you do this at compile time, for example.

Dylan16807 5y ago
Allowing choice is a big overhead with almost no benefit. It's not worth that little bit of optimization in certain circumstances.
- camgunz 5y ago
  Honestly, not to get too high-roady here, I feel like we shouldn't be in the business of not "allowing choice" unless there's a very compelling reason, and "I'm against OOB encoding info" isn't compelling, at least not to me. Should we do this for all file types? Any kind of document or image? Everything's XML, sure it's not optimal, but hey at least we can define any image format we want in it. I don't know why we treat text encodings any differently than any other binary format. We have OOB format info for all of those. People are just downright religious about UTF-8, and it's my goal in these threads to push (hopefully gently) against that. UTF-8 isn't always the best, you always have to validate, you shouldn't be indexing into strings, we should have OOB encoding info (whether in a filesystem config or whatever). This is the only possible path to robust handling of encodings: "always assume UTF-8 and hope non-UTF-8 systems get updated or fall into disuse" is a recipe for dealing with broken systems for decades (what we're doing now).
  - Dylan16807 5y ago
    The first thing I think of there is browsing the web. Different image formats have different use cases and substantial advantages over each other. And it's often impossible to convert without losing quality. But everything gets to be HTML. Or the compatible variant of XHTML. Nothing else has any real support, and when it did, like flash web pages, that was not good. For text unicode wins out easily, with UTF-8 and UTF-16 far above the other encodings, and the differences are minor enough that we should just use the better one. > we should have OOB encoding info (whether in a filesystem config or whatever). This is the only possible path to robust handling of encoding I don't know about that. Old files, which represent almost all the non-unicode non-ascii text we have, are unlikely to ever be properly tagged. For new files, are those tags really needed? If you have a firewall where you mandate tagging behind it, you're already mandating a format for attaching tags, so why not mandate utf-8 instead?

Dylan16807 5y ago

Allowing choice is a big overhead with almost no benefit. It's not worth that little bit of optimization in certain circumstances.

camgunz 5y ago
Honestly, not to get too high-roady here, I feel like we shouldn't be in the business of not "allowing choice" unless there's a very compelling reason, and "I'm against OOB encoding info" isn't compelling, at least not to me. Should we do this for all file types? Any kind of document or image? Everything's XML, sure it's not optimal, but hey at least we can define any image format we want in it. I don't know why we treat text encodings any differently than any other binary format. We have OOB format info for all of those. People are just downright religious about UTF-8, and it's my goal in these threads to push (hopefully gently) against that. UTF-8 isn't always the best, you always have to validate, you shouldn't be indexing into strings, we should have OOB encoding info (whether in a filesystem config or whatever). This is the only possible path to robust handling of encodings: "always assume UTF-8 and hope non-UTF-8 systems get updated or fall into disuse" is a recipe for dealing with broken systems for decades (what we're doing now).
- Dylan16807 5y ago
  The first thing I think of there is browsing the web. Different image formats have different use cases and substantial advantages over each other. And it's often impossible to convert without losing quality. But everything gets to be HTML. Or the compatible variant of XHTML. Nothing else has any real support, and when it did, like flash web pages, that was not good. For text unicode wins out easily, with UTF-8 and UTF-16 far above the other encodings, and the differences are minor enough that we should just use the better one. > we should have OOB encoding info (whether in a filesystem config or whatever). This is the only possible path to robust handling of encoding I don't know about that. Old files, which represent almost all the non-unicode non-ascii text we have, are unlikely to ever be properly tagged. For new files, are those tags really needed? If you have a firewall where you mandate tagging behind it, you're already mandating a format for attaching tags, so why not mandate utf-8 instead?