4 ms·

> Any significant length of text will be compressed and/or have lots of latin characters. A reasonable counterargument is Asian books on Project Gutenberg [0].

> Any significant length of text will be compressed and/or have lots of latin characters. A reasonable counterargument is Asian books on Project Gutenberg [0]. I guess you can say they're compressed on the wire, but they're not compressed in my RAM or cache. >> you definitely can't store UTF-8 in something expecting ASCII > That's not true. Well alright, you can jam bytes pretty much into anything. And you get luckier with UTF-8 than you do with probably any other encoding. But lucky isn't robust, is all I'm saying. [0]: https://www.gutenberg.org/browse/languages/zh https://www.gutenberg.org/browse/languages/zh

Dylan16807 5y ago
> I guess you can say they're compressed on the wire, but they're not compressed in my RAM or cache. You don't really need to have very much of a book decompressed at once, and in your average program handling that book there's going to be much more RAM dedicated to layout and display than the actual character bytes. > Well alright, you can jam bytes pretty much into anything. And you get luckier with UTF-8 than you do with probably any other encoding. But lucky isn't robust, is all I'm saying. It won't work in all systems but it'll work in most of them. And you can check beforehand.
- camgunz 5y ago
  In truth I think these are mostly arguments for letting apps/users pick their encoding. If GBK works for your app and user base, go for it. If UTF-8 ends up being more efficient, have at it. We can sit around hypothesizing about a book reading app or a grammar analyzing app or blah blah blah, but I'm pretty sure UTF-8 won't be the best encoding for every circumstance. Python at least used to let you do this at compile time, for example.
  - Dylan16807 5y ago
    Allowing choice is a big overhead with almost no benefit. It's not worth that little bit of optimization in certain circumstances.
    - camgunz 5y ago
      Honestly, not to get too high-roady here, I feel like we shouldn't be in the business of not "allowing choice" unless there's a very compelling reason, and "I'm against OOB encoding info" isn't compelling, at least not to me. Should we do this for all file types? Any kind of document or image? Everything's XML, sure it's not optimal, but hey at least we can define any image format we want in it. I don't know why we treat text encodings any differently than any other binary format. We have OOB format info for all of those. People are just downright religious about UTF-8, and it's my goal in these threads to push (hopefully gently) against that. UTF-8 isn't always the best, you always have to validate, you shouldn't be indexing into strings, we should have OOB encoding info (whether in a filesystem config or whatever). This is the only possible path to robust handling of encodings: "always assume UTF-8 and hope non-UTF-8 systems get updated or fall into disuse" is a recipe for dealing with broken systems for decades (what we're doing now).
      - Dylan16807 5y ago
        The first thing I think of there is browsing the web. Different image formats have different use cases and substantial advantages over each other. And it's often impossible to convert without losing quality. But everything gets to be HTML. Or the compatible variant of XHTML. Nothing else has any real support, and when it did, like flash web pages, that was not good. For text unicode wins out easily, with UTF-8 and UTF-16 far above the other encodings, and the differences are minor enough that we should just use the better one. > we should have OOB encoding info (whether in a filesystem config or whatever). This is the only possible path to robust handling of encoding I don't know about that. Old files, which represent almost all the non-unicode non-ascii text we have, are unlikely to ever be properly tagged. For new files, are those tags really needed? If you have a firewall where you mandate tagging behind it, you're already mandating a format for attaching tags, so why not mandate utf-8 instead?

Dylan16807 5y ago

> I guess you can say they're compressed on the wire, but they're not compressed in my RAM or cache. You don't really need to have very much of a book decompressed at once, and in your average program handling that book there's going to be much more RAM dedicated to layout and display than the actual character bytes. > Well alright, you can jam bytes pretty much into anything. And you get luckier with UTF-8 than you do with probably any other encoding. But lucky isn't robust, is all I'm saying. It won't work in all systems but it'll work in most of them. And you can check beforehand.

camgunz 5y ago
In truth I think these are mostly arguments for letting apps/users pick their encoding. If GBK works for your app and user base, go for it. If UTF-8 ends up being more efficient, have at it. We can sit around hypothesizing about a book reading app or a grammar analyzing app or blah blah blah, but I'm pretty sure UTF-8 won't be the best encoding for every circumstance. Python at least used to let you do this at compile time, for example.
- Dylan16807 5y ago
  Allowing choice is a big overhead with almost no benefit. It's not worth that little bit of optimization in certain circumstances.
  - camgunz 5y ago
    Honestly, not to get too high-roady here, I feel like we shouldn't be in the business of not "allowing choice" unless there's a very compelling reason, and "I'm against OOB encoding info" isn't compelling, at least not to me. Should we do this for all file types? Any kind of document or image? Everything's XML, sure it's not optimal, but hey at least we can define any image format we want in it. I don't know why we treat text encodings any differently than any other binary format. We have OOB format info for all of those. People are just downright religious about UTF-8, and it's my goal in these threads to push (hopefully gently) against that. UTF-8 isn't always the best, you always have to validate, you shouldn't be indexing into strings, we should have OOB encoding info (whether in a filesystem config or whatever). This is the only possible path to robust handling of encodings: "always assume UTF-8 and hope non-UTF-8 systems get updated or fall into disuse" is a recipe for dealing with broken systems for decades (what we're doing now).
    - Dylan16807 5y ago
      The first thing I think of there is browsing the web. Different image formats have different use cases and substantial advantages over each other. And it's often impossible to convert without losing quality. But everything gets to be HTML. Or the compatible variant of XHTML. Nothing else has any real support, and when it did, like flash web pages, that was not good. For text unicode wins out easily, with UTF-8 and UTF-16 far above the other encodings, and the differences are minor enough that we should just use the better one. > we should have OOB encoding info (whether in a filesystem config or whatever). This is the only possible path to robust handling of encoding I don't know about that. Old files, which represent almost all the non-unicode non-ascii text we have, are unlikely to ever be properly tagged. For new files, are those tags really needed? If you have a firewall where you mandate tagging behind it, you're already mandating a format for attaching tags, so why not mandate utf-8 instead?