3 ms·

> if you give me some ASCII I have a pretty good idea how to interpret it. For Unicode, I don't even know where to look up how to do it. That's not because Uni

> if you give me some ASCII I have a pretty good idea how to interpret it. For Unicode, I don't even know where to look up how to do it. That's not because Unicode is not "just text" whereas ASCII is. It's because "text" is genuinely very complicated and nuanced, and that's the nature of the problem domain. Text is messy. Text is complicated. Text is many things, but text is not "plain". We should also be careful to distinguish between criticisms of Unicode and criticisms of Unicode encodings. They're related, but not all encodings share the same problems (and some problems are baked into Unicode itself, and can't be solved by a novel encoding). (I have plenty of complaints about Unicode and opinions about the various Unicode encodings, but the fact remains that there is no one-size-fits-all approach to text which can completely avoid or hide this complexity.

jstimpfle 11y ago
I totally agree that text is complicated. But that just means it's a black box and I can't do anything meaningful to it other than copying it around. Unless I read the Unicode spec and expect my inputs to also have read it.
- chimeracoder 11y ago
  > But that just means it's a black box and I can't do anything meaningful to it other than copying it around. It means you need some out-of-band context about the domain. The things I can expect do with text if I know I'm parsing C source code is very different from the things I can expect to do with text if I'm parsing Twitter data (especially if the tweets are geotagged to a region that doesn't speak English). Or, if I'm parsing SMS message data (which are likely to contain lots of Unicode emoji). And quite frankly, pretty much any task involving processing arbitrary input text already has so many complicating factors that have nothing to do with Unicode. There's already pretty much nothing interesting you can do to arbitrary text besides copy it around if all you know is the character encoding. Every real-life problem space has more concrete assumptions about the data you're dealing with, so including assumptions about the character encoding and characters used in that set is not terribly burdensome.
  - jstimpfle 11y ago
    Absolutely. And my go-to context is ASCII whenever it's convenient. This is the only context where I have a clear idea what's inside it.

jstimpfle 11y ago

I totally agree that text is complicated. But that just means it's a black box and I can't do anything meaningful to it other than copying it around. Unless I read the Unicode spec and expect my inputs to also have read it.

chimeracoder 11y ago
> But that just means it's a black box and I can't do anything meaningful to it other than copying it around. It means you need some out-of-band context about the domain. The things I can expect do with text if I know I'm parsing C source code is very different from the things I can expect to do with text if I'm parsing Twitter data (especially if the tweets are geotagged to a region that doesn't speak English). Or, if I'm parsing SMS message data (which are likely to contain lots of Unicode emoji). And quite frankly, pretty much any task involving processing arbitrary input text already has so many complicating factors that have nothing to do with Unicode. There's already pretty much nothing interesting you can do to arbitrary text besides copy it around if all you know is the character encoding. Every real-life problem space has more concrete assumptions about the data you're dealing with, so including assumptions about the character encoding and characters used in that set is not terribly burdensome.
- jstimpfle 11y ago
  Absolutely. And my go-to context is ASCII whenever it's convenient. This is the only context where I have a clear idea what's inside it.