3 ms·

You know, if you give me some ASCII I have a pretty good idea how to interpret it. For example, I can easily verify that it has no control codes, or is "plain

You know, if you give me some ASCII I have a pretty good idea how to interpret it. For example, I can easily verify that it has no control codes, or is "plain text". There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.). For Unicode, I don't even know where to look up how to do it. Then there are these special encodings in UTF-8 to shelter binary data which failed to decode. Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)? Then there are these misplaced things like BOM or strange switching codes which affect the interpretation of the following text. I don't even know how I can interpret a string slice in a meaningful way because of this. It's painful. Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction. Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side. So to wrap it up, yes it is absolutely legitimate to restrict yourself to ASCII (or opaque 8-bit data handled as a black box) because it gives so much more meaning to the things you do with it.

chimeracoder 11y ago
> if you give me some ASCII I have a pretty good idea how to interpret it. For Unicode, I don't even know where to look up how to do it. That's not because Unicode is not "just text" whereas ASCII is. It's because "text" is genuinely very complicated and nuanced, and that's the nature of the problem domain. Text is messy. Text is complicated. Text is many things, but text is not "plain". We should also be careful to distinguish between criticisms of Unicode and criticisms of Unicode encodings. They're related, but not all encodings share the same problems (and some problems are baked into Unicode itself, and can't be solved by a novel encoding). (I have plenty of complaints about Unicode and opinions about the various Unicode encodings, but the fact remains that there is no one-size-fits-all approach to text which can completely avoid or hide this complexity.
- jstimpfle 11y ago
  I totally agree that text is complicated. But that just means it's a black box and I can't do anything meaningful to it other than copying it around. Unless I read the Unicode spec and expect my inputs to also have read it.
  - chimeracoder 11y ago
    > But that just means it's a black box and I can't do anything meaningful to it other than copying it around. It means you need some out-of-band context about the domain. The things I can expect do with text if I know I'm parsing C source code is very different from the things I can expect to do with text if I'm parsing Twitter data (especially if the tweets are geotagged to a region that doesn't speak English). Or, if I'm parsing SMS message data (which are likely to contain lots of Unicode emoji). And quite frankly, pretty much any task involving processing arbitrary input text already has so many complicating factors that have nothing to do with Unicode. There's already pretty much nothing interesting you can do to arbitrary text besides copy it around if all you know is the character encoding. Every real-life problem space has more concrete assumptions about the data you're dealing with, so including assumptions about the character encoding and characters used in that set is not terribly burdensome.
    - jstimpfle 11y ago
      Absolutely. And my go-to context is ASCII whenever it's convenient. This is the only context where I have a clear idea what's inside it.
masklinn 11y ago
> You know, if you give me some ASCII I have a pretty good idea how to interpret it. > For example, I can easily verify that it has no control codes, or is "plain text". > There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.). > For Unicode, I don't even know where to look up how to do it. That's not a problem with unicode. > Then there are these special encodings in UTF-8 to shelter binary data which failed to decode. There are no special encodings in UTF-8. What you're talking about is error-handling strategies > Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)? The exact same question exists about ASCII, since \xFF is not valid ASCII. > Then there are these misplaced things like BOM or strange switching codes which affect the interpretation of the following text. I don't even know how I can interpret a string slice in a meaningful way because of this. It's painful. It's painful that your random manglings don't make sense? > Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction. They don't. > Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side. Which is already an issue in ASCII. > So to wrap it up, yes it is absolutely legitimate to restrict yourself to ASCII It is, for low values of legitimate. But don't call it text, and more importantly don't call non-ascii content "not text" which is what the person I replied to did. And if you think dealing with unicode is troublesome, for your sake stay far away from anything having to do with localisation, unicode is the basement level of localisation issues.
- PeCaN 11y ago
  Sigh, thank you for ranting for me. I just want to add to what you said: > > There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.). > > For Unicode, I don't even know where to look up how to do it. UAX #14 and #29 specify perfectly well how to do this[1][2]. There are even C libraries to do this for you[3][4]. In general, if you have a Unicode question, the standard clears things up reasonably well. It's somewhat easy reading for a standardization document. 1. http://www.unicode.org/reports/tr14/ http://www.unicode.org/reports/tr14/ 2. http://unicode.org/reports/tr29/ http://unicode.org/reports/tr29/ 3. http://site.icu-project.org/ http://site.icu-project.org/ 4. http://julialang.org/utf8proc/ http://julialang.org/utf8proc/ (Not Julia specific; it's used to implement Unicode support in Julia and is maintained as part of the Julia project. It's a very nice clean C library.)
GFK_of_xmaspast 11y ago
Have you considered that perhaps the fault is not in your stars but in yourself? (That is, a thing is not inherently negative just because you do not care to understand it.)
- jstimpfle 11y ago
  For some context, read the top level OP. Of course it's not inherently negative.

chimeracoder 11y ago

> if you give me some ASCII I have a pretty good idea how to interpret it. For Unicode, I don't even know where to look up how to do it. That's not because Unicode is not "just text" whereas ASCII is. It's because "text" is genuinely very complicated and nuanced, and that's the nature of the problem domain. Text is messy. Text is complicated. Text is many things, but text is not "plain". We should also be careful to distinguish between criticisms of Unicode and criticisms of Unicode encodings. They're related, but not all encodings share the same problems (and some problems are baked into Unicode itself, and can't be solved by a novel encoding). (I have plenty of complaints about Unicode and opinions about the various Unicode encodings, but the fact remains that there is no one-size-fits-all approach to text which can completely avoid or hide this complexity.

jstimpfle 11y ago
I totally agree that text is complicated. But that just means it's a black box and I can't do anything meaningful to it other than copying it around. Unless I read the Unicode spec and expect my inputs to also have read it.
- chimeracoder 11y ago
  > But that just means it's a black box and I can't do anything meaningful to it other than copying it around. It means you need some out-of-band context about the domain. The things I can expect do with text if I know I'm parsing C source code is very different from the things I can expect to do with text if I'm parsing Twitter data (especially if the tweets are geotagged to a region that doesn't speak English). Or, if I'm parsing SMS message data (which are likely to contain lots of Unicode emoji). And quite frankly, pretty much any task involving processing arbitrary input text already has so many complicating factors that have nothing to do with Unicode. There's already pretty much nothing interesting you can do to arbitrary text besides copy it around if all you know is the character encoding. Every real-life problem space has more concrete assumptions about the data you're dealing with, so including assumptions about the character encoding and characters used in that set is not terribly burdensome.
  - jstimpfle 11y ago
    Absolutely. And my go-to context is ASCII whenever it's convenient. This is the only context where I have a clear idea what's inside it.

masklinn 11y ago

> You know, if you give me some ASCII I have a pretty good idea how to interpret it. > For example, I can easily verify that it has no control codes, or is "plain text". > There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.). > For Unicode, I don't even know where to look up how to do it. That's not a problem with unicode. > Then there are these special encodings in UTF-8 to shelter binary data which failed to decode. There are no special encodings in UTF-8. What you're talking about is error-handling strategies > Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)? The exact same question exists about ASCII, since \xFF is not valid ASCII. > Then there are these misplaced things like BOM or strange switching codes which affect the interpretation of the following text. I don't even know how I can interpret a string slice in a meaningful way because of this. It's painful. It's painful that your random manglings don't make sense? > Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction. They don't. > Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side. Which is already an issue in ASCII. > So to wrap it up, yes it is absolutely legitimate to restrict yourself to ASCII It is, for low values of legitimate. But don't call it text, and more importantly don't call non-ascii content "not text" which is what the person I replied to did. And if you think dealing with unicode is troublesome, for your sake stay far away from anything having to do with localisation, unicode is the basement level of localisation issues.

PeCaN 11y ago
Sigh, thank you for ranting for me. I just want to add to what you said: > > There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.). > > For Unicode, I don't even know where to look up how to do it. UAX #14 and #29 specify perfectly well how to do this[1][2]. There are even C libraries to do this for you[3][4]. In general, if you have a Unicode question, the standard clears things up reasonably well. It's somewhat easy reading for a standardization document. 1. http://www.unicode.org/reports/tr14/ http://www.unicode.org/reports/tr14/ 2. http://unicode.org/reports/tr29/ http://unicode.org/reports/tr29/ 3. http://site.icu-project.org/ http://site.icu-project.org/ 4. http://julialang.org/utf8proc/ http://julialang.org/utf8proc/ (Not Julia specific; it's used to implement Unicode support in Julia and is maintained as part of the Julia project. It's a very nice clean C library.)

GFK_of_xmaspast 11y ago

Have you considered that perhaps the fault is not in your stars but in yourself? (That is, a thing is not inherently negative just because you do not care to understand it.)

jstimpfle 11y ago
For some context, read the top level OP. Of course it's not inherently negative.