7 ms·

These tools are exactly why ASII is not dead. Unicode is ideal for storing text exactly as the user wanted it. Which may be crazy. But when I am writing a pro

These tools are exactly why ASII is not dead. Unicode is ideal for storing text exactly as the user wanted it. Which may be crazy. But when I am writing a program for internal use, or creating a communication protocol, or trying to parse one program's output into another, or trying to make a 7-segment display show some characters, I don't want to have to handle these crazy possibilities. Just text is fine, thank you!

teach 11y ago
I'd argue that there's no meaningful difference between ASCII text and UTF-8 that just happens to not use any characters higher than '~'. As long as there's no BOM, anyway.
- deleted 11y ago
  [deleted]
- dghf 11y ago
  s/meaningful// Any ASCII text is ipso facto UTF-8 text.
  - jstimpfle 11y ago
    If you consider representation, yes. If you consider semantics, no.
    - dghf 11y ago
      Can you explain? UTF-8 was deliberately designed to be "backwards compatible" (for want of a better phrase) with ASCII. Absent a BOM mark, there is no way to tell the difference between a piece of ASCII text on the one hand and, on the other, a piece of UTF-8 text that uses no code point above U+007f. How do they differ semantically?
      - jstimpfle 11y ago
        Yes. I already tried to explain with an analogy about faithfulness (which I removed because it got many downvotes). When you deal with ASCII, that's also the semantic level on which you operate. Much simpler than Unicode (I don't think there are many people who really understand Unicode). When you have a Unicode string that uses only characters that are also present in ASCII, that doesn't make it an ASCII string. It has more meaning. Btw that has nothing to do with BOM. BOM is itself not part of ASCII, and not encoded as a single byte, and it doesn't affect single-byte sequences.
        dghf 11y ago
        But wherein does that meaning lie? If the two strings represent the same text, you can't tell the difference between them, because there is none. (I guess technically you can't tell the difference between an ASCII string and a string of random 8-bit numbers in the range 0 <= x <= 127, but if when represented as ASCII characters those numbers are meaningful -- comprehensible English, for example -- the odds are good it's the former not the latter.)
        technofire 11y ago
        I think what he's getting at is that whatever program is reading the text must be prepared to read UTF-8 and not just ASCII. Sure, the underlying data is the same, but the data type is not the same; analogously, a byte with the decimal value of 65 might be the number 65 if cast as an integer, but might be the character 'A' if cast as a char.
        emddudley 11y ago
        Are you talking about how, when someone is limited to ASCII, he makes substitutions he wouldn't make if he were using Unicode? Such as using :-) instead of U+263A? Or are you maybe talking at a software level, where you can make some simplifications and assumptions about a string when it is guaranteed to be ASCII? Because otherwise I can't see how there could be any semantic or meaningful difference between ASCII and Unicode when the text is the same. An ASCII 'a' has exactly the same meaning as a Unicode 'Latin Small Letter A'.
vmorgulis 11y ago
> Just text is fine, thank you! And Unicode has a non-negligible footprint. I tried few times to handle that in C or C++ and everything becomes complicated (even with a library like ICU). Each time,I had to ask myself: - What's the encoding ? - How to detect it ? - What's the collation ? - ... I read Java strings are switching to ASCII (or 16b) because it's too much inefficient.
- jstimpfle 11y ago
  I think most software (like programming languages) has actually switched to UTF-8 for internal representation, because it has low footprint (for most applications). You give up efficient O(1) indexing, but I don't think you typically need that. Indexing is far less meaningful for Unicode compared to ASCII.
  - vmorgulis 11y ago
    > ... but I don't think you typically need that ... Yes. String processing often occurs in a loop (like for trim, contains, split ...).
    - jstimpfle 11y ago
      trim, contains, split (at whitespace, or at a given byte position) can all be implemented efficiently in UTF-8.
- ygra 11y ago
  > Each time,I had to ask myself: - What's the encoding ? - How to detect it ? - What's the collation ? You always have to ask yourself that if you deal with text, otherwise your application will most likely be broken. Heck, Unicode actually made this easier because you can use it as an internal encoding to convert everything else into. Yes, Unicode has overhead and complexities, but these are because text and language is complicated.
- lmm 11y ago
  If you're dealing with text from outside you have to handle encodings whether you like it or not. (I mean you can blindly copy bytes if you like, but you end up with corrupt data that you have no idea what it means, and may ultimately have to spend months or years recovering that information. I'm speaking from experience here). Internally it very rarely matters what representation you use as long as you know what it is - pick one particular representation to use for all your internal functions and stick to it and you're fine. You shouldn't need to do any detection except at the boundaries, and efficiency concerns are usually overblown.
gjm11 11y ago
For internal use, OK, fine. But what part of the following is not "just text"? "Don’t be naïve. £100 is a lot more than €100—and Hélène knows you know that."
- TazeTSchnitzel 11y ago
  While you can't represent that as plain ASCII, I think Windows-1252, an 8-bit single-byte encoding, can handle that quite well.
  - pekk 11y ago
    I guess we can get some arbitrary text, then check whether all its characters can be encoded as Windows-1252, store it with a note that it is in Windows-1252, and then check that note before decoding it. Clearly so much better than UTF-8.
    - TazeTSchnitzel 11y ago
      They tried: https://en.wikipedia.org/wiki/ISO/IEC_2022 https://en.wikipedia.org/wiki/ISO/IEC_2022 It was... horrible.
  - masklinn 11y ago
    Sure I mean who wants to visit Госуда́рственный Эрмита́ж or eat 삼겹살? Moonspeak's not really text right?
  - jquast 11y ago
    There is an art in discovering which encoding the text vs. represented by application, by making sense of nonsense. https://jeffquast.com/soviet-encodings.jpg https://jeffquast.com/soviet-encodings.jpg
  - kstrauser 11y ago
    That's an excellent point. Let's generalize that Windows-centric idea of a code page to something that can handle most/all known languages: a Universal Coded character set, if you will.
masklinn 11y ago
> Just text is fine, thank you! Unicode is text, that's the whole point. "ᐃᓄᐃᑦ ᐅᖃᐅᓯᓕᕆᔨᐅᑉ. ᐃᓄᑦᑎᑑᖅᐳᑦ ᓄᓇᕗᒻᒥ, ᓄᓇᕕᒻᒥ, ᐊᑯᑭᑦᑐᒻᒥ, ᓄᓇᑦᓯᐊᕗᒻᒥ, ᓄᓇᑦᓯᐊᕐᒥ' ᐊᒻᒪᓗ ᐊᓛᓯᑲᒥ" is text, "한글 또는 조선글은 1443년 조선 제4대 임금 세종이 훈민정음(訓民正音)이라는 이름으로 창제하여 1446년에 반포한 문자로, 한국어를 표기하기 위해 만들어졌다.[1][2] 이후 한문을 고수하는 사대부들에게는 경시되기도 하였으나, 조선 왕실과 일부 양반층과 서민층을 중심으로 이어지다가 1894년 갑오개혁에서 한국의 공식적인 나라 글자가 되었고, 1910년대에 이르러 한글학자인 주시경이 '한글'이라는 이름을 사용하였다. 갈래는 표음 문자 가운데 음소 문자에 속한다. 한국에서는 한글전용법이 시행되고 있다" is text, "દેવનાગરી એક લિપિ છે. સંસ્કૃત દેવનાગરી લિપિમાં લખાતી આવી છે. દેવનાગરી લિપિ મૂળ તો સંસ્કૃત માટે જ બની છે, એટલે એમાં દરેક ચિન્હ માટે એક અને માત્ર એક જ ધ્વનિ છે. દેવનાગરીમાં ૧૨ સ્વર અને ૩૪ વ્યંજન છે" is text, and "Алфавиты на основе кириллицы являются или являлись системой письменности для 108 естественных языков, включая следующие славянские языки" is text. It may be bothersome text I don't understand and you can't be arsed to handle, but it's text nonetheless.
- jstimpfle 11y ago
  You know, if you give me some ASCII I have a pretty good idea how to interpret it. For example, I can easily verify that it has no control codes, or is "plain text". There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.). For Unicode, I don't even know where to look up how to do it. Then there are these special encodings in UTF-8 to shelter binary data which failed to decode. Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)? Then there are these misplaced things like BOM or strange switching codes which affect the interpretation of the following text. I don't even know how I can interpret a string slice in a meaningful way because of this. It's painful. Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction. Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side. So to wrap it up, yes it is absolutely legitimate to restrict yourself to ASCII (or opaque 8-bit data handled as a black box) because it gives so much more meaning to the things you do with it.
  - chimeracoder 11y ago
    > if you give me some ASCII I have a pretty good idea how to interpret it. For Unicode, I don't even know where to look up how to do it. That's not because Unicode is not "just text" whereas ASCII is. It's because "text" is genuinely very complicated and nuanced, and that's the nature of the problem domain. Text is messy. Text is complicated. Text is many things, but text is not "plain". We should also be careful to distinguish between criticisms of Unicode and criticisms of Unicode encodings. They're related, but not all encodings share the same problems (and some problems are baked into Unicode itself, and can't be solved by a novel encoding). (I have plenty of complaints about Unicode and opinions about the various Unicode encodings, but the fact remains that there is no one-size-fits-all approach to text which can completely avoid or hide this complexity.

teach 11y ago

I'd argue that there's no meaningful difference between ASCII text and UTF-8 that just happens to not use any characters higher than '~'. As long as there's no BOM, anyway.

deleted 11y ago
[deleted]
dghf 11y ago
s/meaningful// Any ASCII text is ipso facto UTF-8 text.
- jstimpfle 11y ago
  If you consider representation, yes. If you consider semantics, no.
  - dghf 11y ago
    Can you explain? UTF-8 was deliberately designed to be "backwards compatible" (for want of a better phrase) with ASCII. Absent a BOM mark, there is no way to tell the difference between a piece of ASCII text on the one hand and, on the other, a piece of UTF-8 text that uses no code point above U+007f. How do they differ semantically?
    - jstimpfle 11y ago
      Yes. I already tried to explain with an analogy about faithfulness (which I removed because it got many downvotes). When you deal with ASCII, that's also the semantic level on which you operate. Much simpler than Unicode (I don't think there are many people who really understand Unicode). When you have a Unicode string that uses only characters that are also present in ASCII, that doesn't make it an ASCII string. It has more meaning. Btw that has nothing to do with BOM. BOM is itself not part of ASCII, and not encoded as a single byte, and it doesn't affect single-byte sequences.
      - dghf 11y ago
        But wherein does that meaning lie? If the two strings represent the same text, you can't tell the difference between them, because there is none. (I guess technically you can't tell the difference between an ASCII string and a string of random 8-bit numbers in the range 0 <= x <= 127, but if when represented as ASCII characters those numbers are meaningful -- comprehensible English, for example -- the odds are good it's the former not the latter.)
        technofire 11y ago
        I think what he's getting at is that whatever program is reading the text must be prepared to read UTF-8 and not just ASCII. Sure, the underlying data is the same, but the data type is not the same; analogously, a byte with the decimal value of 65 might be the number 65 if cast as an integer, but might be the character 'A' if cast as a char.
      - emddudley 11y ago
        Are you talking about how, when someone is limited to ASCII, he makes substitutions he wouldn't make if he were using Unicode? Such as using :-) instead of U+263A? Or are you maybe talking at a software level, where you can make some simplifications and assumptions about a string when it is guaranteed to be ASCII? Because otherwise I can't see how there could be any semantic or meaningful difference between ASCII and Unicode when the text is the same. An ASCII 'a' has exactly the same meaning as a Unicode 'Latin Small Letter A'.

vmorgulis 11y ago

> Just text is fine, thank you! And Unicode has a non-negligible footprint. I tried few times to handle that in C or C++ and everything becomes complicated (even with a library like ICU). Each time,I had to ask myself: - What's the encoding ? - How to detect it ? - What's the collation ? - ... I read Java strings are switching to ASCII (or 16b) because it's too much inefficient.

jstimpfle 11y ago
I think most software (like programming languages) has actually switched to UTF-8 for internal representation, because it has low footprint (for most applications). You give up efficient O(1) indexing, but I don't think you typically need that. Indexing is far less meaningful for Unicode compared to ASCII.
- vmorgulis 11y ago
  > ... but I don't think you typically need that ... Yes. String processing often occurs in a loop (like for trim, contains, split ...).
  - jstimpfle 11y ago
    trim, contains, split (at whitespace, or at a given byte position) can all be implemented efficiently in UTF-8.
ygra 11y ago
> Each time,I had to ask myself: - What's the encoding ? - How to detect it ? - What's the collation ? You always have to ask yourself that if you deal with text, otherwise your application will most likely be broken. Heck, Unicode actually made this easier because you can use it as an internal encoding to convert everything else into. Yes, Unicode has overhead and complexities, but these are because text and language is complicated.
lmm 11y ago
If you're dealing with text from outside you have to handle encodings whether you like it or not. (I mean you can blindly copy bytes if you like, but you end up with corrupt data that you have no idea what it means, and may ultimately have to spend months or years recovering that information. I'm speaking from experience here). Internally it very rarely matters what representation you use as long as you know what it is - pick one particular representation to use for all your internal functions and stick to it and you're fine. You shouldn't need to do any detection except at the boundaries, and efficiency concerns are usually overblown.

gjm11 11y ago

For internal use, OK, fine. But what part of the following is not "just text"? "Don’t be naïve. £100 is a lot more than €100—and Hélène knows you know that."

TazeTSchnitzel 11y ago
While you can't represent that as plain ASCII, I think Windows-1252, an 8-bit single-byte encoding, can handle that quite well.
- pekk 11y ago
  I guess we can get some arbitrary text, then check whether all its characters can be encoded as Windows-1252, store it with a note that it is in Windows-1252, and then check that note before decoding it. Clearly so much better than UTF-8.
  - TazeTSchnitzel 11y ago
    They tried: https://en.wikipedia.org/wiki/ISO/IEC_2022 https://en.wikipedia.org/wiki/ISO/IEC_2022 It was... horrible.
- masklinn 11y ago
  Sure I mean who wants to visit Госуда́рственный Эрмита́ж or eat 삼겹살? Moonspeak's not really text right?
- jquast 11y ago
  There is an art in discovering which encoding the text vs. represented by application, by making sense of nonsense. https://jeffquast.com/soviet-encodings.jpg https://jeffquast.com/soviet-encodings.jpg
- kstrauser 11y ago
  That's an excellent point. Let's generalize that Windows-centric idea of a code page to something that can handle most/all known languages: a Universal Coded character set, if you will.

masklinn 11y ago

> Just text is fine, thank you! Unicode is text, that's the whole point. "ᐃᓄᐃᑦ ᐅᖃᐅᓯᓕᕆᔨᐅᑉ. ᐃᓄᑦᑎᑑᖅᐳᑦ ᓄᓇᕗᒻᒥ, ᓄᓇᕕᒻᒥ, ᐊᑯᑭᑦᑐᒻᒥ, ᓄᓇᑦᓯᐊᕗᒻᒥ, ᓄᓇᑦᓯᐊᕐᒥ' ᐊᒻᒪᓗ ᐊᓛᓯᑲᒥ" is text, "한글 또는 조선글은 1443년 조선 제4대 임금 세종이 훈민정음(訓民正音)이라는 이름으로 창제하여 1446년에 반포한 문자로, 한국어를 표기하기 위해 만들어졌다.[1][2] 이후 한문을 고수하는 사대부들에게는 경시되기도 하였으나, 조선 왕실과 일부 양반층과 서민층을 중심으로 이어지다가 1894년 갑오개혁에서 한국의 공식적인 나라 글자가 되었고, 1910년대에 이르러 한글학자인 주시경이 '한글'이라는 이름을 사용하였다. 갈래는 표음 문자 가운데 음소 문자에 속한다. 한국에서는 한글전용법이 시행되고 있다" is text, "દેવનાગરી એક લિપિ છે. સંસ્કૃત દેવનાગરી લિપિમાં લખાતી આવી છે. દેવનાગરી લિપિ મૂળ તો સંસ્કૃત માટે જ બની છે, એટલે એમાં દરેક ચિન્હ માટે એક અને માત્ર એક જ ધ્વનિ છે. દેવનાગરીમાં ૧૨ સ્વર અને ૩૪ વ્યંજન છે" is text, and "Алфавиты на основе кириллицы являются или являлись системой письменности для 108 естественных языков, включая следующие славянские языки" is text. It may be bothersome text I don't understand and you can't be arsed to handle, but it's text nonetheless.

jstimpfle 11y ago
You know, if you give me some ASCII I have a pretty good idea how to interpret it. For example, I can easily verify that it has no control codes, or is "plain text". There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.). For Unicode, I don't even know where to look up how to do it. Then there are these special encodings in UTF-8 to shelter binary data which failed to decode. Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)? Then there are these misplaced things like BOM or strange switching codes which affect the interpretation of the following text. I don't even know how I can interpret a string slice in a meaningful way because of this. It's painful. Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction. Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side. So to wrap it up, yes it is absolutely legitimate to restrict yourself to ASCII (or opaque 8-bit data handled as a black box) because it gives so much more meaning to the things you do with it.
- chimeracoder 11y ago
  > if you give me some ASCII I have a pretty good idea how to interpret it. For Unicode, I don't even know where to look up how to do it. That's not because Unicode is not "just text" whereas ASCII is. It's because "text" is genuinely very complicated and nuanced, and that's the nature of the problem domain. Text is messy. Text is complicated. Text is many things, but text is not "plain". We should also be careful to distinguish between criticisms of Unicode and criticisms of Unicode encodings. They're related, but not all encodings share the same problems (and some problems are baked into Unicode itself, and can't be solved by a novel encoding). (I have plenty of complaints about Unicode and opinions about the various Unicode encodings, but the fact remains that there is no one-size-fits-all approach to text which can completely avoid or hide this complexity.