6 ms·

ASCII is by far the most successful character encoding that computers have used. It was invented in 1963, back in the era of punch cards and core memory. Mode

ASCII is by far the most successful character encoding that computers have used. It was invented in 1963, back in the era of punch cards and core memory. Modern RAM did not exist until 1975 -- a decade later. Unicode is the replacement, not the competitor, like 64-bit IP addresses are the replacement for 32-bit IP addresses. It was developed in the early 1990s when RAM got cheap enough that you could afford two-bytes per character. Personally, I deal with data all the time and rarely encounter unicode. Of course, I'm in the US dealing with big files out of financial and marketing databases. In fact, I've seen more EBCDIC than UNICODE.

chongli 13y ago
>64-bit IP addresses are the replacement for 32-bit IP addresses IPv6 addresses are 128-bit.
lotsofcows 13y ago
It's not even really a replacement. More of an extension. Anything that talks Unicode can talk ASCII (although not vice versa).
- qznc 13y ago
  Again the common mistake to confuse codes and encoding. ASCII is an encoding which can represent a very small subset of Unicode. UTF8 is an encoding which can represent all of Unicode and it is a superset of ASCII. UTF32 can represent all of Unicode, but is not a superset of ASCII. Therefore, something can talk Unicode via UTF32 without being able to talk ASCII.
  - tome 13y ago
    But the collection of Unicode codepoints is an extension of the ASCII codepoints, isn't it? (Regardless of how they are encoded).
    - ygra 13y ago
      Yes, but that's the distinction between character sets and encodings. A character set is just that, a set of characters. In this sense Unicode is a superset of ASCII, Latin 1, and every other pre-existing character set. Encodings map characters to byte( sequence)?s and they differ a lot in that respect. Practically both character sets and encodings prove ample opportunities to mangle input, either by replacing unavailable characters by ? or �, or by mapping to the wrong characters because the wrong encoding was assumed (or by throwing errors because the input made no sense).
      - tome 13y ago
        I mean when ASCII assigns a number to a character, Unicode assigns the same number to the character. UTF16 and UTF32 may encode those numbers differently from how ASCII encodes them, but they are nonetheless the same numbers. Unicode is not an extension of EDBIC or Latin1 because characters under them have different codepoints than they do under Unicode. [Please correct me if I'm wrong. I'm not an expert but this is how it seems to me]
        DougBTX 13y ago
        > [Please correct me if I'm wrong. I'm not an expert but this is how it seems to me] You're basically right, but the subtle point is that these comments are in response to lotsofcows who said, "Anything that talks Unicode can talk ASCII (although not vice versa)." The implication of "talk" is that the numbers are being encoded. qznc was trying to clarify that lotsofcows's statement is only true if the Unicode system is using an encoding which is backwards compatible with 7-bit ASCII, such as UTF-8.
        tome 13y ago
        Thanks, I see :) My comment wasn't particularly relevant.
        VLM 13y ago
        "Unicode assigns the same number to the character." Extremely close but not quite right for some weird corner cases. There's either a cool feature or hideous bug in the unicode design, depends how you look at it, that lets you compose multiple unicode codes together into one glyph. So you can say, "Gimme the code for A with a bar over it" or "Gimme the code for A, now gimme the code for put a bar over the last glyph". Even worse you can stack compose characters, so you can write "A with circle on top and squiggly underneath" at least five different ways in binary that should be rendered visually identical. There is the one true normalization technique that most people use to convert what they consider poor unicode grammar into a standard form. Some people don't use it of course. And anytime you give users freeform input who knows what kind of crud they'll feed you. So you can never really assume any unicode string is normalized unless you personally normalized it yourself. Even then theoretically two normalized strings, concatenated, might or might not be normalized anymore (although this is often not much of a problem). And this strikes substring manipulation too. A fun source of buffer overflows is normalization can shrink OR EXPAND a unicode string, in theory. So if you use one of those languages without variable length strings, look out. Or if the language tries to take care of it for you, this leads to weird memory fragmentation. Maybe a DOS/DDOS opportunity? Which leads to philosophical argument you must decide in your code, if two bitstreams don't match, but you get matching glyph renderings, from the program's point of view are they a match or not? Depends if what your program is trying to do I guess. Most languages have a library to handle this. Writing your own unicode handler functions is a "here be dragons" moment. Composing characters are also a fun source of swear words if you're trying to count the number of characters in a file. Something that renders literally identically will have an identical number of characters but somewhat varying number of bytes. Other fun philosophical arguments are if you read in a un-normalized string and output a normalized string have you changed the string? Well, both no, and yes. I'm sure there's crypto steagongraphy implications. Google for "unicode normalization form nfc" and stuff like that. The first page I found with a good faq was: http://www.macchiato.com/unicode/nfc-faq http://www.macchiato.com/unicode/nfc-faq
  - brudgers 13y ago
    Joel Spolsky provides the necessary (in Joel's opinion of 2003) minimum background in, The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) http://www.joelonsoftware.com/articles/Unicode.html http://www.joelonsoftware.com/articles/Unicode.html
VLM 13y ago
"Modern RAM did not exist until 1975 -- a decade later." What does that even mean? It doesn't mean DIP packaged DRAM because my dad was buying COTS Intel 1103's in 1971 or so before I was even born. And the first "I'm gonna store one bit of data in a capacitor" was done over the pond in the .uk during WWII at their code breaking plant. "like 64-bit IP addresses are the replacement for 32-bit IP addresses." Um...
- joshuaellinger 13y ago
  I just looked it up on Wiki. I remember my dad showing me core memory (after it wasn't in production) in the late 70s.
shrikant 13y ago
I know what you mean. I've worked on automated data submissions for banks in the UK, and the insistence on fixed-width, EBCDIC encoded data files for many regulatory filings (FSA, credit rating agencies) was annoying. On the other hand, it was so easy to automate in a VBA macro that I could have quite a bit of free time.
thomasjames 13y ago
It indeed was so widespread and lasted so long that the Arab world developed a really clever standard transliteration system. https://en.wikipedia.org/wiki/Arabic_chat_alphabet https://en.wikipedia.org/wiki/Arabic_chat_alphabet

chongli 13y ago

>64-bit IP addresses are the replacement for 32-bit IP addresses IPv6 addresses are 128-bit.

lotsofcows 13y ago

It's not even really a replacement. More of an extension. Anything that talks Unicode can talk ASCII (although not vice versa).

qznc 13y ago
Again the common mistake to confuse codes and encoding. ASCII is an encoding which can represent a very small subset of Unicode. UTF8 is an encoding which can represent all of Unicode and it is a superset of ASCII. UTF32 can represent all of Unicode, but is not a superset of ASCII. Therefore, something can talk Unicode via UTF32 without being able to talk ASCII.
- tome 13y ago
  But the collection of Unicode codepoints is an extension of the ASCII codepoints, isn't it? (Regardless of how they are encoded).
  - ygra 13y ago
    Yes, but that's the distinction between character sets and encodings. A character set is just that, a set of characters. In this sense Unicode is a superset of ASCII, Latin 1, and every other pre-existing character set. Encodings map characters to byte( sequence)?s and they differ a lot in that respect. Practically both character sets and encodings prove ample opportunities to mangle input, either by replacing unavailable characters by ? or �, or by mapping to the wrong characters because the wrong encoding was assumed (or by throwing errors because the input made no sense).
    - tome 13y ago
      I mean when ASCII assigns a number to a character, Unicode assigns the same number to the character. UTF16 and UTF32 may encode those numbers differently from how ASCII encodes them, but they are nonetheless the same numbers. Unicode is not an extension of EDBIC or Latin1 because characters under them have different codepoints than they do under Unicode. [Please correct me if I'm wrong. I'm not an expert but this is how it seems to me]
      - DougBTX 13y ago
        > [Please correct me if I'm wrong. I'm not an expert but this is how it seems to me] You're basically right, but the subtle point is that these comments are in response to lotsofcows who said, "Anything that talks Unicode can talk ASCII (although not vice versa)." The implication of "talk" is that the numbers are being encoded. qznc was trying to clarify that lotsofcows's statement is only true if the Unicode system is using an encoding which is backwards compatible with 7-bit ASCII, such as UTF-8.
        tome 13y ago
        Thanks, I see :) My comment wasn't particularly relevant.
      - VLM 13y ago
        "Unicode assigns the same number to the character." Extremely close but not quite right for some weird corner cases. There's either a cool feature or hideous bug in the unicode design, depends how you look at it, that lets you compose multiple unicode codes together into one glyph. So you can say, "Gimme the code for A with a bar over it" or "Gimme the code for A, now gimme the code for put a bar over the last glyph". Even worse you can stack compose characters, so you can write "A with circle on top and squiggly underneath" at least five different ways in binary that should be rendered visually identical. There is the one true normalization technique that most people use to convert what they consider poor unicode grammar into a standard form. Some people don't use it of course. And anytime you give users freeform input who knows what kind of crud they'll feed you. So you can never really assume any unicode string is normalized unless you personally normalized it yourself. Even then theoretically two normalized strings, concatenated, might or might not be normalized anymore (although this is often not much of a problem). And this strikes substring manipulation too. A fun source of buffer overflows is normalization can shrink OR EXPAND a unicode string, in theory. So if you use one of those languages without variable length strings, look out. Or if the language tries to take care of it for you, this leads to weird memory fragmentation. Maybe a DOS/DDOS opportunity? Which leads to philosophical argument you must decide in your code, if two bitstreams don't match, but you get matching glyph renderings, from the program's point of view are they a match or not? Depends if what your program is trying to do I guess. Most languages have a library to handle this. Writing your own unicode handler functions is a "here be dragons" moment. Composing characters are also a fun source of swear words if you're trying to count the number of characters in a file. Something that renders literally identically will have an identical number of characters but somewhat varying number of bytes. Other fun philosophical arguments are if you read in a un-normalized string and output a normalized string have you changed the string? Well, both no, and yes. I'm sure there's crypto steagongraphy implications. Google for "unicode normalization form nfc" and stuff like that. The first page I found with a good faq was: http://www.macchiato.com/unicode/nfc-faq http://www.macchiato.com/unicode/nfc-faq
- brudgers 13y ago
  Joel Spolsky provides the necessary (in Joel's opinion of 2003) minimum background in, The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) http://www.joelonsoftware.com/articles/Unicode.html http://www.joelonsoftware.com/articles/Unicode.html

VLM 13y ago

"Modern RAM did not exist until 1975 -- a decade later." What does that even mean? It doesn't mean DIP packaged DRAM because my dad was buying COTS Intel 1103's in 1971 or so before I was even born. And the first "I'm gonna store one bit of data in a capacitor" was done over the pond in the .uk during WWII at their code breaking plant. "like 64-bit IP addresses are the replacement for 32-bit IP addresses." Um...

joshuaellinger 13y ago
I just looked it up on Wiki. I remember my dad showing me core memory (after it wasn't in production) in the late 70s.

shrikant 13y ago

I know what you mean. I've worked on automated data submissions for banks in the UK, and the insistence on fixed-width, EBCDIC encoded data files for many regulatory filings (FSA, credit rating agencies) was annoying. On the other hand, it was so easy to automate in a VBA macro that I could have quite a bit of free time.

thomasjames 13y ago

It indeed was so widespread and lasted so long that the Arab world developed a really clever standard transliteration system. https://en.wikipedia.org/wiki/Arabic_chat_alphabet https://en.wikipedia.org/wiki/Arabic_chat_alphabet