9 ms·

How ASCII lost and unicode won

82 points by stevejalim 13y ago

ygra 13y ago
I'm impressed. Easily readable and understandable, short and as far as I can tell no factual inaccuracies and wrong information (unlike many other Unicode introductions and tutorials).
- rjh29 13y ago
  What do you think about http://richardharr.is/unicode-in-five-minutes.html http://richardharr.is/unicode-in-five-minutes.html ?
- aidos 13y ago
  I think this is great explanation for anyone who's approaching the subject for the first time. It gives a good introduction as to why you're staring at broken data coming from a db and just how royally screwed your afternoon is going to be getting back into shape :)
- gnosis 13y ago
  I liked the explanation at the beginning of the Strings chapter in the Dive Into Python 3 tutorial: http://getpython3.com/diveintopython3/strings.html http://getpython3.com/diveintopython3/strings.html
- qznc 13y ago
  You want some criticism, because we have too little of that here on HN? I'll bite. ;) "A byte is a set of 8 bits. Computers typically move data around a byte at a time." A byte being 8 bits is ok. Historically, a byte might have a different number of bits, but all modern architectures use 8 bits. Since this is an introductory article, this is fine. (more details: http://en.wikipedia.org/wiki/Byte http://en.wikipedia.org/wiki/Byte) Computers do not typically move data in byte chunks. You could say "a byte is the smallest unit of data a CPU can load or store". If you talk about moving data, the question is between what. Probably memory. However, there are caches nowadays, since bandwidth is cheap and latency is expensive. Data is moved in cache line chunks, which means 4-64 byte chunks depending on architecture and cache level. Bigger chunks in upcoming architectures.
  - chiph 13y ago
    IIRC, the IBM System 360 is the machine that really set the 8 bits == 1 byte convention in stone. Before that you had CDC machines with 6-bit bytes (or Nybbles if you owned an Apple ][ with a floppy disk)
    - mjn 13y ago
      There's some subjectivity in where you consider it standardized, but I would put it later than that. Throughout the period when IBM's System/360 was popular, DEC's PDP series was also popular, which used 36-bit words and a somewhat varying definition of what constituted a byte. If I had to pick a date, I'd pick 1977 as when it solidified. A few elements: ARPAnet standardized on octet-based protocols in the early 1970s, and was getting widespread by the mid 1970s; three popular microcomputers based on 8-bit architectures were introduced in 1977 (the Apple II, TRS-80, and Commodore PET); and DEC introduced the 32-bit VAX in 1977.
    - kalleboo 13y ago
      And thanks to good old SMTP originally not being specified as 8-bit, we get freakshows like UTF-7 http://en.wikipedia.org/wiki/UTF-7 http://en.wikipedia.org/wiki/UTF-7
    - npongratz 13y ago
      I knew bytes were ambiguously sized, but I had thought a nibble/nybble was (A) always half an octet, and (B) a fun Apple coders magazine [0]. But sure enough, I was very wrong on (A): Apple II disk writes were done in "sets of 5-bit or, later, 6-bit nibbles" [1]. Thank you, chiph! [0] http://www.nibblemagazine.com/ http://www.nibblemagazine.com/ [1] https://en.wikipedia.org/wiki/Nibble#History https://en.wikipedia.org/wiki/Nibble#History
      - chiph 13y ago
        There's probably some variation in the terminology, but I always used these terms: bit nibble (4 bits) nybble (6 bits) byte (8 bits) word (16 bits) :
  - 13y ago
timthorn 13y ago
I really hate to nitpick, but the article implies that ASCII was the first character encoding. In fact, there was a rich history of different encodings before that, with different word sizes and/or incompatible 8 bit encodings. It's quite interesting to look back and see what trade-offs were made and why.
- ygra 13y ago
  Well, it's the oldest character set and encoding that's still semi-relevant today. I doubt many people nowadays encounter EBCDIC and the like (and if they do, the article isn't aimed at them, I guess).
  - mcherm 13y ago
    I encounter it occasionally. (Although I am certainly not the article's target demographic.)
    - huhtenberg 13y ago
      > What’s a bit? > What’s a byte? I'm not sure what the target demographic is.
      - drdaeman 13y ago
        > What’s a byte? > A byte is a set of 8 bits. Nitpicking, that's octet, not byte. I believe the distinction was lost and don't think byte has a proper definition anymore.
        fosap 13y ago
        But for historical encodings that kind of matters. Because a byte was not always 8 bit we have to deal with utf-7 and had to ensure things where 8 bit clean. And utf-9.
  - VLM 13y ago
    But it misses some relevant semi-dramatic story. For all practical purposes EBCDIC is the "IBM Standard" as opposed to ASCII's "American Standard". The mood in the 70s/80s outside the business world was "In your face IBM!" And putting "Information Interchange" in the name itself is another "In Your Face" posturing to the mainframe world. We're the future of data transmission and you'd best get used to it, IBM... ASCII really was a rebellion in the olden times. One that won. Another story was before ASCII note that teletype codes and such were usually modal, LTRS/FIGS to switch from 5-bit letters to 5-bit numbers. So there's that dramatic great circular wheel of IT where we've oscillated both before and after ASCII between simple encodings and modal encodings. This was an early whine against unicode, who cares about codepages, just embed it like, or in, what amounts to the MIME media type, and glyph-like Asian languages should just be drawn in gif files anyway. Or so the complaints went at that time. Another design statement story: Kind of like the Uni- in unicode uniting all the extended code pages into one really huge space. There are other dramatic stories not in the article. For example the Klingon in Unicode movement. Basically about 15 years ago they tried to get Klingon script into unicode, about a decade ago the unicode people (who?) said no, so the Klingon people squatted in the unicode equivalent of what networking people would call RFC1918 space, and its simmered since then. Will Klingon actually make it into unicode officially or not, who knows. You can add to the fire by pointing out that Unicode is already stuffed with scripts that no living human culture currently uses, and numerous glyph symbols (think of math stuff like + or %) On the other hand this would inevitably result in Tolkien Elvish script being included in unicode. And does this really matter one way or the other? And this maps perfectly into the wikipedia battle between the deletionist (expletive deleted) and the inclusionist saviors of humanity. Well maybe that last line was a little biased to my opinions... There's tons of extra fun drama to tell about unicode and cutting off the story at the founding of ASCII misses some of the fun drama. Aside from some fun drama that wasn't mentioned in a post aimed at non-techs who we're told really like drama. You could probably turn the Unicode story into a trashy reality TV show somehow; Vampire Romance Fiction is going to be a harder translation although I'd love to see it.
gnosis 13y ago
"Designed as a single, global replacement for localised character sets, the Unicode standard is beautiful in its simplicity. In essence: collect all the characters in all the scripts known to humanity and number them in one single, canonical list. If new characters are invented or discovered, no problem, just add them to the list. The list isn’t an 8-bit list, or a 16-bit list, it’s just a list, with no limit on its length." Is this really true? My impression was that UTF-32 is a fixed-length encoding which uses 32 bits to encode all of Unicode. It seems that this means that Unicode can never have more code points than could fit in 32 bits. Right?
- alexjeffrey 13y ago
  I believe it depends on the encoding - UTF-32 has a limit of 2^32 characters, but UTF-8 could potentially expand infinitely (ignoring the current arbitrary 4-byte limit).
  - cygx 13y ago
    That's incorrect - the UTF-8 scheme encodes the sequence length in the first byte as the number of bits set to 1 until the first 0. The original limit was 6 bytes encoding 31 bits, but this was lowered to 4 bytes encoding 21 bits. If I'm not mistaken, 7 bytes encoding 36 bits (or 8 bytes encoding 42 bits if you make the 0 optional) should be the hard limit.
    - drdaeman 13y ago
      If we ignore this artificial restriction, we can use arbitrary lengths, turning UTF-8 into varint encoding. I.e.: Oct.1 Oct.2 Oct.3 11111111 110DDDDD DDDDDDDD (7 octets more) \_________/ 10 bits ("D"s represent data bits) This is what'd be necessary to use UTF-8 if we'd need to go beyond the current limits of 0x010FFFF upper bound.
      - cygx 13y ago
        In your scheme, we'll end up with multiple leading bytes, though. Also, how do you plan to handle 9 leading bits instead of the 10 from your example? That would make the second byte start with 10, the marker for continuation bytes.
joshuaellinger 13y ago
ASCII is by far the most successful character encoding that computers have used. It was invented in 1963, back in the era of punch cards and core memory. Modern RAM did not exist until 1975 -- a decade later. Unicode is the replacement, not the competitor, like 64-bit IP addresses are the replacement for 32-bit IP addresses. It was developed in the early 1990s when RAM got cheap enough that you could afford two-bytes per character. Personally, I deal with data all the time and rarely encounter unicode. Of course, I'm in the US dealing with big files out of financial and marketing databases. In fact, I've seen more EBCDIC than UNICODE.
- chongli 13y ago
  >64-bit IP addresses are the replacement for 32-bit IP addresses IPv6 addresses are 128-bit.
- lotsofcows 13y ago
  It's not even really a replacement. More of an extension. Anything that talks Unicode can talk ASCII (although not vice versa).
  - qznc 13y ago
    Again the common mistake to confuse codes and encoding. ASCII is an encoding which can represent a very small subset of Unicode. UTF8 is an encoding which can represent all of Unicode and it is a superset of ASCII. UTF32 can represent all of Unicode, but is not a superset of ASCII. Therefore, something can talk Unicode via UTF32 without being able to talk ASCII.
    - tome 13y ago
      But the collection of Unicode codepoints is an extension of the ASCII codepoints, isn't it? (Regardless of how they are encoded).
      - ygra 13y ago
        Yes, but that's the distinction between character sets and encodings. A character set is just that, a set of characters. In this sense Unicode is a superset of ASCII, Latin 1, and every other pre-existing character set. Encodings map characters to byte( sequence)?s and they differ a lot in that respect. Practically both character sets and encodings prove ample opportunities to mangle input, either by replacing unavailable characters by ? or �, or by mapping to the wrong characters because the wrong encoding was assumed (or by throwing errors because the input made no sense).
Digit-Al 13y ago
>ASCII really should have been named ASCIIWOA: the American Standard Code for Information Exchange With Other Americans. So he thinks Americans are the only people to use the English language does he?
- pfortuny 13y ago
  No, he says that because there is no Pound, Shilling or other specific symbols for English-speaking countries, I guess.
- kps 13y ago
  It's worse than that, actually, as ASCII from the start¹ included provisions for variants for non-English latin characters and alternate currency symbols, and ASCII was essentially the same project as ECMA-6² (ECMA being the European Computer Manufacturers' Association³, a standardization group founded in 1961). ASCII as we know it (which is essentially the 1967 version⁴) like the corresponding ECMA standard⁵ provided for overloading punctuation characters as diacritics ("/¨ ^/ˆ ~/˜ '/´ ‘/` ,/¸) to be overstruck in typewriter fashion; ECMA-35⁶ (1971⁷) defines further extension techniques using control and/or escape sequences. So, yes, it's just a failed attempt at an anti-American cheap shot from someone who isn't familiar with the development of character set encodings. ¹ American Standard Code for Information Interchange, http://www.wps.com/projects/codes/X3.4-1963/index.html http://www.wps.com/projects/codes/X3.4-1963/index.html ² 7-bit Coded Character Set, http://www.ecma-international.org/publications/standards/Ecma-006.htm http://www.ecma-international.org/publications/standards/Ecm... ³ http://www.ecma-international.org/default.htm http://www.ecma-international.org/default.htm ⁴ http://www.wps.com/J/codes/Revised-ASCII/index.html http://www.wps.com/J/codes/Revised-ASCII/index.html ⁵ 7-bit Input/Output Coded Character Set, 4th Edition is unfortunately the oldest available online; http://www.ecma-international.org/publications/files/ECMA-ST-WITHDRAWN/ECMA-6,%204th%20Edition,%20August%201973.pdf http://www.ecma-international.org/publications/files/ECMA-ST... ⁶ Character Code Structure and Extension Techniques, http://www.ecma-international.org/publications/standards/Ecma-035.htm http://www.ecma-international.org/publications/standards/Ecm... ⁷ Extension of the 7-bit Coded Character Set, http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-35,%201st%20Edition,%20December%201971.pdf http://www.ecma-international.org/publications/files/ECMA-ST...
stuartcw 13y ago
It's not a matter of winning or loosing. The pre-unicode mix of character sets was a mess when it came internationalization. Try truncating a Japanese Shift-JIS string in C. That will learn you..
- ygra 13y ago
  Arguably Unicode (UTF-8 and -16) doesn't necessarily make this any easier. Or any variable-length encoding, really. You see halved code points quite frequently, and if not that, then halved " and the like.
danso 13y ago
OT and out of curiosity...how do non-native English speakers experience typing/keyboard education? I can barely remember how to make any of the basic accents over the `e` when trying to sound French...are typing classes in non-English schooling systems much more sophisticated than in English (i.e. ASCII-centric) schools? I wonder if non-native English typists come away with a better handling of the power of keyboard shortcuts (whether to create accents or not)
- estel 13y ago
  The easiest solution is to use country specific keyboards that include characters most needed by that language ie. http://en.wikipedia.org/wiki/File:KB_France.svg http://en.wikipedia.org/wiki/File:KB_France.svg
- gilgoomesh 13y ago
  Japanese often type in "Romaji" (a romanized transliteration of Japanese). Basically, they type phonetically in roman characters and the computer suggests/autocompletes characters which match the phonetics. It's a little like typing SMSs in T9 on old numberpad phones.
- kalleboo 13y ago
  Most countries just have different keyboards so they don't need any dead keys. Look to the right of P/L http://www.danielschlaug.com/journal/wp-content/uploads/2011/03/Screen-shot-2011-03-03-at-20.44.46.png http://www.danielschlaug.com/journal/wp-content/uploads/2011...
  - yebyen 13y ago
    What do you do without square and curly braces, I'm curious? Everyone writes only lisp and html? :)
    - deleted 13y ago
      [deleted]
- salmonellaeater 13y ago
peterkelly 13y ago
Another good article on this topic is the one by Joel Spolsky: http://www.joelonsoftware.com/articles/Unicode.html http://www.joelonsoftware.com/articles/Unicode.html
lmm 13y ago
Given the controversy over Han unification, I suspect that incompatible charactersets will be with us for a while yet, more's the pity.
salmonellaeater 13y ago
The fact that UTF-8 and UTF-16 are often exposed to programmers when dealing with text is a major failure of separation-of-concerns. If you had a stream of data that was gzipped, would it ever make sense to look at the bytes in the data stream before decompressing it? Variable-length text encodings are the same. Application code should only see Unicode code points. In general it was a mistake to put variable-length encodings into the Unicode standard. A much better design would have been to use UTF-32 for the application-level interface to characters, and use a separate compression standard that is optimized for fixed alphabets when transporting or storing text. This has the advantage that the compression scheme can be dynamically updated to match the letter frequencies in the real-world text, and it logically separates the ideas of encoding and compression so that the compression container is easier to swap out. And, of course, an entire class of bugs would be eliminated from application code. Edited first paragraph to clarify: Variable-length text encodings are the same.
- est 13y ago
  This looks like how Python3 handles Unicode. 1byte for ASCII, 2 bytes for BMP, 4 bytes for everything else.
- cygx 13y ago
  That makes it sound as if UTF-32 would be a silver bullet - it's not. Application code normally has to deal with user-perceived characters/glyphs/grapheme clusters, which means you'll have to treat UTF-32 as variable-length as well. If you want to get back the supposed benefits of UTF-32, you'll have to dynamically assign codepoints to grapheme clusters.
- Someone 13y ago
  I agree that that is the ideal end situation, but Unicode would have been dead on delivery if they had chosen that approach. Memory just was too expensive at the time to make a system that, in most of the computer-using world, wasted 75% in every text string. And no, just-in-time decompression wouldn't have worked either; CPU cycles also were too expensive at the time to do that. Unicode also would have been too incompatible with existing code that copied 8-bit character strings around. See http://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt http://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt for some rationale behind UTF-8.
okwa 13y ago
> These mappings of numbers to characters are just a convention that someone decided on when ASCII was developed in the 1960s. There’s nothing fundamental that dictates that a capital A has to be character number 65, that’s just the number they chose back in the day. I don't think it's mere coincidence that the capital letters start at 65 and the lower case at 97 and the decimal digits at 48.
rayiner 13y ago
Unicode, meh. Nobody will ever need more than 128 characters.
lelf 13y ago
Well, when 99% think unicode = encoding = ucs2 = utf-16, don't believe there's something outside BMP, and wtf is the only word coming to their mind when they hear about graphemes… Unicode won?

82 points by stevejalim 13y ago

ygra 13y ago

I'm impressed. Easily readable and understandable, short and as far as I can tell no factual inaccuracies and wrong information (unlike many other Unicode introductions and tutorials).

rjh29 13y ago
What do you think about http://richardharr.is/unicode-in-five-minutes.html http://richardharr.is/unicode-in-five-minutes.html ?
aidos 13y ago
I think this is great explanation for anyone who's approaching the subject for the first time. It gives a good introduction as to why you're staring at broken data coming from a db and just how royally screwed your afternoon is going to be getting back into shape :)
gnosis 13y ago
I liked the explanation at the beginning of the Strings chapter in the Dive Into Python 3 tutorial: http://getpython3.com/diveintopython3/strings.html http://getpython3.com/diveintopython3/strings.html
qznc 13y ago
You want some criticism, because we have too little of that here on HN? I'll bite. ;) "A byte is a set of 8 bits. Computers typically move data around a byte at a time." A byte being 8 bits is ok. Historically, a byte might have a different number of bits, but all modern architectures use 8 bits. Since this is an introductory article, this is fine. (more details: http://en.wikipedia.org/wiki/Byte http://en.wikipedia.org/wiki/Byte) Computers do not typically move data in byte chunks. You could say "a byte is the smallest unit of data a CPU can load or store". If you talk about moving data, the question is between what. Probably memory. However, there are caches nowadays, since bandwidth is cheap and latency is expensive. Data is moved in cache line chunks, which means 4-64 byte chunks depending on architecture and cache level. Bigger chunks in upcoming architectures.
- chiph 13y ago
  IIRC, the IBM System 360 is the machine that really set the 8 bits == 1 byte convention in stone. Before that you had CDC machines with 6-bit bytes (or Nybbles if you owned an Apple ][ with a floppy disk)
  - mjn 13y ago
    There's some subjectivity in where you consider it standardized, but I would put it later than that. Throughout the period when IBM's System/360 was popular, DEC's PDP series was also popular, which used 36-bit words and a somewhat varying definition of what constituted a byte. If I had to pick a date, I'd pick 1977 as when it solidified. A few elements: ARPAnet standardized on octet-based protocols in the early 1970s, and was getting widespread by the mid 1970s; three popular microcomputers based on 8-bit architectures were introduced in 1977 (the Apple II, TRS-80, and Commodore PET); and DEC introduced the 32-bit VAX in 1977.
  - kalleboo 13y ago
    And thanks to good old SMTP originally not being specified as 8-bit, we get freakshows like UTF-7 http://en.wikipedia.org/wiki/UTF-7 http://en.wikipedia.org/wiki/UTF-7
  - npongratz 13y ago
    I knew bytes were ambiguously sized, but I had thought a nibble/nybble was (A) always half an octet, and (B) a fun Apple coders magazine [0]. But sure enough, I was very wrong on (A): Apple II disk writes were done in "sets of 5-bit or, later, 6-bit nibbles" [1]. Thank you, chiph! [0] http://www.nibblemagazine.com/ http://www.nibblemagazine.com/ [1] https://en.wikipedia.org/wiki/Nibble#History https://en.wikipedia.org/wiki/Nibble#History
    - chiph 13y ago
      There's probably some variation in the terminology, but I always used these terms: bit nibble (4 bits) nybble (6 bits) byte (8 bits) word (16 bits) :
- 13y ago

timthorn 13y ago

I really hate to nitpick, but the article implies that ASCII was the first character encoding. In fact, there was a rich history of different encodings before that, with different word sizes and/or incompatible 8 bit encodings. It's quite interesting to look back and see what trade-offs were made and why.

ygra 13y ago
Well, it's the oldest character set and encoding that's still semi-relevant today. I doubt many people nowadays encounter EBCDIC and the like (and if they do, the article isn't aimed at them, I guess).
- mcherm 13y ago
  I encounter it occasionally. (Although I am certainly not the article's target demographic.)
  - huhtenberg 13y ago
    > What’s a bit? > What’s a byte? I'm not sure what the target demographic is.
    - drdaeman 13y ago
      > What’s a byte? > A byte is a set of 8 bits. Nitpicking, that's octet, not byte. I believe the distinction was lost and don't think byte has a proper definition anymore.
      - fosap 13y ago
        But for historical encodings that kind of matters. Because a byte was not always 8 bit we have to deal with utf-7 and had to ensure things where 8 bit clean. And utf-9.
- VLM 13y ago
  But it misses some relevant semi-dramatic story. For all practical purposes EBCDIC is the "IBM Standard" as opposed to ASCII's "American Standard". The mood in the 70s/80s outside the business world was "In your face IBM!" And putting "Information Interchange" in the name itself is another "In Your Face" posturing to the mainframe world. We're the future of data transmission and you'd best get used to it, IBM... ASCII really was a rebellion in the olden times. One that won. Another story was before ASCII note that teletype codes and such were usually modal, LTRS/FIGS to switch from 5-bit letters to 5-bit numbers. So there's that dramatic great circular wheel of IT where we've oscillated both before and after ASCII between simple encodings and modal encodings. This was an early whine against unicode, who cares about codepages, just embed it like, or in, what amounts to the MIME media type, and glyph-like Asian languages should just be drawn in gif files anyway. Or so the complaints went at that time. Another design statement story: Kind of like the Uni- in unicode uniting all the extended code pages into one really huge space. There are other dramatic stories not in the article. For example the Klingon in Unicode movement. Basically about 15 years ago they tried to get Klingon script into unicode, about a decade ago the unicode people (who?) said no, so the Klingon people squatted in the unicode equivalent of what networking people would call RFC1918 space, and its simmered since then. Will Klingon actually make it into unicode officially or not, who knows. You can add to the fire by pointing out that Unicode is already stuffed with scripts that no living human culture currently uses, and numerous glyph symbols (think of math stuff like + or %) On the other hand this would inevitably result in Tolkien Elvish script being included in unicode. And does this really matter one way or the other? And this maps perfectly into the wikipedia battle between the deletionist (expletive deleted) and the inclusionist saviors of humanity. Well maybe that last line was a little biased to my opinions... There's tons of extra fun drama to tell about unicode and cutting off the story at the founding of ASCII misses some of the fun drama. Aside from some fun drama that wasn't mentioned in a post aimed at non-techs who we're told really like drama. You could probably turn the Unicode story into a trashy reality TV show somehow; Vampire Romance Fiction is going to be a harder translation although I'd love to see it.

gnosis 13y ago

"Designed as a single, global replacement for localised character sets, the Unicode standard is beautiful in its simplicity. In essence: collect all the characters in all the scripts known to humanity and number them in one single, canonical list. If new characters are invented or discovered, no problem, just add them to the list. The list isn’t an 8-bit list, or a 16-bit list, it’s just a list, with no limit on its length." Is this really true? My impression was that UTF-32 is a fixed-length encoding which uses 32 bits to encode all of Unicode. It seems that this means that Unicode can never have more code points than could fit in 32 bits. Right?

alexjeffrey 13y ago
I believe it depends on the encoding - UTF-32 has a limit of 2^32 characters, but UTF-8 could potentially expand infinitely (ignoring the current arbitrary 4-byte limit).
- cygx 13y ago
  That's incorrect - the UTF-8 scheme encodes the sequence length in the first byte as the number of bits set to 1 until the first 0. The original limit was 6 bytes encoding 31 bits, but this was lowered to 4 bytes encoding 21 bits. If I'm not mistaken, 7 bytes encoding 36 bits (or 8 bytes encoding 42 bits if you make the 0 optional) should be the hard limit.
  - drdaeman 13y ago
    If we ignore this artificial restriction, we can use arbitrary lengths, turning UTF-8 into varint encoding. I.e.: Oct.1 Oct.2 Oct.3 11111111 110DDDDD DDDDDDDD (7 octets more) \_________/ 10 bits ("D"s represent data bits) This is what'd be necessary to use UTF-8 if we'd need to go beyond the current limits of 0x010FFFF upper bound.
    - cygx 13y ago
      In your scheme, we'll end up with multiple leading bytes, though. Also, how do you plan to handle 9 leading bits instead of the 10 from your example? That would make the second byte start with 10, the marker for continuation bytes.

joshuaellinger 13y ago

ASCII is by far the most successful character encoding that computers have used. It was invented in 1963, back in the era of punch cards and core memory. Modern RAM did not exist until 1975 -- a decade later. Unicode is the replacement, not the competitor, like 64-bit IP addresses are the replacement for 32-bit IP addresses. It was developed in the early 1990s when RAM got cheap enough that you could afford two-bytes per character. Personally, I deal with data all the time and rarely encounter unicode. Of course, I'm in the US dealing with big files out of financial and marketing databases. In fact, I've seen more EBCDIC than UNICODE.

chongli 13y ago
>64-bit IP addresses are the replacement for 32-bit IP addresses IPv6 addresses are 128-bit.
lotsofcows 13y ago
It's not even really a replacement. More of an extension. Anything that talks Unicode can talk ASCII (although not vice versa).
- qznc 13y ago
  Again the common mistake to confuse codes and encoding. ASCII is an encoding which can represent a very small subset of Unicode. UTF8 is an encoding which can represent all of Unicode and it is a superset of ASCII. UTF32 can represent all of Unicode, but is not a superset of ASCII. Therefore, something can talk Unicode via UTF32 without being able to talk ASCII.
  - tome 13y ago
    But the collection of Unicode codepoints is an extension of the ASCII codepoints, isn't it? (Regardless of how they are encoded).
    - ygra 13y ago
      Yes, but that's the distinction between character sets and encodings. A character set is just that, a set of characters. In this sense Unicode is a superset of ASCII, Latin 1, and every other pre-existing character set. Encodings map characters to byte( sequence)?s and they differ a lot in that respect. Practically both character sets and encodings prove ample opportunities to mangle input, either by replacing unavailable characters by ? or �, or by mapping to the wrong characters because the wrong encoding was assumed (or by throwing errors because the input made no sense).

Digit-Al 13y ago

>ASCII really should have been named ASCIIWOA: the American Standard Code for Information Exchange With Other Americans. So he thinks Americans are the only people to use the English language does he?

pfortuny 13y ago
No, he says that because there is no Pound, Shilling or other specific symbols for English-speaking countries, I guess.
kps 13y ago
It's worse than that, actually, as ASCII from the start¹ included provisions for variants for non-English latin characters and alternate currency symbols, and ASCII was essentially the same project as ECMA-6² (ECMA being the European Computer Manufacturers' Association³, a standardization group founded in 1961). ASCII as we know it (which is essentially the 1967 version⁴) like the corresponding ECMA standard⁵ provided for overloading punctuation characters as diacritics ("/¨ ^/ˆ ~/˜ '/´ ‘/` ,/¸) to be overstruck in typewriter fashion; ECMA-35⁶ (1971⁷) defines further extension techniques using control and/or escape sequences. So, yes, it's just a failed attempt at an anti-American cheap shot from someone who isn't familiar with the development of character set encodings. ¹ American Standard Code for Information Interchange, http://www.wps.com/projects/codes/X3.4-1963/index.html http://www.wps.com/projects/codes/X3.4-1963/index.html ² 7-bit Coded Character Set, http://www.ecma-international.org/publications/standards/Ecma-006.htm http://www.ecma-international.org/publications/standards/Ecm... ³ http://www.ecma-international.org/default.htm http://www.ecma-international.org/default.htm ⁴ http://www.wps.com/J/codes/Revised-ASCII/index.html http://www.wps.com/J/codes/Revised-ASCII/index.html ⁵ 7-bit Input/Output Coded Character Set, 4th Edition is unfortunately the oldest available online; http://www.ecma-international.org/publications/files/ECMA-ST-WITHDRAWN/ECMA-6,%204th%20Edition,%20August%201973.pdf http://www.ecma-international.org/publications/files/ECMA-ST... ⁶ Character Code Structure and Extension Techniques, http://www.ecma-international.org/publications/standards/Ecma-035.htm http://www.ecma-international.org/publications/standards/Ecm... ⁷ Extension of the 7-bit Coded Character Set, http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-35,%201st%20Edition,%20December%201971.pdf http://www.ecma-international.org/publications/files/ECMA-ST...

stuartcw 13y ago

It's not a matter of winning or loosing. The pre-unicode mix of character sets was a mess when it came internationalization. Try truncating a Japanese Shift-JIS string in C. That will learn you..

ygra 13y ago
Arguably Unicode (UTF-8 and -16) doesn't necessarily make this any easier. Or any variable-length encoding, really. You see halved code points quite frequently, and if not that, then halved " and the like.

danso 13y ago

OT and out of curiosity...how do non-native English speakers experience typing/keyboard education? I can barely remember how to make any of the basic accents over the `e` when trying to sound French...are typing classes in non-English schooling systems much more sophisticated than in English (i.e. ASCII-centric) schools? I wonder if non-native English typists come away with a better handling of the power of keyboard shortcuts (whether to create accents or not)

estel 13y ago
The easiest solution is to use country specific keyboards that include characters most needed by that language ie. http://en.wikipedia.org/wiki/File:KB_France.svg http://en.wikipedia.org/wiki/File:KB_France.svg
gilgoomesh 13y ago
Japanese often type in "Romaji" (a romanized transliteration of Japanese). Basically, they type phonetically in roman characters and the computer suggests/autocompletes characters which match the phonetics. It's a little like typing SMSs in T9 on old numberpad phones.
kalleboo 13y ago
Most countries just have different keyboards so they don't need any dead keys. Look to the right of P/L http://www.danielschlaug.com/journal/wp-content/uploads/2011/03/Screen-shot-2011-03-03-at-20.44.46.png http://www.danielschlaug.com/journal/wp-content/uploads/2011...
- yebyen 13y ago
  What do you do without square and curly braces, I'm curious? Everyone writes only lisp and html? :)
  - deleted 13y ago
    [deleted]
salmonellaeater 13y ago

peterkelly 13y ago

Another good article on this topic is the one by Joel Spolsky: http://www.joelonsoftware.com/articles/Unicode.html http://www.joelonsoftware.com/articles/Unicode.html

lmm 13y ago

Given the controversy over Han unification, I suspect that incompatible charactersets will be with us for a while yet, more's the pity.

salmonellaeater 13y ago

The fact that UTF-8 and UTF-16 are often exposed to programmers when dealing with text is a major failure of separation-of-concerns. If you had a stream of data that was gzipped, would it ever make sense to look at the bytes in the data stream before decompressing it? Variable-length text encodings are the same. Application code should only see Unicode code points. In general it was a mistake to put variable-length encodings into the Unicode standard. A much better design would have been to use UTF-32 for the application-level interface to characters, and use a separate compression standard that is optimized for fixed alphabets when transporting or storing text. This has the advantage that the compression scheme can be dynamically updated to match the letter frequencies in the real-world text, and it logically separates the ideas of encoding and compression so that the compression container is easier to swap out. And, of course, an entire class of bugs would be eliminated from application code. Edited first paragraph to clarify: Variable-length text encodings are the same.

est 13y ago
This looks like how Python3 handles Unicode. 1byte for ASCII, 2 bytes for BMP, 4 bytes for everything else.
cygx 13y ago
That makes it sound as if UTF-32 would be a silver bullet - it's not. Application code normally has to deal with user-perceived characters/glyphs/grapheme clusters, which means you'll have to treat UTF-32 as variable-length as well. If you want to get back the supposed benefits of UTF-32, you'll have to dynamically assign codepoints to grapheme clusters.
Someone 13y ago
I agree that that is the ideal end situation, but Unicode would have been dead on delivery if they had chosen that approach. Memory just was too expensive at the time to make a system that, in most of the computer-using world, wasted 75% in every text string. And no, just-in-time decompression wouldn't have worked either; CPU cycles also were too expensive at the time to do that. Unicode also would have been too incompatible with existing code that copied 8-bit character strings around. See http://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt http://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt for some rationale behind UTF-8.

okwa 13y ago

> These mappings of numbers to characters are just a convention that someone decided on when ASCII was developed in the 1960s. There’s nothing fundamental that dictates that a capital A has to be character number 65, that’s just the number they chose back in the day. I don't think it's mere coincidence that the capital letters start at 65 and the lower case at 97 and the decimal digits at 48.

rayiner 13y ago

Unicode, meh. Nobody will ever need more than 128 characters.

lelf 13y ago

Well, when 99% think unicode = encoding = ucs2 = utf-16, don't believe there's something outside BMP, and wtf is the only word coming to their mind when they hear about graphemes… Unicode won?