5 ms·

> The original ASCII standard defined characters from 0 to 126. 0 to 127. 127 is a power of 2 minus 1, which should be a hint; in specific, it's two to the sev

by derleth 13y ago

> The original ASCII standard defined characters from 0 to 126. 0 to 127. 127 is a power of 2 minus 1, which should be a hint; in specific, it's two to the seventh minus one, since ASCII defines codepoints for all possible combinations of seven bits, which is 128 possible codepoints, so the enumeration ends at 127 if you count starting from zero, as computer programmers are wont to do. > all possible 127 ASCII characters 128 characters, as mentioned above. > the ASCII guys, who by the way, were American ASCII stands for American Standard Code for Information Interchange. The ethnocentrism was unfortunate but it isn't like you weren't warned. > The numbers are called “magic numbers” and they begin with U+. He can call them "magic numbers" but everyone else calls them "codepoints". > UTF-8 was an amazing concept: it single handedly and brilliantly handled backward ASCII compatibility making sure that Unicode is adopted by masses. Whoever came up with it must at least receive the Nobel Peace Prize. I'm sure Ken Thompson and Rob Pike will be happy to hear someone thinks that way.

yuhong 13y ago
>ASCII stands for American Standard Code for Information Interchange. The ethnocentrism was unfortunate but it isn't like you weren't warned. Yea, I think it reflects how much technology development comes from America even back then as well as now.
spacehunt 13y ago
>> UTF-8 was an amazing concept: it single handedly and brilliantly handled backward ASCII compatibility making sure that Unicode is adopted by masses. Whoever came up with it must at least receive the Nobel Peace Prize. > I'm sure Ken Thompson and Rob Pike will be happy to hear someone thinks that way. If that's the case then I think the designers of GB18030 deserve it more, because they achieved an encoding that is able to map all Unicode codepoints while being backwards compatible with GB2312, which is itself backwards compatible with ASCII. But seriously UTF-8 is like sliced bread after having dealt with we-thought-64K-is-enough-so-lets-all-use-16-bits UCS-2/UTF-16.
- Dylan16807 13y ago
  And even more so, if the current 20.1-bit format turns out to be insufficient, UTF-8 is happy to expand to at least 31 bits, where UTF-16 would be forced to undergo very awkward redesign, such as dedicating an entire plane to double-surrogates.
  - millstone 13y ago
    The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot be decoded as UTF-8. UTF-16 does not suffer from that particular problem. Of course, there's a long ways to go from "a sequence of valid code units" to "a valid string," but it's still relevant.
    - spacehunt 13y ago
      UTF-8 is already much better than the legacy double-byte encodings (Big5, GB2312, Shift-JIS etc etc) because, at least you can tell from any byte whether it is supposed to be a part of a code sequence, without having to start all the way from the beginning of the string.
    - KayEss 13y ago
      >UTF-16 does not suffer from that particular problem what about surrogate pairs? You can't have only one 16 bit word for a pair and have a valid UTF-16 sequence. This problem is real easy to do if you substring a UTF-16 sequence naively.
      - millstone 13y ago
        Right, there's still plenty of ways you can screw up a string. My point was merely that UTF-8 has one failure mode (invalid code units) that UTF-16 does not suffer from, and that has implications in encoding converters and APIs.
    - lucian1900 13y ago
      Invalid bytes is a feature. It makes it much easier to figure out the bytes you have are not actually UTF8.
lutusp 13y ago
> 0 to 127. 127 is a power of 2 minus 1, which should be a hint; in specific, it's two to the seventh minus one ... To expand on this a tiny bit, a power of two minus one is called a "Mersenne number". If the number is prime, it's called ... wait for it ... a Mersenne prime. When expressed in binary, Mersenne numbers are an uninterrupted series of one digits: 11111... of varying lengths. http://mathworld.wolfram.com/MersenneNumber.html http://mathworld.wolfram.com/MersenneNumber.html
namsral 13y ago
Rob Pike's story on how he and Ken Thompson invented UTF-8 over dinner: https://plus.google.com/101960720994009339267/posts/Rz1udTvtiMg https://plus.google.com/101960720994009339267/posts/Rz1udTvt...

by derleth 13y ago

yuhong 13y ago

>ASCII stands for American Standard Code for Information Interchange. The ethnocentrism was unfortunate but it isn't like you weren't warned. Yea, I think it reflects how much technology development comes from America even back then as well as now.

spacehunt 13y ago

>> UTF-8 was an amazing concept: it single handedly and brilliantly handled backward ASCII compatibility making sure that Unicode is adopted by masses. Whoever came up with it must at least receive the Nobel Peace Prize. > I'm sure Ken Thompson and Rob Pike will be happy to hear someone thinks that way. If that's the case then I think the designers of GB18030 deserve it more, because they achieved an encoding that is able to map all Unicode codepoints while being backwards compatible with GB2312, which is itself backwards compatible with ASCII. But seriously UTF-8 is like sliced bread after having dealt with we-thought-64K-is-enough-so-lets-all-use-16-bits UCS-2/UTF-16.

Dylan16807 13y ago
And even more so, if the current 20.1-bit format turns out to be insufficient, UTF-8 is happy to expand to at least 31 bits, where UTF-16 would be forced to undergo very awkward redesign, such as dedicating an entire plane to double-surrogates.
- millstone 13y ago
  The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot be decoded as UTF-8. UTF-16 does not suffer from that particular problem. Of course, there's a long ways to go from "a sequence of valid code units" to "a valid string," but it's still relevant.
  - spacehunt 13y ago
    UTF-8 is already much better than the legacy double-byte encodings (Big5, GB2312, Shift-JIS etc etc) because, at least you can tell from any byte whether it is supposed to be a part of a code sequence, without having to start all the way from the beginning of the string.
  - KayEss 13y ago
    >UTF-16 does not suffer from that particular problem what about surrogate pairs? You can't have only one 16 bit word for a pair and have a valid UTF-16 sequence. This problem is real easy to do if you substring a UTF-16 sequence naively.
    - millstone 13y ago
      Right, there's still plenty of ways you can screw up a string. My point was merely that UTF-8 has one failure mode (invalid code units) that UTF-16 does not suffer from, and that has implications in encoding converters and APIs.
  - lucian1900 13y ago
    Invalid bytes is a feature. It makes it much easier to figure out the bytes you have are not actually UTF8.

lutusp 13y ago

> 0 to 127. 127 is a power of 2 minus 1, which should be a hint; in specific, it's two to the seventh minus one ... To expand on this a tiny bit, a power of two minus one is called a "Mersenne number". If the number is prime, it's called ... wait for it ... a Mersenne prime. When expressed in binary, Mersenne numbers are an uninterrupted series of one digits: 11111... of varying lengths. http://mathworld.wolfram.com/MersenneNumber.html http://mathworld.wolfram.com/MersenneNumber.html

namsral 13y ago

Rob Pike's story on how he and Ken Thompson invented UTF-8 over dinner: https://plus.google.com/101960720994009339267/posts/Rz1udTvtiMg https://plus.google.com/101960720994009339267/posts/Rz1udTvt...