8 ms·

Love hotels and Unicode

329 points by gioele 15y ago

derleth 15y ago
When he says byte-order marks are optional, does he mean just in UTF-8 (where they are) or also in UTF-16 (where I strongly suspect they are not)? (Yes, you can use heuristics to guess which endianness is in use. The problem is that while this is trivial for Western languages I don't even know how you'd begin when presented with arbitrary text using an East Asian or African written language.)
- pmjordan 15y ago
  There's nothing stopping you from leaving off the BOM as far as I know. The BOM is pointless in UTF-8 (the encoding is inherently endian-independent) and if you can guarantee by convention that your UTF-16 text will be read in the same byte order as it was written, you also don't need it. Another thing I noticed in the article: the encodings issue was historically even more complicated: Unicode previously only covered what is now the basic multilingual plane (BMP), which meant all code points could be encoded with a single 2-byte value, a.k.a. UCS-2. When more code points got added, that exceeded UCS-2's range, so that mutated into the variable-length UTF-16 encoding. Had Unicode been introduced into Windows and Java later, who knows if they'd ended up using UTF-16.
- btn 15y ago
  The BOM is optional in UTF-16 too: The UTF-16 encoding scheme may or may not begin with a BOM. However, when there is no BOM, and in the absence of a higher-level protocol, the byte order of the UTF-16 encoding scheme is big-endian. (D98 in http://www.unicode.org/versions/Unicode6.1.0/ch03.pdf http://www.unicode.org/versions/Unicode6.1.0/ch03.pdf)
  - prodigal_erik 15y ago
    Oh jeez, that's just begging to get mangled by a transition between protocols, e.g., HTTP PUT followed by rsync (to a box which doesn't know the PUT was UTF-16LE).
- patio11 15y ago
  The problem is that while this is trivial for Western languages I don't even know how you'd begin when presented with arbitrary text using an East Asian or African written language Not actually that hard. Consider a document which is encoded in either a) ASCII like you know it or b) ASCII where the top 4 bits and bottom 4 bits are transposed. How would you tell the difference? Well, one can imagine creating a histogram of the bits for each half of the bytes and comparing them to expectations based on the distribution of bits in naturally occurring English text. The half with most of the entries in the 0x5, 0x6, and 0x7 is the upper order half. If you don't know what naturally occurring e.g. Japanese looks like in Unicode code points, take this on faith: flipping the order does not give you a document which looks probably correct. (Also, crucially, Japanese with the order flipped doesn't resemble any sensible document in any language -- you end up with Unicode code points from a mishmash of unrelated pages.) P.S. Why care about that algorithm? Here's a hypothetical: you're a forensic investigator or system administrator who, given a hard drive which has been damaged, need to extract as much information as possible from it. The BOM is very possibly not in the same undamaged sector which you are reading right now, and it may be impossible to stitch the sectors without first reading the text. How would you determine a) whether an arbitrary stream of bytes was likely a textual document, b) what the encoding was, c) what endianness it was, if appropriate, and d) what human language it was written in?
  - funkah 15y ago
    Sounds hard to me.
    - chc 15y ago
      How about this simplified version: 1. Try both byte orders 2. If one produces valid text and the other does not, choose that one (this will get you the correct answer almost every time, even if the source text is Chinese) 3. If both happen to produce valid text, use the one with the smallest number of scripts (Note that this just determines byte order, while Patrick was talking about the more ambitious task of heuristically determining whether a random string of bytes is text and if so what encoding it is. My point is just that you really don't need to be told the order of the bytes in most cases.)
guard-of-terra 15y ago
Minor point: I see copy-paste from wikipedia about ISO-8859-5. It's unfortunate since nobody ever used ISO-8859-5. They probably should change it to ISO-8858-something-else in the wikipedia article.
- pmjordan 15y ago
  It also entirely glosses over the fact that before the ISO-8859 standards, there were the horrendous code pages in DOS and numerous other encodings on other platforms, which made things hard even for Europeans, let alone languages with a non-latin-derived alphabet.
  - guard-of-terra 15y ago
    And after ISO-8859, there were Windows (aka ANSI) code pages, also not matching perfectly (or sometimes at all) with either DOS or ISO-8859.
  - derleth 15y ago
    And before that, you get into encodings like EBCDIC, RAD50, SIXBIT, FIELDATA, and even more failed schemes now largely forgotten. Why should a brief overview go back even to the pre-ISO-8859 days except to mention ASCII? None of them are directly relevant: The world we're dealing with now on the Web begins with ASCII, moves through a Pre-Unicode Period, and finishes up in the Land of Unicode, where it's at least possible to do things Right. All history tells a narrative; when it comes to character encodings, that's a good default unless you really think your audience cares about why FORTRAN was spelled that way back in the Before Time. Tom Jennings has an interesting history: http://www.wps.com/projects/codes/ http://www.wps.com/projects/codes/
    - Someone 15y ago
      "The world we're dealing with now on the Web begins with ASCII" I know nothing about the implementation of early web browsers/gopher/etc, but I doubt there ever was anything on the web that used ASCII. 7-bit email may have been around at e time, but I would guess Tim Berners Lee just used whatever character set his system used by default (corrections welcome; being snarky isn't the only reason I write this)
davidw 15y ago
I loved the idea of the slides interspersed with text. Just posting slides is usually lame, because you lose out on the actual talk, which contains most/much of the actual information.
- pmjordan 15y ago
  This really ought to be the standard way to post presentations online. Unfortunately, it's more work than uploading to the awful slideshare.
  - _delirium 15y ago
    Yeah, it's an interesting format I've recently started experimenting with for some of my own talks (though only preliminarily). It seems sort of halfway between the "throw up slides" non-solution, and the traditional academic solution, which is to accompany a talk with a written paper. Then the paper would serve as the "durable" version of the talk suitable for archiving, laying out the same material but in a way more suited to text, and usually a bit more formally, with more details (e.g. to enable someone to actually reproduce the work). The slide interspersed with text format could be seen as a more heavily illustrated, easy-reading sort of paper. More similar to the talk than a traditional accompanying paper would be, but not just the slides. But I think it's still non-trivial work to make a good version, which is why people often just throw up the slides, because that's almost literally no additional work. You basically have to take the same source material and think how you would write it up into a blog post, which is not 100% the same as thinking of how to give it as a talk. Though I guess a first-cut solution could be to just record the talk and type up a transcript in between the slides.
    - pmjordan 15y ago
      You could probably do a lot worse to prepare for a talk than writing such a text+slides version of it, then distilling key words as notes from the prose and then practicing and giving the talk based on those.
  - corbet 15y ago
mavroprovato 15y ago
Mirror from Google cache, but unfortunately text only: http://webcache.googleusercontent.com/search?q=cache:http://www.reigndesign.com/blog/love-hotels-and-unicode/&hl=en&strip=1 http://webcache.googleusercontent.com/search?q=cache:http://...
- icebraining 15y ago
  The Coral cache has images: http://www.reigndesign.com.nyud.net/blog/love-hotels-and-unicode/ http://www.reigndesign.com.nyud.net/blog/love-hotels-and-uni...
bgruber 15y ago
i had never before read about the unicode flags, and the rather (politically and technically) brilliant end run around the problem. fascinating.
- jerf 15y ago
  I am torn between it being brilliant, versus it should have been taken as a sign that adding all those symbols was just a waste of time. I sort of feel like Unicode 5 is jumping the shark here. At the point where you're arguing about how to refer to the color of the hair on one of your "graphemes", may I humbly suggest that the goal of creating a complete global encoding scheme is apparently done and the committee ought to disband. Unicode has apparently ceased being about being a universal encoding for human text and expanded its mission into becoming a universal icon catalog. This is just silly. Next up in Unicode 7, we deprecate the face icons in favor of a series of combining characters that allow to mix and match hairstyles, face colors, eyes, noses, etc., to create arbitrary faces, why not.
  - maaku 15y ago
    There is still important work to be done with respect to code point unification. Notably, encoding ancient or minority scripts that don't have national governments advocating on their behalf. Unfortunately, entrenched business and political interests have made encoding flags and emoticons a higher priority.
  - greggman 15y ago
    Or you could consider that 140million Japanese (not sure about other countries) have been using emoji every day for the last 10+ years on their cellphones. They've become part of the language and it would be culturally insensitive to give them a big "fuck you, us westerner's don't need your damn icons, either keep those to yourself and hence don't be able to inter-operate with everyone else or stop using them even though they've been a big part of your culture for 10+ years"
    - sk5t 15y ago
      Should every fad or handset feature adopted by 2% or more of the global population be incorporated into Unicode?
akrifa 15y ago
This was a fantastic read. Thank you!
brown9-2 15y ago
Does anyone happen to know what the arguments for big-endianness versus little-endianness in the Unicode format were? I've never really understood the advantage of one over the other, but this section on Wikipedia http://en.wikipedia.org/wiki/Endianness#Optimization http://en.wikipedia.org/wiki/Endianness#Optimization helps explain that there are optimizations that can be made at a hardware level when performing arithmetic on little-endianness values. Is the argument for using LE in Unicode similar?
- jonshea 15y ago
  http://www.ietf.org/rfc/ien/ien137.txt http://www.ietf.org/rfc/ien/ien137.txt
- ajross 15y ago
  If you're using, say, UCS2 (not UTF16, lest we get too confused), it's awfully nice if the wide character (i.e. a C short) for "A" can be equal to the literal 'A' and not 0x4100. Making that work depends on the endianness of the host architecture, not just the data.
adavies42 15y ago
i'd've noted that endianness is hardly unique to unicode.
sbierwagen 15y ago
ASCII was actually invented in the US in the 1960s, as a standardised way of encoding text on a computer. ASCII defined 128 characters, that's a character for half of the 256 possible bytes in an 8-bit computer system. Uh? Wouldn't it be easier to just say that it's a 7-bit coding system? And what does he mean by "256 possible bytes in an 8-bit computer system"?
- hardy263 15y ago
  "7 bits" versus "half of 8 bits" are two slightly different things. One has a padding, the other does not. So the file size for a 7 bit encoding would be slightly smaller than an 8 bit one.
- TazeTSchnitzel 15y ago
  Because there's 256 possible byte values. Like there's 65536 possible short values. (Using short to mean 2 bytes here, I'm aware it varies)
vorg 15y ago
> Every character, or to be more exact, every "grapheme", is assigned a Unicode code point. Every character is assigned a Unicode code point. The Unicode consortium defines a grapheme as a "user perceived character", usually made up of one Unicode code point, but sometimes two or more. A base character can be followed by one or more non-spacing marks, together forming a "grapheme", the most common of which usually have a "canonical mapping" to a single character, but need not.
davvid 15y ago
The one thing he forgot to say is, "If you have to choose an encoding, use utf-8." Someone who doesn't know the difference between utf-16, utf-8, etc. might not know which to use.

329 points by gioele 15y ago

When he says byte-order marks are optional, does he mean just in UTF-8 (where they are) or also in UTF-16 (where I strongly suspect they are not)? (Yes, you can use heuristics to guess which endianness is in use. The problem is that while this is trivial for Western languages I don't even know how you'd begin when presented with arbitrary text using an East Asian or African written language.)

pmjordan 15y ago
There's nothing stopping you from leaving off the BOM as far as I know. The BOM is pointless in UTF-8 (the encoding is inherently endian-independent) and if you can guarantee by convention that your UTF-16 text will be read in the same byte order as it was written, you also don't need it. Another thing I noticed in the article: the encodings issue was historically even more complicated: Unicode previously only covered what is now the basic multilingual plane (BMP), which meant all code points could be encoded with a single 2-byte value, a.k.a. UCS-2. When more code points got added, that exceeded UCS-2's range, so that mutated into the variable-length UTF-16 encoding. Had Unicode been introduced into Windows and Java later, who knows if they'd ended up using UTF-16.
btn 15y ago
The BOM is optional in UTF-16 too: The UTF-16 encoding scheme may or may not begin with a BOM. However, when there is no BOM, and in the absence of a higher-level protocol, the byte order of the UTF-16 encoding scheme is big-endian. (D98 in http://www.unicode.org/versions/Unicode6.1.0/ch03.pdf http://www.unicode.org/versions/Unicode6.1.0/ch03.pdf)
- prodigal_erik 15y ago
  Oh jeez, that's just begging to get mangled by a transition between protocols, e.g., HTTP PUT followed by rsync (to a box which doesn't know the PUT was UTF-16LE).
patio11 15y ago
The problem is that while this is trivial for Western languages I don't even know how you'd begin when presented with arbitrary text using an East Asian or African written language Not actually that hard. Consider a document which is encoded in either a) ASCII like you know it or b) ASCII where the top 4 bits and bottom 4 bits are transposed. How would you tell the difference? Well, one can imagine creating a histogram of the bits for each half of the bytes and comparing them to expectations based on the distribution of bits in naturally occurring English text. The half with most of the entries in the 0x5, 0x6, and 0x7 is the upper order half. If you don't know what naturally occurring e.g. Japanese looks like in Unicode code points, take this on faith: flipping the order does not give you a document which looks probably correct. (Also, crucially, Japanese with the order flipped doesn't resemble any sensible document in any language -- you end up with Unicode code points from a mishmash of unrelated pages.) P.S. Why care about that algorithm? Here's a hypothetical: you're a forensic investigator or system administrator who, given a hard drive which has been damaged, need to extract as much information as possible from it. The BOM is very possibly not in the same undamaged sector which you are reading right now, and it may be impossible to stitch the sectors without first reading the text. How would you determine a) whether an arbitrary stream of bytes was likely a textual document, b) what the encoding was, c) what endianness it was, if appropriate, and d) what human language it was written in?
- funkah 15y ago
  Sounds hard to me.
  - chc 15y ago
    How about this simplified version: 1. Try both byte orders 2. If one produces valid text and the other does not, choose that one (this will get you the correct answer almost every time, even if the source text is Chinese) 3. If both happen to produce valid text, use the one with the smallest number of scripts (Note that this just determines byte order, while Patrick was talking about the more ambitious task of heuristically determining whether a random string of bytes is text and if so what encoding it is. My point is just that you really don't need to be told the order of the bytes in most cases.)

guard-of-terra 15y ago

Minor point: I see copy-paste from wikipedia about ISO-8859-5. It's unfortunate since nobody ever used ISO-8859-5. They probably should change it to ISO-8858-something-else in the wikipedia article.

pmjordan 15y ago
It also entirely glosses over the fact that before the ISO-8859 standards, there were the horrendous code pages in DOS and numerous other encodings on other platforms, which made things hard even for Europeans, let alone languages with a non-latin-derived alphabet.
- guard-of-terra 15y ago
  And after ISO-8859, there were Windows (aka ANSI) code pages, also not matching perfectly (or sometimes at all) with either DOS or ISO-8859.
- derleth 15y ago
  And before that, you get into encodings like EBCDIC, RAD50, SIXBIT, FIELDATA, and even more failed schemes now largely forgotten. Why should a brief overview go back even to the pre-ISO-8859 days except to mention ASCII? None of them are directly relevant: The world we're dealing with now on the Web begins with ASCII, moves through a Pre-Unicode Period, and finishes up in the Land of Unicode, where it's at least possible to do things Right. All history tells a narrative; when it comes to character encodings, that's a good default unless you really think your audience cares about why FORTRAN was spelled that way back in the Before Time. Tom Jennings has an interesting history: http://www.wps.com/projects/codes/ http://www.wps.com/projects/codes/
  - Someone 15y ago
    "The world we're dealing with now on the Web begins with ASCII" I know nothing about the implementation of early web browsers/gopher/etc, but I doubt there ever was anything on the web that used ASCII. 7-bit email may have been around at e time, but I would guess Tim Berners Lee just used whatever character set his system used by default (corrections welcome; being snarky isn't the only reason I write this)

davidw 15y ago

I loved the idea of the slides interspersed with text. Just posting slides is usually lame, because you lose out on the actual talk, which contains most/much of the actual information.

pmjordan 15y ago
This really ought to be the standard way to post presentations online. Unfortunately, it's more work than uploading to the awful slideshare.
- _delirium 15y ago
  Yeah, it's an interesting format I've recently started experimenting with for some of my own talks (though only preliminarily). It seems sort of halfway between the "throw up slides" non-solution, and the traditional academic solution, which is to accompany a talk with a written paper. Then the paper would serve as the "durable" version of the talk suitable for archiving, laying out the same material but in a way more suited to text, and usually a bit more formally, with more details (e.g. to enable someone to actually reproduce the work). The slide interspersed with text format could be seen as a more heavily illustrated, easy-reading sort of paper. More similar to the talk than a traditional accompanying paper would be, but not just the slides. But I think it's still non-trivial work to make a good version, which is why people often just throw up the slides, because that's almost literally no additional work. You basically have to take the same source material and think how you would write it up into a blog post, which is not 100% the same as thinking of how to give it as a talk. Though I guess a first-cut solution could be to just record the talk and type up a transcript in between the slides.
  - pmjordan 15y ago
    You could probably do a lot worse to prepare for a talk than writing such a text+slides version of it, then distilling key words as notes from the prose and then practicing and giving the talk based on those.
- corbet 15y ago

mavroprovato 15y ago

Mirror from Google cache, but unfortunately text only: http://webcache.googleusercontent.com/search?q=cache:http://www.reigndesign.com/blog/love-hotels-and-unicode/&hl=en&strip=1 http://webcache.googleusercontent.com/search?q=cache:http://...

icebraining 15y ago
The Coral cache has images: http://www.reigndesign.com.nyud.net/blog/love-hotels-and-unicode/ http://www.reigndesign.com.nyud.net/blog/love-hotels-and-uni...

bgruber 15y ago

i had never before read about the unicode flags, and the rather (politically and technically) brilliant end run around the problem. fascinating.

jerf 15y ago
I am torn between it being brilliant, versus it should have been taken as a sign that adding all those symbols was just a waste of time. I sort of feel like Unicode 5 is jumping the shark here. At the point where you're arguing about how to refer to the color of the hair on one of your "graphemes", may I humbly suggest that the goal of creating a complete global encoding scheme is apparently done and the committee ought to disband. Unicode has apparently ceased being about being a universal encoding for human text and expanded its mission into becoming a universal icon catalog. This is just silly. Next up in Unicode 7, we deprecate the face icons in favor of a series of combining characters that allow to mix and match hairstyles, face colors, eyes, noses, etc., to create arbitrary faces, why not.
- maaku 15y ago
  There is still important work to be done with respect to code point unification. Notably, encoding ancient or minority scripts that don't have national governments advocating on their behalf. Unfortunately, entrenched business and political interests have made encoding flags and emoticons a higher priority.
- greggman 15y ago
  Or you could consider that 140million Japanese (not sure about other countries) have been using emoji every day for the last 10+ years on their cellphones. They've become part of the language and it would be culturally insensitive to give them a big "fuck you, us westerner's don't need your damn icons, either keep those to yourself and hence don't be able to inter-operate with everyone else or stop using them even though they've been a big part of your culture for 10+ years"
  - sk5t 15y ago
    Should every fad or handset feature adopted by 2% or more of the global population be incorporated into Unicode?

akrifa 15y ago

This was a fantastic read. Thank you!

brown9-2 15y ago

Does anyone happen to know what the arguments for big-endianness versus little-endianness in the Unicode format were? I've never really understood the advantage of one over the other, but this section on Wikipedia http://en.wikipedia.org/wiki/Endianness#Optimization http://en.wikipedia.org/wiki/Endianness#Optimization helps explain that there are optimizations that can be made at a hardware level when performing arithmetic on little-endianness values. Is the argument for using LE in Unicode similar?

jonshea 15y ago
http://www.ietf.org/rfc/ien/ien137.txt http://www.ietf.org/rfc/ien/ien137.txt
ajross 15y ago
If you're using, say, UCS2 (not UTF16, lest we get too confused), it's awfully nice if the wide character (i.e. a C short) for "A" can be equal to the literal 'A' and not 0x4100. Making that work depends on the endianness of the host architecture, not just the data.

adavies42 15y ago

i'd've noted that endianness is hardly unique to unicode.

sbierwagen 15y ago

ASCII was actually invented in the US in the 1960s, as a standardised way of encoding text on a computer. ASCII defined 128 characters, that's a character for half of the 256 possible bytes in an 8-bit computer system. Uh? Wouldn't it be easier to just say that it's a 7-bit coding system? And what does he mean by "256 possible bytes in an 8-bit computer system"?

hardy263 15y ago
"7 bits" versus "half of 8 bits" are two slightly different things. One has a padding, the other does not. So the file size for a 7 bit encoding would be slightly smaller than an 8 bit one.
TazeTSchnitzel 15y ago
Because there's 256 possible byte values. Like there's 65536 possible short values. (Using short to mean 2 bytes here, I'm aware it varies)

vorg 15y ago

> Every character, or to be more exact, every "grapheme", is assigned a Unicode code point. Every character is assigned a Unicode code point. The Unicode consortium defines a grapheme as a "user perceived character", usually made up of one Unicode code point, but sometimes two or more. A base character can be followed by one or more non-spacing marks, together forming a "grapheme", the most common of which usually have a "canonical mapping" to a single character, but need not.

davvid 15y ago

The one thing he forgot to say is, "If you have to choose an encoding, use utf-8." Someone who doesn't know the difference between utf-16, utf-8, etc. might not know which to use.