6 ms·

Can an article be honorably retired? Joel's article was important 14 years ago, but that was a time when it was not clear that UTF-8 was going to become the def

by rspeer 10y ago

Can an article be honorably retired? Joel's article was important 14 years ago, but that was a time when it was not clear that UTF-8 was going to become the default encoding of the Internet, and a time before emoji. In 2003, codepoints outside the Basic Multilingual Plane could be safely ignored unless you were a historical linguist; now, people use them every day. This article is much better at telling you relevant things (with nice data-driven visualizations!) about how Unicode is used now.

ryandrake 10y ago
The content might be a little long in the tooth, but the exasperation about how programmers don't seem to give a shit about character encoding is, sadly, still relevant today. Unicode has been around since, what, the 90's? And UTF-8 has pretty much won the encoding war since 2008 or so. Yet, I still encounter software that fails to handle anything but ASCII, and it's 2017! Pick a random open source library, or look at a random company's software, and you're likely to find software that doesn't handle character encodings well. I think web software tends to be much better than desktop/mobile/embedded for some reason. Maybe an artifact of web guys tending to work with more recent/modern libraries and frameworks. Who's seen this before: You're getting up to speed on your new company's legacy code base. Q: "Hey, senior developer, This API takes a string as an argument and we need to display it, fine. What encoding are we using?" A: ¯\_(ツ)_/¯
- stevekemp 10y ago
  I don't disagree, but when you're an old-school developer it's hard to get this stuff right! I wrote a console-based mail client, and I struggled with UTF for months before it was all done, and even then there were niggles caused by libraries - for example passing a C++ string to a Lua script, and then using the `string:len()` lua function would return the number of bytes, rather than the length of the rendered string. Something I had to work around: https://github.com/lumail/lumail2/issues/277 https://github.com/lumail/lumail2/issues/277
  - chrisseaton 10y ago
    > rather than the length of the rendered string Do unicode strings really have a rendered length? https://eev.ee/blog/2015/09/12/dark-corners-of-unicode/#combining-characters-and-character-width https://eev.ee/blog/2015/09/12/dark-corners-of-unicode/#comb... > So the “length” of ... characters depends ... on the display font
    - rspeer 10y ago
      Well, when you're writing a console program, you can assume your font is monospaced and you can assume you have a function like wcwidth() that will tell you how many monospaced cells a string will take up. I know that eev.ee points out that wcwidth() is inconsistent (emoji screwed it up, then Unicode 9 gave a consistent width to emoji, but most libraries aren't on Unicode 9 so the situation is currently more inconsistent than before). But the situation is a lot better than just saying "I don't know, blame fonts".
    - ryandrake 10y ago
      And don't forget, it's not sum(glyph sizes) either. You ought to be taking into account kerning, too.
vorg 10y ago
> Can an article be honorably retired? Putting a link to that Spolski article in the comments on HN, reddit, etc whenever some other article on Unicode is posted has become an in-joke. You'd find it impossible to "retire" it, even honorably. Though andrewl didn't follow the apparent protocol of referring to that article with some adjective like "excellent" or "brilliant".

by rspeer 10y ago

ryandrake 10y ago

The content might be a little long in the tooth, but the exasperation about how programmers don't seem to give a shit about character encoding is, sadly, still relevant today. Unicode has been around since, what, the 90's? And UTF-8 has pretty much won the encoding war since 2008 or so. Yet, I still encounter software that fails to handle anything but ASCII, and it's 2017! Pick a random open source library, or look at a random company's software, and you're likely to find software that doesn't handle character encodings well. I think web software tends to be much better than desktop/mobile/embedded for some reason. Maybe an artifact of web guys tending to work with more recent/modern libraries and frameworks. Who's seen this before: You're getting up to speed on your new company's legacy code base. Q: "Hey, senior developer, This API takes a string as an argument and we need to display it, fine. What encoding are we using?" A: ¯\_(ツ)_/¯

stevekemp 10y ago
I don't disagree, but when you're an old-school developer it's hard to get this stuff right! I wrote a console-based mail client, and I struggled with UTF for months before it was all done, and even then there were niggles caused by libraries - for example passing a C++ string to a Lua script, and then using the `string:len()` lua function would return the number of bytes, rather than the length of the rendered string. Something I had to work around: https://github.com/lumail/lumail2/issues/277 https://github.com/lumail/lumail2/issues/277
- chrisseaton 10y ago
  > rather than the length of the rendered string Do unicode strings really have a rendered length? https://eev.ee/blog/2015/09/12/dark-corners-of-unicode/#combining-characters-and-character-width https://eev.ee/blog/2015/09/12/dark-corners-of-unicode/#comb... > So the “length” of ... characters depends ... on the display font
  - rspeer 10y ago
    Well, when you're writing a console program, you can assume your font is monospaced and you can assume you have a function like wcwidth() that will tell you how many monospaced cells a string will take up. I know that eev.ee points out that wcwidth() is inconsistent (emoji screwed it up, then Unicode 9 gave a consistent width to emoji, but most libraries aren't on Unicode 9 so the situation is currently more inconsistent than before). But the situation is a lot better than just saying "I don't know, blame fonts".
  - ryandrake 10y ago
    And don't forget, it's not sum(glyph sizes) either. You ought to be taking into account kerning, too.

vorg 10y ago

> Can an article be honorably retired? Putting a link to that Spolski article in the comments on HN, reddit, etc whenever some other article on Unicode is posted has become an in-joke. You'd find it impossible to "retire" it, even honorably. Though andrewl didn't follow the apparent protocol of referring to that article with some adjective like "excellent" or "brilliant".