6 ms·

There's also Joel Spolsky's The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!): http

by andrewl 10y ago

There's also Joel Spolsky's The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!): https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

thomk 10y ago
Haha, I was going to post the same thing! If you like Joel Spolsky you might like this video where he angrily abuses everyone about how bad they are at Excel and shows you some very cool tricks: https://www.youtube.com/watch?v=0nbkaYsR94c&t=1206s https://www.youtube.com/watch?v=0nbkaYsR94c&t=1206s
- drewmate 10y ago
  I considered myself reasonably good at Excel prior to watching this. Thanks for posting this; I learned a lot!
- deleted 10y ago
  [deleted]
- slededit 10y ago
  For most of that I was getting so annoyed "JUST USE A TABLE!!!", but he got there in the end. For those that don't know Joel Spolsky used to be a PM on Excel back in the early 90s.
rspeer 10y ago
Can an article be honorably retired? Joel's article was important 14 years ago, but that was a time when it was not clear that UTF-8 was going to become the default encoding of the Internet, and a time before emoji. In 2003, codepoints outside the Basic Multilingual Plane could be safely ignored unless you were a historical linguist; now, people use them every day. This article is much better at telling you relevant things (with nice data-driven visualizations!) about how Unicode is used now.
- ryandrake 10y ago
  The content might be a little long in the tooth, but the exasperation about how programmers don't seem to give a shit about character encoding is, sadly, still relevant today. Unicode has been around since, what, the 90's? And UTF-8 has pretty much won the encoding war since 2008 or so. Yet, I still encounter software that fails to handle anything but ASCII, and it's 2017! Pick a random open source library, or look at a random company's software, and you're likely to find software that doesn't handle character encodings well. I think web software tends to be much better than desktop/mobile/embedded for some reason. Maybe an artifact of web guys tending to work with more recent/modern libraries and frameworks. Who's seen this before: You're getting up to speed on your new company's legacy code base. Q: "Hey, senior developer, This API takes a string as an argument and we need to display it, fine. What encoding are we using?" A: ¯\_(ツ)_/¯
  - stevekemp 10y ago
    I don't disagree, but when you're an old-school developer it's hard to get this stuff right! I wrote a console-based mail client, and I struggled with UTF for months before it was all done, and even then there were niggles caused by libraries - for example passing a C++ string to a Lua script, and then using the `string:len()` lua function would return the number of bytes, rather than the length of the rendered string. Something I had to work around: https://github.com/lumail/lumail2/issues/277 https://github.com/lumail/lumail2/issues/277
    - chrisseaton 10y ago
      > rather than the length of the rendered string Do unicode strings really have a rendered length? https://eev.ee/blog/2015/09/12/dark-corners-of-unicode/#combining-characters-and-character-width https://eev.ee/blog/2015/09/12/dark-corners-of-unicode/#comb... > So the “length” of ... characters depends ... on the display font
      - rspeer 10y ago
        Well, when you're writing a console program, you can assume your font is monospaced and you can assume you have a function like wcwidth() that will tell you how many monospaced cells a string will take up. I know that eev.ee points out that wcwidth() is inconsistent (emoji screwed it up, then Unicode 9 gave a consistent width to emoji, but most libraries aren't on Unicode 9 so the situation is currently more inconsistent than before). But the situation is a lot better than just saying "I don't know, blame fonts".
nabla9 10y ago
Joel's article confuses characters and code points and can be very misleading. >In Unicode, a letter maps to something called a code point which is still just a theoretical concept. This is incorrect. Letter or character (user perceived character) can be multiple code points. Code points don't have intrinsic meaning across languages. If you want to count number of letters in a text or edit text, you must work at grapheme cluster level. Absolute minimum about Unicode must at least mention grapheme clusters and user perceived characters.
- Manishearth 10y ago
  I wrote about this a bit in http://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning-to-unicode-code-points/ http://manishearth.github.io/blog/2017/01/14/stop-ascribing-...
- blipmusic 10y ago
  Yes! Vietnamese and International Phonetic Alphabet says hello (among others).

by andrewl 10y ago

thomk 10y ago

Haha, I was going to post the same thing! If you like Joel Spolsky you might like this video where he angrily abuses everyone about how bad they are at Excel and shows you some very cool tricks: https://www.youtube.com/watch?v=0nbkaYsR94c&t=1206s https://www.youtube.com/watch?v=0nbkaYsR94c&t=1206s

drewmate 10y ago
I considered myself reasonably good at Excel prior to watching this. Thanks for posting this; I learned a lot!
deleted 10y ago
[deleted]
slededit 10y ago
For most of that I was getting so annoyed "JUST USE A TABLE!!!", but he got there in the end. For those that don't know Joel Spolsky used to be a PM on Excel back in the early 90s.

rspeer 10y ago

Can an article be honorably retired? Joel's article was important 14 years ago, but that was a time when it was not clear that UTF-8 was going to become the default encoding of the Internet, and a time before emoji. In 2003, codepoints outside the Basic Multilingual Plane could be safely ignored unless you were a historical linguist; now, people use them every day. This article is much better at telling you relevant things (with nice data-driven visualizations!) about how Unicode is used now.

ryandrake 10y ago
The content might be a little long in the tooth, but the exasperation about how programmers don't seem to give a shit about character encoding is, sadly, still relevant today. Unicode has been around since, what, the 90's? And UTF-8 has pretty much won the encoding war since 2008 or so. Yet, I still encounter software that fails to handle anything but ASCII, and it's 2017! Pick a random open source library, or look at a random company's software, and you're likely to find software that doesn't handle character encodings well. I think web software tends to be much better than desktop/mobile/embedded for some reason. Maybe an artifact of web guys tending to work with more recent/modern libraries and frameworks. Who's seen this before: You're getting up to speed on your new company's legacy code base. Q: "Hey, senior developer, This API takes a string as an argument and we need to display it, fine. What encoding are we using?" A: ¯\_(ツ)_/¯
- stevekemp 10y ago
  I don't disagree, but when you're an old-school developer it's hard to get this stuff right! I wrote a console-based mail client, and I struggled with UTF for months before it was all done, and even then there were niggles caused by libraries - for example passing a C++ string to a Lua script, and then using the `string:len()` lua function would return the number of bytes, rather than the length of the rendered string. Something I had to work around: https://github.com/lumail/lumail2/issues/277 https://github.com/lumail/lumail2/issues/277
  - chrisseaton 10y ago
    > rather than the length of the rendered string Do unicode strings really have a rendered length? https://eev.ee/blog/2015/09/12/dark-corners-of-unicode/#combining-characters-and-character-width https://eev.ee/blog/2015/09/12/dark-corners-of-unicode/#comb... > So the “length” of ... characters depends ... on the display font
    - rspeer 10y ago
      Well, when you're writing a console program, you can assume your font is monospaced and you can assume you have a function like wcwidth() that will tell you how many monospaced cells a string will take up. I know that eev.ee points out that wcwidth() is inconsistent (emoji screwed it up, then Unicode 9 gave a consistent width to emoji, but most libraries aren't on Unicode 9 so the situation is currently more inconsistent than before). But the situation is a lot better than just saying "I don't know, blame fonts".

nabla9 10y ago

Joel's article confuses characters and code points and can be very misleading. >In Unicode, a letter maps to something called a code point which is still just a theoretical concept. This is incorrect. Letter or character (user perceived character) can be multiple code points. Code points don't have intrinsic meaning across languages. If you want to count number of letters in a text or edit text, you must work at grapheme cluster level. Absolute minimum about Unicode must at least mention grapheme clusters and user perceived characters.

Manishearth 10y ago
I wrote about this a bit in http://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning-to-unicode-code-points/ http://manishearth.github.io/blog/2017/01/14/stop-ascribing-...
blipmusic 10y ago
Yes! Vietnamese and International Phonetic Alphabet says hello (among others).