4 ms·

> The dataset started out as a Microsoft Excel file, presumably in Windows-1252 encoding. This was converted into a CSV, then loaded into the R environment for

> The dataset started out as a Microsoft Excel file, presumably in Windows-1252 encoding. This was converted into a CSV, then loaded into the R environment for adding additional information required by GBIF, then exported from R as a text file with the command option fileEncoding = "UTF-8". Quite a journey. The ultimate culprit was an R text cleaning function but I wonder why the Excel sheet was in Windows-1252 encoding and can't R import Excel files directly?

tialaramex 7y ago
Much of Microsoft Office for Windows dates to an awkward period when it was apparent that ASCII isn't enough, but UTF-8 isn't yet the obvious winner. In particular there's a period where you get "strings" that aren't actually text as we'd understand it but instead a sequence of glyph numbers for a typeface. So instead of ASCII's "A" or Unicode's U+0041 LATIN CAPITAL LETTER A you're just encoding that you want whatever the typeface named "Typewriter Sans" has put in slot number 65. Maybe it's a capital A and maybe it isn't. This works pretty well on one Windows PC, or even a LAN full of identically configured Windows PCs. You can see why it was at least superficially attractive to Windows application programmers. Got a niche that needs Bulgarian ? No problem, the software doesn't care what the glyphs "mean", just define how to type the codes in and you're done. But then somebody tries to open the Excel file with the Bulgarian text in "Steve's Bulgarian" font on a Mac and it doesn't work, it's gibberish. Oh dear. You have to install "Steve's Bulgarian" on the Mac and it look better although it isn't quite exactly right. Today this is obviously completely insane, but by then the problem is backwards compatibility.
- peterburkimsher 7y ago
  Just some specific dates. Microsoft Office was released in 1989 [1], and came out on the Mac before it was ported to Windows. UTF-8 was designed in 1992, on a placemat [2]. It has been dominant on the web [3] since 2009, which seems like ages ago now, but is really quite recent. [1] https://en.wikipedia.org/wiki/Microsoft_Office https://en.wikipedia.org/wiki/Microsoft_Office [2] https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt [3] https://en.wikipedia.org/wiki/UTF-8#/media/File:Utf8webgrowth.svg https://en.wikipedia.org/wiki/UTF-8#/media/File:Utf8webgrowt...
  - mkozlows 7y ago
    That's a misleading date for UTF-dominance -- as the link notes, it's counting things as ASCII if they only have ASCII characters, no matter what the encoding was specified as, and ASCII is a subset of UTF-8. I'd put UTF-8's real prominence with the popularity of XML (where it was the default encoding). That was specified in 1998, and ubiquitous within a few years. Nobody at the height of XML would have tried doing a non-UTF design for anything.
    - db48x 7y ago
      XML is almost UTF-8, but some characters were disallowed in XML 1.0, and it wasn't fixed until later. This still causes bugs where I work, here in 2019.
  - caf 7y ago
    Microsoft Office was initially just a bundle of separate programs, the constituent applications are even older (though Excel still debuted on the Mac first, in 1985).
  - forgotmypw 7y ago
    My Windows ME with IE 5.5 supports UTF-8.
- tyingq 7y ago
  A pretty good overview of the awkward period with Excel: https://donatstudios.com/CSV-An-Encoding-Nightmare https://donatstudios.com/CSV-An-Encoding-Nightmare
- jancsika 7y ago
  There needs to be an "alternateTimelineHackerNews" subreddit where each post picks a historical technology, removes it from history, and then tries to argue convincingly for the status quo because that now-missing technology could not possibly be created. I'd love to read a thread of apology theater for these "glyph slots" and how something like unicode/utf-8 would never work as a replacement. :)
  - db48x 7y ago
    Well, for one thing you'd have to form some kind of giant committee to decide on what characters actually exist, and assign them all unique numbers. Can you imagine how ridiculous that would be?
    - AnIdiotOnTheNet 7y ago
      Said committee would be mired in politics of what should and shouldn't be included. Cuneiform? Klingon? Clip art? They are all technically human communication.
      - tialaramex 7y ago
        The comment you're replying to is joking about the Unicode Technical Committee, which of course exists and really does what was described. So it's sarcasm. But the Technical Committee doesn't spend very much time worrying about whether it should include Cuneiform (yes), Klingon (no) or Clip Art (mostly no but there are some things you might think of as "clip art" that are in ISO-10646 / Unicode).
        AnIdiotOnTheNet 7y ago
        Yes, I'm aware of this. My post is also made in jest along the same lines, albeit with a grain of genuine distaste for what is and isn't allowed in Unicode.
      - db48x 7y ago
        So much better to have local control instead of ending up beholden to a committee of busybodies.
- ACS_Solver 7y ago
  The whole encoding thing was so annoying. I had experience with this when learning computing, in the 90s in Latvia. The common use case was needing both Latvian and Russian languages, at least the ability to read them. Neither was convenient. Russian had its own Cyrillic encodings, but of course there wasn't a single one. Much of the time it was Windows-1251, which you can still run into when browsing the Russian internet even if UTF-8 dominates. But sometimes it would instead be KOI8-R, a different encoding. So you had to know how to convert encodings with special tools, and of course you needed fonts that had the right glyphs in the first place. As an extra fun note, Windows-1251 uses the code point 0xFF for the letter я, which could break poorly-written software that used 0xFF as a sentinel value of some kind. And then on top of that, there was dealing with Latvian. It's a small and relatively obscure language, using an extended Latin alphabet, with Latvian letters like ā, ž, ļ and ķ. So that meant another encoding, Windows-1257, and needing the fonts - some Latvian letters like ž or č existed in many fonts due to also existing in some bigger languages, but ā or ļ were less common. Unlike Russian, Latvian documents would at least be understandable if rendered with the wrong encoding, but input was a bigger problem. Windows had no out-of-the-box support until Windows XP. So from Windows 3.1 up until XP, additional third-party software was required to type Latvian, and the software would do things like hooking into the keyboard driver, which was of course not always side-effect free. There are things that have gotten worse in computing, but the victory of Unicode is a huge step forward. Up until fairly recently, software was very obviously geared towards the 26-letter Latin alphabet, which means English (or Malay).

tialaramex 7y ago

Much of Microsoft Office for Windows dates to an awkward period when it was apparent that ASCII isn't enough, but UTF-8 isn't yet the obvious winner. In particular there's a period where you get "strings" that aren't actually text as we'd understand it but instead a sequence of glyph numbers for a typeface. So instead of ASCII's "A" or Unicode's U+0041 LATIN CAPITAL LETTER A you're just encoding that you want whatever the typeface named "Typewriter Sans" has put in slot number 65. Maybe it's a capital A and maybe it isn't. This works pretty well on one Windows PC, or even a LAN full of identically configured Windows PCs. You can see why it was at least superficially attractive to Windows application programmers. Got a niche that needs Bulgarian ? No problem, the software doesn't care what the glyphs "mean", just define how to type the codes in and you're done. But then somebody tries to open the Excel file with the Bulgarian text in "Steve's Bulgarian" font on a Mac and it doesn't work, it's gibberish. Oh dear. You have to install "Steve's Bulgarian" on the Mac and it look better although it isn't quite exactly right. Today this is obviously completely insane, but by then the problem is backwards compatibility.

peterburkimsher 7y ago
Just some specific dates. Microsoft Office was released in 1989 [1], and came out on the Mac before it was ported to Windows. UTF-8 was designed in 1992, on a placemat [2]. It has been dominant on the web [3] since 2009, which seems like ages ago now, but is really quite recent. [1] https://en.wikipedia.org/wiki/Microsoft_Office https://en.wikipedia.org/wiki/Microsoft_Office [2] https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt [3] https://en.wikipedia.org/wiki/UTF-8#/media/File:Utf8webgrowth.svg https://en.wikipedia.org/wiki/UTF-8#/media/File:Utf8webgrowt...
- mkozlows 7y ago
  That's a misleading date for UTF-dominance -- as the link notes, it's counting things as ASCII if they only have ASCII characters, no matter what the encoding was specified as, and ASCII is a subset of UTF-8. I'd put UTF-8's real prominence with the popularity of XML (where it was the default encoding). That was specified in 1998, and ubiquitous within a few years. Nobody at the height of XML would have tried doing a non-UTF design for anything.
  - db48x 7y ago
    XML is almost UTF-8, but some characters were disallowed in XML 1.0, and it wasn't fixed until later. This still causes bugs where I work, here in 2019.
- caf 7y ago
  Microsoft Office was initially just a bundle of separate programs, the constituent applications are even older (though Excel still debuted on the Mac first, in 1985).
- forgotmypw 7y ago
  My Windows ME with IE 5.5 supports UTF-8.
tyingq 7y ago
A pretty good overview of the awkward period with Excel: https://donatstudios.com/CSV-An-Encoding-Nightmare https://donatstudios.com/CSV-An-Encoding-Nightmare
jancsika 7y ago
There needs to be an "alternateTimelineHackerNews" subreddit where each post picks a historical technology, removes it from history, and then tries to argue convincingly for the status quo because that now-missing technology could not possibly be created. I'd love to read a thread of apology theater for these "glyph slots" and how something like unicode/utf-8 would never work as a replacement. :)
- db48x 7y ago
  Well, for one thing you'd have to form some kind of giant committee to decide on what characters actually exist, and assign them all unique numbers. Can you imagine how ridiculous that would be?
  - AnIdiotOnTheNet 7y ago
    Said committee would be mired in politics of what should and shouldn't be included. Cuneiform? Klingon? Clip art? They are all technically human communication.
    - tialaramex 7y ago
      The comment you're replying to is joking about the Unicode Technical Committee, which of course exists and really does what was described. So it's sarcasm. But the Technical Committee doesn't spend very much time worrying about whether it should include Cuneiform (yes), Klingon (no) or Clip Art (mostly no but there are some things you might think of as "clip art" that are in ISO-10646 / Unicode).
      - AnIdiotOnTheNet 7y ago
        Yes, I'm aware of this. My post is also made in jest along the same lines, albeit with a grain of genuine distaste for what is and isn't allowed in Unicode.
    - db48x 7y ago
      So much better to have local control instead of ending up beholden to a committee of busybodies.
ACS_Solver 7y ago
The whole encoding thing was so annoying. I had experience with this when learning computing, in the 90s in Latvia. The common use case was needing both Latvian and Russian languages, at least the ability to read them. Neither was convenient. Russian had its own Cyrillic encodings, but of course there wasn't a single one. Much of the time it was Windows-1251, which you can still run into when browsing the Russian internet even if UTF-8 dominates. But sometimes it would instead be KOI8-R, a different encoding. So you had to know how to convert encodings with special tools, and of course you needed fonts that had the right glyphs in the first place. As an extra fun note, Windows-1251 uses the code point 0xFF for the letter я, which could break poorly-written software that used 0xFF as a sentinel value of some kind. And then on top of that, there was dealing with Latvian. It's a small and relatively obscure language, using an extended Latin alphabet, with Latvian letters like ā, ž, ļ and ķ. So that meant another encoding, Windows-1257, and needing the fonts - some Latvian letters like ž or č existed in many fonts due to also existing in some bigger languages, but ā or ļ were less common. Unlike Russian, Latvian documents would at least be understandable if rendered with the wrong encoding, but input was a bigger problem. Windows had no out-of-the-box support until Windows XP. So from Windows 3.1 up until XP, additional third-party software was required to type Latvian, and the software would do things like hooking into the keyboard driver, which was of course not always side-effect free. There are things that have gotten worse in computing, but the victory of Unicode is a huge step forward. Up until fairly recently, software was very obviously geared towards the 26-letter Latin alphabet, which means English (or Malay).