6 ms·

Default text file encoding being platform-dependent always drove me nuts. This is a welcome change. I also appreciate that they did not attempt to tackle files

Default text file encoding being platform-dependent always drove me nuts. This is a welcome change. I also appreciate that they did not attempt to tackle filesystem encoding here, which is a separate issue that drives me nuts, but separately.

layer8 3y ago
Historically it made sense, when most software was local-only, and text files were expected to be in the local encoding. Not just platform-dependent, but user’s preferred locale-dependent. This is also how the C standard library operates. For example, on Unix/Linux, using iso-8859-1 was common when using Western-European languages, and in Europe it became common to switch to iso-8859-15 after the Euro was introduced, because it contained the € symbol. UTF-8 only began to work flawlessly in the later aughts. Debian switched to it as the default with the Etch release in 2010.
- anthk 3y ago
  Emacs was amazing for that; builtin text encoders/decoders/transcoders for everything.
  - hollerith 3y ago
    My experience was that brittleness around text encoding in Emacs (versions 22 and 23 or so) was a constant source of annoyance for years. IIRC, the main way this brittleness bit me was that every time a buffer containing a non-ASCII character was saved, Emacs would engage me in a conversation (which I found tedious and distracting) about what coding system I would like to use to save the file, and I never found a sane way to configure it to avoid such conversations even after spending hours learning about how Emacs does coding systems: I simply had to wait (a year or 3) for a new version of Emacs in which the code for saving buffers worked better. I think some people like engaging in these conversations with their computers even though the conversations are very boring and repetitive and that such conversation-likers are numerous among Emacs users or at least Emacs maintainers.
    - anthk 3y ago
      TBH Gvim and most editors did the same on saving prompts, but for sure you could edit that under Emacs with M-x configure, and Emacs supported weirdly encoded files on the spot.
- Dylan16807 3y ago
  > Not just platform-dependent, but user’s preferred locale-dependent. Historically it made sense to be locale-dependent, but even then it was annoying to be platform-dependent. One is not a subset of the other.
  - hermitdev 3y ago
    > platform-dependent. It's 2024 and we still can't all agree on line endings. Mac vs Win vs Unix...
    - Y-bar 3y ago
      Mac OS and Unix agreed about twenty years ago to use the same ending: https://superuser.com/a/439443 https://superuser.com/a/439443
      - Dylan16807 3y ago
        By which time XP was already in the middle of releasing, so it was too late to get Windows on board. It's too bad, with a bit more planning and an earlier realization that Unicode cannot in fact fit into 16 bits then Windows might have used UTF-8 internally.
        jmb99 3y ago
        Unless I’m mistaken, Rhapsody (released 1997) used LF, not CR. At that point it was pretty clear Mac was moving towards Unix through NeXTSTEP, meaning every OS except windows would be using LF. Microsoft would’ve had around 6 years before the release of XP, and probably would’ve had time to start the transition with Win2K at the end of 1999.
        mixmastamyk 3y ago
        Every OS except the one that had 95% market share in late 90s. Apple was only propped up “Weekend at Bernies” style to appease regulators.
- da_chicken 3y ago
  It's still not that uncommon to see programs on Linux not understanding multibyte UTF-8. It's also true that essentially nothing on Linux supports the UTF-8 byte order mark. Yes, it's meaningless for UTF-8, but it is explicitly allowed in the specifications. Since Microsoft tends to always include a BOM in any flavor of Unicode, this means Linux often chokes on valid UTF-8 text files from Windows systems.
  - nerdponx 3y ago
    Interestingly, Python is one of those programs. You need to use the special "utf-8-sig" encoding for that, which is not prominently advertised anywhere in the documentation (but it is stated deep inside the "Unicode HOWTO"). I never understood why ignoring this special character requires a totally separate encoding.
    - duskwuff 3y ago
      > I never understood why ignoring this special character requires a totally separate encoding. Because the BOM is indistinguishable from the "real" UTF-8 encoding of U+FEFF (zero-width no-break space). Trimming that codepoint in the UTF-8 decoder means that some strings like "\uFEFF" can't be safely round-tripped; adding it in the encoder is invalid in many contexts.
  - tialaramex 3y ago
    The BOM cases are at best a consequence of trying to use poor quality Windows software to do stuff it's not suited to. It's true that in terms of Unicode text it's valid for a UTF-8 string to have a BOM, but just because that's true in the text itself doesn't magically change file formats which long pre-dated that. Most obviously shebang (the practice of writing #!/path/to/interpreter at the start of a script) is specifically defined on those first two bytes. It doesn't make any sense have a BOM here because that's not the format, and inventing a new rule later which says you can do it doesn't make that true, any more than in 2024 the German government can decide Germany didn't invade Poland in 1939, that's not how Time's Arrow works.
    - 3y ago
- andrewshadura 3y ago
  Etch came out in 2007, not 2010.
  - layer8 3y ago
    Ah, I had misremembered, and misread https://www.debian.org/releases/etch/ https://www.debian.org/releases/etch/.
avidphantasm 3y ago
Yeah, this has bitten me several times as soon as a people use the code on Windows.
fbdab103 3y ago
A different one that just bit me the other day was implicitly changing line endings. Local testing on my corporate laptop all went according to plan. Deploy to linux host and downstream application cannot consume it because it requires CRLF. Just one of those stupid little things you have to remember from time to time. Although, why does newly written software require a specific line terminator is a valid question.
Dwedit 3y ago
With system-default code pages on Windows, it's not only platform-dependent, it's also System Locale dependent. Windows badly dropped the ball here by not providing a simple opt-in way to make all the Ansi functions (TextOutA, etc) use the UTF-8 code page, until many many years later with the manifest file. This should have been a feature introduced in NT4 or Windows 98, not something that's put off until midway through Windows 10's development cycle.
- kevin_thibedeau 3y ago
  "UCS-2 is enough for anyone"
  - Dwedit 3y ago
    UCS-2 is why we have the WTF-8 encoding standard, which allows mismatched UTF-16 surrogate pairs to survive a round-trip through an 8-bit encoding. https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/
- sheepscreek 3y ago
  I suspect that is a symptom of Microsoft being an enormously large organization. Coordinating a change like this that cuts across all apps, services and drivers is monumental. Honestly it is quite refreshing to see them do it with Copilot integration across all things MS. I don’t use it though, just admire the valiant effort and focus it takes to pull off something like this. Of course - goes without saying, only works when the directive comes from all the way at the top. Otherwise there will be just too many conflicting incentives for any real change to happen. While I am on this topic - I want to mention Apple. It is absolutely bonkers how they have done exactly the is countless times. Like changing your entire platform architecture! It could have been like opening a can of worms but they knew what they were doing. Kudos to them. Also..(sorry, this is becoming a long post) civil and industrial engineering firms routinely pull off projects like that. But the point I wanted to emphasize is that it’s very uncommon in tech which prides on having decentralized and semi-autonomous teams vs centralized and highly aligned teams.
  - 3y ago

layer8 3y ago

Historically it made sense, when most software was local-only, and text files were expected to be in the local encoding. Not just platform-dependent, but user’s preferred locale-dependent. This is also how the C standard library operates. For example, on Unix/Linux, using iso-8859-1 was common when using Western-European languages, and in Europe it became common to switch to iso-8859-15 after the Euro was introduced, because it contained the € symbol. UTF-8 only began to work flawlessly in the later aughts. Debian switched to it as the default with the Etch release in 2010.

anthk 3y ago
Emacs was amazing for that; builtin text encoders/decoders/transcoders for everything.
- hollerith 3y ago
  My experience was that brittleness around text encoding in Emacs (versions 22 and 23 or so) was a constant source of annoyance for years. IIRC, the main way this brittleness bit me was that every time a buffer containing a non-ASCII character was saved, Emacs would engage me in a conversation (which I found tedious and distracting) about what coding system I would like to use to save the file, and I never found a sane way to configure it to avoid such conversations even after spending hours learning about how Emacs does coding systems: I simply had to wait (a year or 3) for a new version of Emacs in which the code for saving buffers worked better. I think some people like engaging in these conversations with their computers even though the conversations are very boring and repetitive and that such conversation-likers are numerous among Emacs users or at least Emacs maintainers.
  - anthk 3y ago
    TBH Gvim and most editors did the same on saving prompts, but for sure you could edit that under Emacs with M-x configure, and Emacs supported weirdly encoded files on the spot.
Dylan16807 3y ago
> Not just platform-dependent, but user’s preferred locale-dependent. Historically it made sense to be locale-dependent, but even then it was annoying to be platform-dependent. One is not a subset of the other.
- hermitdev 3y ago
  > platform-dependent. It's 2024 and we still can't all agree on line endings. Mac vs Win vs Unix...
  - Y-bar 3y ago
    Mac OS and Unix agreed about twenty years ago to use the same ending: https://superuser.com/a/439443 https://superuser.com/a/439443
    - Dylan16807 3y ago
      By which time XP was already in the middle of releasing, so it was too late to get Windows on board. It's too bad, with a bit more planning and an earlier realization that Unicode cannot in fact fit into 16 bits then Windows might have used UTF-8 internally.
      - jmb99 3y ago
        Unless I’m mistaken, Rhapsody (released 1997) used LF, not CR. At that point it was pretty clear Mac was moving towards Unix through NeXTSTEP, meaning every OS except windows would be using LF. Microsoft would’ve had around 6 years before the release of XP, and probably would’ve had time to start the transition with Win2K at the end of 1999.
        mixmastamyk 3y ago
        Every OS except the one that had 95% market share in late 90s. Apple was only propped up “Weekend at Bernies” style to appease regulators.
da_chicken 3y ago
It's still not that uncommon to see programs on Linux not understanding multibyte UTF-8. It's also true that essentially nothing on Linux supports the UTF-8 byte order mark. Yes, it's meaningless for UTF-8, but it is explicitly allowed in the specifications. Since Microsoft tends to always include a BOM in any flavor of Unicode, this means Linux often chokes on valid UTF-8 text files from Windows systems.
- nerdponx 3y ago
  Interestingly, Python is one of those programs. You need to use the special "utf-8-sig" encoding for that, which is not prominently advertised anywhere in the documentation (but it is stated deep inside the "Unicode HOWTO"). I never understood why ignoring this special character requires a totally separate encoding.
  - duskwuff 3y ago
    > I never understood why ignoring this special character requires a totally separate encoding. Because the BOM is indistinguishable from the "real" UTF-8 encoding of U+FEFF (zero-width no-break space). Trimming that codepoint in the UTF-8 decoder means that some strings like "\uFEFF" can't be safely round-tripped; adding it in the encoder is invalid in many contexts.
- tialaramex 3y ago
  The BOM cases are at best a consequence of trying to use poor quality Windows software to do stuff it's not suited to. It's true that in terms of Unicode text it's valid for a UTF-8 string to have a BOM, but just because that's true in the text itself doesn't magically change file formats which long pre-dated that. Most obviously shebang (the practice of writing #!/path/to/interpreter at the start of a script) is specifically defined on those first two bytes. It doesn't make any sense have a BOM here because that's not the format, and inventing a new rule later which says you can do it doesn't make that true, any more than in 2024 the German government can decide Germany didn't invade Poland in 1939, that's not how Time's Arrow works.
  - 3y ago
andrewshadura 3y ago
Etch came out in 2007, not 2010.
- layer8 3y ago
  Ah, I had misremembered, and misread https://www.debian.org/releases/etch/ https://www.debian.org/releases/etch/.

avidphantasm 3y ago

Yeah, this has bitten me several times as soon as a people use the code on Windows.

fbdab103 3y ago

A different one that just bit me the other day was implicitly changing line endings. Local testing on my corporate laptop all went according to plan. Deploy to linux host and downstream application cannot consume it because it requires CRLF. Just one of those stupid little things you have to remember from time to time. Although, why does newly written software require a specific line terminator is a valid question.

Dwedit 3y ago

With system-default code pages on Windows, it's not only platform-dependent, it's also System Locale dependent. Windows badly dropped the ball here by not providing a simple opt-in way to make all the Ansi functions (TextOutA, etc) use the UTF-8 code page, until many many years later with the manifest file. This should have been a feature introduced in NT4 or Windows 98, not something that's put off until midway through Windows 10's development cycle.

kevin_thibedeau 3y ago
"UCS-2 is enough for anyone"
- Dwedit 3y ago
  UCS-2 is why we have the WTF-8 encoding standard, which allows mismatched UTF-16 surrogate pairs to survive a round-trip through an 8-bit encoding. https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/
sheepscreek 3y ago
I suspect that is a symptom of Microsoft being an enormously large organization. Coordinating a change like this that cuts across all apps, services and drivers is monumental. Honestly it is quite refreshing to see them do it with Copilot integration across all things MS. I don’t use it though, just admire the valiant effort and focus it takes to pull off something like this. Of course - goes without saying, only works when the directive comes from all the way at the top. Otherwise there will be just too many conflicting incentives for any real change to happen. While I am on this topic - I want to mention Apple. It is absolutely bonkers how they have done exactly the is countless times. Like changing your entire platform architecture! It could have been like opening a can of worms but they knew what they were doing. Kudos to them. Also..(sorry, this is becoming a long post) civil and industrial engineering firms routinely pull off projects like that. But the point I wanted to emphasize is that it’s very uncommon in tech which prides on having decentralized and semi-autonomous teams vs centralized and highly aligned teams.
- 3y ago