7 ms·

Developers should assume UTF-8 for text files going forward. UTF-8 should have no BOM. It is the default. And there are no undefined Byte sequences that need a

Developers should assume UTF-8 for text files going forward. UTF-8 should have no BOM. It is the default. And there are no undefined Byte sequences that need an Order. Requiring a UTF-8 BOM just destroys the happy planned property that ASCII-is-UTF8. Why spoil that good work? Others variants of Unicode have BOMs, e.g. UTF-16BE. We know CJK languages need UTF-16 for compression. The BOM is only a couple more bytes. No problem, so far so good. But there are old files, that are in 'platform encoding'. Fine, let there be an OS 'locale', that has a default encoding. That default can be overridden with another OS 'encoding' variable. And that can be overridden by an application arg. And there may be a final default for a specific application, that is only ever used with one language encoding. Then individual files can override all of the above ... Text file (serialization) formats should define handling of optional BOM followed by an ASCII header that defines the encoding of the body that follows. One can also imagine .3 filetypes that have a unique or default encoding, with values maintained by, say, IANA (like MIME types). XML got this right. JSON and CSV are woefully neglectful, almost to the point of criminal liability. But in the absence of all of the above, the default-default-default-default-default is UTF-8. We are talking about the future, not the past. Design for UTF-8 default, and BOMs for other Unicodes. Microsoft should have defined BOMs for Windows-Blah encodings, not for UTF-8! When the whole world looks to the future, Microsoft will follow. Eventually. Reluctantly.

calpaterson 2y ago
If csv files bring criminal liability then I am guilty. Sidenote: this particular criminal conspiracy is open to potential new members. Please join the Committee To Keep Csv Evil: https://discord.gg/uqu4BkNP5G https://discord.gg/uqu4BkNP5G Jokes aside, talking about the future is grand but the problem is that data was written in the past and we need to read it in the present. That means that you do have to detect encoding and you can't just decide that the world runs on UTF-8. Even today, mainland China does not use UTF-8 and is not, as far as I know, in the process of switching. I understand UTF-8 is mostly fine even for east asian languages though - and bytes are cheap
- mikhailfranco 2y ago
  I explained how old files could be read. Let me reframe it as a Schelling Point [1] - the uncoordinated coordination problem. You arrange to meet your file on a certain day in New York, no place or time were mentioned. When and where will you go? It seems impossible. But perhaps you go at noon, to the UTF-8 Building in midtown Manhattan. Are you there now? [1] https://en.wikipedia.org/wiki/Focal_point_(game_theory) https://en.wikipedia.org/wiki/Focal_point_(game_theory)
  - calpaterson 2y ago
    I'm sorry, quite right - you did. Using the computer locale. The trouble is that people transmit data from one computer to another and so from one locale to another. And sadly, they do not always set the character encoding header correctly, if they even know what that is. I mean take csvbase's case. It has to accept csv files from anyone. And christ preserve me, they aren't going to label them as "Win-1252" or "UTF-16" or whatever. There is no alternative but statistical detection. And there is good evidence that this solution is fairly satisfactory, because millions of computers are using it right now! csvbase uploads run into more problems with trailing commas than with character detection getting it wrong at this point, that is your "Schelling Point" I'm afraid!
    - mikhailfranco 2y ago
      So trailing commas are another failure. That issue is mildly entangled with headers. Perhaps start making a list...
    - mikhailfranco 2y ago
      HTTP actually does quite a good job of providing headers containing MIME type and encoding. There is a little work to get the default (e.g. HTML and XML are different), and decide on the case where the XML payload encoding is different to the HTTP transport encoding (e.g. perhaps XML parsers need a way to override the embedded header). So we end up at another plausible future-directed design decision: computer-computer communication should use HTTP. I think many systems have ended up there already, perhaps prompted by the issues we have discussed. Moral: good specs attract usage; bad, incomprehensible or inconsistently implemented specs fade away.
teknopaul 2y ago
Imho Presume 8 bit, encapsulting 7bit usacsii. That includes utf8 and many many other encodings. Don't interpret user supplied strings at all. Defines max lengths as byte lengths. Remain agnostic of encoding. Especially in libraries. It's easier than people think it is thanks to some very clever people's work a long time ago.
deleted 2y ago
[deleted]
Pet_Ant 2y ago
> Requiring a UTF-8 BOM just destroys the happy planned property that ASCII-is-UTF8. Why spoil that good work? So that you know you are dealing with UTF-8. Assuming ASCII only works if you are only dealing with English texts and data.
- remram 2y ago
  Who is assuming ASCII?
  - bobmcnamara 2y ago
    Much code incorrectly assumes that ASCII can be rendered in any code page.
- themerone 2y ago
  The BOM only tells you that your are not dealing with ASCII. You could encounter anything after it.
WorldMaker 2y ago
> We know CJK languages need UTF-16 for compression. My understanding is that it is for the opposite of compression: it saves memory when uncompressed versus the UTF-8 surrogates needing more bytes. My understanding is that UTF-8 surrogates compress pretty well as they have common patterns that form dictionary "words" just as easily anything else. UTF-8 seems to be winning in the long run for even CJK and astral plane languages on disk and the operating systems and applications that were preferring UTF-16 in memory are mostly only doing so out of backwards compatibility and are themselves often using more UTF-8 buffers internally as those reflect the files at rest. (.NET has a backwards compatibility based on using UTF-16 codepoint strings by default but has more and more UTF-8 only pathways and has some interesting compile time options now to use UTF-8 only today. Python 3 made the choice that UTF-8 was the only string format to support, even with input from CJK communities. UTF-8 really does seem to be slowly winning everything.) > JSON and CSV are woefully neglectful, As the article also points out, JSON probably got it right: UTF-8 only and BOM is an error (because UTF-8) (but parsers are allowed to gently ignore that error if they wish). https://www.rfc-editor.org/rfc/rfc8259#section-8.1 https://www.rfc-editor.org/rfc/rfc8259#section-8.1 That seems to be the way forward for new text-based formats that only care about backward compatibility with low byte ASCII: UTF-8 only, no BOM. UTF-8 (unlike UTF-16 and missing reservations for some of its needed surrogates) is infinitely expandable if we ever do find a reason to extend past the "astral plane". (Anyone still working in CSV by choice is maybe guilty of criminal liability though. I still think the best thing Excel could do to help murder CSV is give us a file extension to force Excel to open a JSON file, like .XLJSON. Every time I've resorted to CSV has been because "the user needs to double click the file and open in Excel". Excel has great JSON support, it just won't let you double click a file for it, which is the only problem, because no business executive wants the training on "Data > From JSON" no matter how prominent in the ribbon that tool is.) > When the whole world looks to the future, Microsoft will follow. That ship is turning slowly. Windows backward compatibility guarantees likely mean that Windows will always have some UTF-16, but the terminals in Windows now correctly default to UTF-8 (since Windows 10) and even .NET with its compatibility decrees is more "UTF-8 native" than ever (especially when compiling for running on Linux, which is several layers of surprise for anyone that was around in the era where Microsoft picked UCS-2 as its one true format in the first place).
- 2y ago
PaulHoule 2y ago
Programming languages have lumbered slowly towards UTF-8 by default but from time to time you find an environment with a corrupted character encoding. I worked at an AI company that was ahead of its time (actually I worked at a few of those) where the data scientists had a special talent for finding Docker images with strange configurations so all the time I'd find out one particular container was running a Hungarian or other wrong charset. (And that's the problem with Docker... People give up on being in control of their environment and just throw in five different kitchen sinks and it works... Sometimes)
jrochkind1 2y ago
The specific use case the OP author was focusing on was CSV. (A format which has no place to signal the encoding inline). They noted that, to this day, Windows Excel will output CSV in Win-1252. (And the user doing the CSV export has probably never even heard of encodings). If you assume UTF-8, you will have corrupted text. I agree that I'm mad about Excel outputting Win-1252 CSV by default.
- out_of_protocol 2y ago
  More pressure on M$ - faster the change. Also, almost any software nowadays assuming UTF8 for plaintext anyway
  - jrochkind1 2y ago
    You are suggesting that if software developers willfully refuse to implement measures to detect CP-1252 in CSVs, instead insisting on assumign UTF-8 even though they know that will result in lots of corruption with data from the most popular producer of CSVs -- you are suggesting that will be pressure on MS to make it output in UTF-8 by default instead? If the world worked that way, it would be a very different world than the one we've got.
    - out_of_protocol 2y ago
      If you think you can detect only cp-1252 i have news for you. This is 1-byte encoding, it can't fit much. So, be prepared for whole zoo of other 1-byte encodings from that era (e.g. cyrillic letters - welcome to cp-1251, where everything above first 128 chars have totally different meaning). Writing encoding detector is not easy at all. Chance of guessing wrong is high. I'm glad most of the world (but not excel) moved on away from that can of worms > You are suggesting that if software developers willfully refuse to implement it will happen anyway, sooner or later > will result in lots of corruption there are lots of corruption anyway, e.g euro sign, some european letters, whole other alphabets like cyrillic one). The most dangerous are subtle ones like german ẞ) P.S. I don't have office installed to check, but online version exports non cp-1252 chars as "?" . So nice P.P.S. Apple Numbers offers choice of encoding on export, with UTF-8 default. Google Sheets exports as UTF-8 without choice

calpaterson 2y ago

If csv files bring criminal liability then I am guilty. Sidenote: this particular criminal conspiracy is open to potential new members. Please join the Committee To Keep Csv Evil: https://discord.gg/uqu4BkNP5G https://discord.gg/uqu4BkNP5G Jokes aside, talking about the future is grand but the problem is that data was written in the past and we need to read it in the present. That means that you do have to detect encoding and you can't just decide that the world runs on UTF-8. Even today, mainland China does not use UTF-8 and is not, as far as I know, in the process of switching. I understand UTF-8 is mostly fine even for east asian languages though - and bytes are cheap

mikhailfranco 2y ago
I explained how old files could be read. Let me reframe it as a Schelling Point [1] - the uncoordinated coordination problem. You arrange to meet your file on a certain day in New York, no place or time were mentioned. When and where will you go? It seems impossible. But perhaps you go at noon, to the UTF-8 Building in midtown Manhattan. Are you there now? [1] https://en.wikipedia.org/wiki/Focal_point_(game_theory) https://en.wikipedia.org/wiki/Focal_point_(game_theory)
- calpaterson 2y ago
  I'm sorry, quite right - you did. Using the computer locale. The trouble is that people transmit data from one computer to another and so from one locale to another. And sadly, they do not always set the character encoding header correctly, if they even know what that is. I mean take csvbase's case. It has to accept csv files from anyone. And christ preserve me, they aren't going to label them as "Win-1252" or "UTF-16" or whatever. There is no alternative but statistical detection. And there is good evidence that this solution is fairly satisfactory, because millions of computers are using it right now! csvbase uploads run into more problems with trailing commas than with character detection getting it wrong at this point, that is your "Schelling Point" I'm afraid!
  - mikhailfranco 2y ago
    So trailing commas are another failure. That issue is mildly entangled with headers. Perhaps start making a list...
  - mikhailfranco 2y ago
    HTTP actually does quite a good job of providing headers containing MIME type and encoding. There is a little work to get the default (e.g. HTML and XML are different), and decide on the case where the XML payload encoding is different to the HTTP transport encoding (e.g. perhaps XML parsers need a way to override the embedded header). So we end up at another plausible future-directed design decision: computer-computer communication should use HTTP. I think many systems have ended up there already, perhaps prompted by the issues we have discussed. Moral: good specs attract usage; bad, incomprehensible or inconsistently implemented specs fade away.

teknopaul 2y ago

Imho Presume 8 bit, encapsulting 7bit usacsii. That includes utf8 and many many other encodings. Don't interpret user supplied strings at all. Defines max lengths as byte lengths. Remain agnostic of encoding. Especially in libraries. It's easier than people think it is thanks to some very clever people's work a long time ago.

deleted 2y ago

[deleted]

Pet_Ant 2y ago

> Requiring a UTF-8 BOM just destroys the happy planned property that ASCII-is-UTF8. Why spoil that good work? So that you know you are dealing with UTF-8. Assuming ASCII only works if you are only dealing with English texts and data.

remram 2y ago
Who is assuming ASCII?
- bobmcnamara 2y ago
  Much code incorrectly assumes that ASCII can be rendered in any code page.
themerone 2y ago
The BOM only tells you that your are not dealing with ASCII. You could encounter anything after it.

WorldMaker 2y ago

> We know CJK languages need UTF-16 for compression. My understanding is that it is for the opposite of compression: it saves memory when uncompressed versus the UTF-8 surrogates needing more bytes. My understanding is that UTF-8 surrogates compress pretty well as they have common patterns that form dictionary "words" just as easily anything else. UTF-8 seems to be winning in the long run for even CJK and astral plane languages on disk and the operating systems and applications that were preferring UTF-16 in memory are mostly only doing so out of backwards compatibility and are themselves often using more UTF-8 buffers internally as those reflect the files at rest. (.NET has a backwards compatibility based on using UTF-16 codepoint strings by default but has more and more UTF-8 only pathways and has some interesting compile time options now to use UTF-8 only today. Python 3 made the choice that UTF-8 was the only string format to support, even with input from CJK communities. UTF-8 really does seem to be slowly winning everything.) > JSON and CSV are woefully neglectful, As the article also points out, JSON probably got it right: UTF-8 only and BOM is an error (because UTF-8) (but parsers are allowed to gently ignore that error if they wish). https://www.rfc-editor.org/rfc/rfc8259#section-8.1 https://www.rfc-editor.org/rfc/rfc8259#section-8.1 That seems to be the way forward for new text-based formats that only care about backward compatibility with low byte ASCII: UTF-8 only, no BOM. UTF-8 (unlike UTF-16 and missing reservations for some of its needed surrogates) is infinitely expandable if we ever do find a reason to extend past the "astral plane". (Anyone still working in CSV by choice is maybe guilty of criminal liability though. I still think the best thing Excel could do to help murder CSV is give us a file extension to force Excel to open a JSON file, like .XLJSON. Every time I've resorted to CSV has been because "the user needs to double click the file and open in Excel". Excel has great JSON support, it just won't let you double click a file for it, which is the only problem, because no business executive wants the training on "Data > From JSON" no matter how prominent in the ribbon that tool is.) > When the whole world looks to the future, Microsoft will follow. That ship is turning slowly. Windows backward compatibility guarantees likely mean that Windows will always have some UTF-16, but the terminals in Windows now correctly default to UTF-8 (since Windows 10) and even .NET with its compatibility decrees is more "UTF-8 native" than ever (especially when compiling for running on Linux, which is several layers of surprise for anyone that was around in the era where Microsoft picked UCS-2 as its one true format in the first place).

2y ago

PaulHoule 2y ago

Programming languages have lumbered slowly towards UTF-8 by default but from time to time you find an environment with a corrupted character encoding. I worked at an AI company that was ahead of its time (actually I worked at a few of those) where the data scientists had a special talent for finding Docker images with strange configurations so all the time I'd find out one particular container was running a Hungarian or other wrong charset. (And that's the problem with Docker... People give up on being in control of their environment and just throw in five different kitchen sinks and it works... Sometimes)

jrochkind1 2y ago

The specific use case the OP author was focusing on was CSV. (A format which has no place to signal the encoding inline). They noted that, to this day, Windows Excel will output CSV in Win-1252. (And the user doing the CSV export has probably never even heard of encodings). If you assume UTF-8, you will have corrupted text. I agree that I'm mad about Excel outputting Win-1252 CSV by default.

out_of_protocol 2y ago
More pressure on M$ - faster the change. Also, almost any software nowadays assuming UTF8 for plaintext anyway
- jrochkind1 2y ago
  You are suggesting that if software developers willfully refuse to implement measures to detect CP-1252 in CSVs, instead insisting on assumign UTF-8 even though they know that will result in lots of corruption with data from the most popular producer of CSVs -- you are suggesting that will be pressure on MS to make it output in UTF-8 by default instead? If the world worked that way, it would be a very different world than the one we've got.
  - out_of_protocol 2y ago
    If you think you can detect only cp-1252 i have news for you. This is 1-byte encoding, it can't fit much. So, be prepared for whole zoo of other 1-byte encodings from that era (e.g. cyrillic letters - welcome to cp-1251, where everything above first 128 chars have totally different meaning). Writing encoding detector is not easy at all. Chance of guessing wrong is high. I'm glad most of the world (but not excel) moved on away from that can of worms > You are suggesting that if software developers willfully refuse to implement it will happen anyway, sooner or later > will result in lots of corruption there are lots of corruption anyway, e.g euro sign, some european letters, whole other alphabets like cyrillic one). The most dangerous are subtle ones like german ẞ) P.S. I don't have office installed to check, but online version exports non cp-1252 chars as "?" . So nice P.P.S. Apple Numbers offers choice of encoding on export, with UTF-8 default. Google Sheets exports as UTF-8 without choice