3 ms·

This is not good advice. Most software is not an "app," and even if it is, you at least need to be able to tell the difference between, e.g., ISO 8859-1, ISO 8

This is not good advice. Most software is not an "app," and even if it is, you at least need to be able to tell the difference between, e.g., ISO 8859-1, ISO 8859-15, and UTF-8. Otherwise you'll get text talking about cutting doorways 8œ feet tall or submitting a rÃ©sumÃ©. Any software that needs to process information effectively needs to understand character sets, including Unicode.

theamk 3y ago
Unless you deal with very old data, just use UTF-8 [0]. Mac terminal uses UTF-8 by default, most Linuxes use UTF-8 by default, and if you are stuck with Windows, there are libraries which let you use UTF-8 everywhere. And if you do deal with old data, still prefer to use UTF-8, and make sure you only translate character sets if user asks / they are explicitly specified. And for the love of god, _please_ don't just hardcode two common encodings (like 8859-1 and 8859-15)... I used to deal with encodings a lot, and the only time I actually had data loss from encoding problems was from misguided apps which assume 8859-1 (and tried to convert to ASCII by stripping accents). Please either assume UTF-8 or assume nothing and keep data as-is. And that's it, _most_ software does not need to know more. Let's look at some examples: - For file archiver, assume utf-8 everywhere and don't worry about encoding at all (unless you planning to deal with very old archives, in this case you'd need encoding support. But for something made from scratch, don't bother.) _Especially_ don't add any encoding support for "extract to console" function -- if I need a different encoding I can pipe into iconv myself, thank you very much. - For API client, you don't need to worry about encoding either.. A very old HTTP server might return data in something other than utf-8 but (1) your HTTP library likely handles it already (2) how many such servers are left, anyway? - For XML analyzer, use proper XML library, they will handle encodings for you. - For web dashboard or a database, keep everything in utf-8 and things will just work. - For ETL-like applications, many modern data sources are already in UTF-8.. and if they are not, convert everything to utf-8 ASAP. In other words: while it's useful to know that encodings exist, unless you are working with very old data or legacy systems, utf-8 is the only thing you need. [0] http://utf8everywhere.org/ http://utf8everywhere.org/
- torstenvl 3y ago
  Yeah. This is objectively terrible advice. Ideology is not an excuse for data loss.
  - theamk 3y ago
    Do you have any specific examples, or are you just saying general statements? Back in the day, I worked quite a bit various encodings (my language had 2 primary one and 2 secondary one, and it was a guess which one the text was), and the data loss usually happened from programs that tried to support encodings. When program would not touch encodings, there might be some mojibake and unreadable text, but you could generally fix things.. I had to write some scripts which changed encoding of filenames and fixup random database or five, but there were no data loss. It's the programs which were "able to tell the difference between, e.g., ISO 8859-1, ISO 8859-15, and UTF-8" which caused data loss. There were so many cases when I ended up with directory full of "?????????" or "aoeoaoao" files and that was it, there is no way to recover. So please don't add Unicode support unless you have to, it is not ideology but rather the results of hard practical experience.
    - torstenvl 3y ago
      > Unless you deal with very old data, just use UTF-8 > I had to write some scripts which changed encoding of filenames and fixup random database or five, but there were no data loss. If you do the first, it is impossible to do the second. Are you just disagreeing for the sake of being disagreeable? Why would you do that?

theamk 3y ago

Unless you deal with very old data, just use UTF-8 [0]. Mac terminal uses UTF-8 by default, most Linuxes use UTF-8 by default, and if you are stuck with Windows, there are libraries which let you use UTF-8 everywhere. And if you do deal with old data, still prefer to use UTF-8, and make sure you only translate character sets if user asks / they are explicitly specified. And for the love of god, _please_ don't just hardcode two common encodings (like 8859-1 and 8859-15)... I used to deal with encodings a lot, and the only time I actually had data loss from encoding problems was from misguided apps which assume 8859-1 (and tried to convert to ASCII by stripping accents). Please either assume UTF-8 or assume nothing and keep data as-is. And that's it, _most_ software does not need to know more. Let's look at some examples: - For file archiver, assume utf-8 everywhere and don't worry about encoding at all (unless you planning to deal with very old archives, in this case you'd need encoding support. But for something made from scratch, don't bother.) _Especially_ don't add any encoding support for "extract to console" function -- if I need a different encoding I can pipe into iconv myself, thank you very much. - For API client, you don't need to worry about encoding either.. A very old HTTP server might return data in something other than utf-8 but (1) your HTTP library likely handles it already (2) how many such servers are left, anyway? - For XML analyzer, use proper XML library, they will handle encodings for you. - For web dashboard or a database, keep everything in utf-8 and things will just work. - For ETL-like applications, many modern data sources are already in UTF-8.. and if they are not, convert everything to utf-8 ASAP. In other words: while it's useful to know that encodings exist, unless you are working with very old data or legacy systems, utf-8 is the only thing you need. [0] http://utf8everywhere.org/ http://utf8everywhere.org/

torstenvl 3y ago
Yeah. This is objectively terrible advice. Ideology is not an excuse for data loss.
- theamk 3y ago
  Do you have any specific examples, or are you just saying general statements? Back in the day, I worked quite a bit various encodings (my language had 2 primary one and 2 secondary one, and it was a guess which one the text was), and the data loss usually happened from programs that tried to support encodings. When program would not touch encodings, there might be some mojibake and unreadable text, but you could generally fix things.. I had to write some scripts which changed encoding of filenames and fixup random database or five, but there were no data loss. It's the programs which were "able to tell the difference between, e.g., ISO 8859-1, ISO 8859-15, and UTF-8" which caused data loss. There were so many cases when I ended up with directory full of "?????????" or "aoeoaoao" files and that was it, there is no way to recover. So please don't add Unicode support unless you have to, it is not ideology but rather the results of hard practical experience.
  - torstenvl 3y ago
    > Unless you deal with very old data, just use UTF-8 > I had to write some scripts which changed encoding of filenames and fixup random database or five, but there were no data loss. If you do the first, it is impossible to do the second. Are you just disagreeing for the sake of being disagreeable? Why would you do that?