2 ms·

More pressure on M$ - faster the change. Also, almost any software nowadays assuming UTF8 for plaintext anyway

jrochkind1 2y ago
You are suggesting that if software developers willfully refuse to implement measures to detect CP-1252 in CSVs, instead insisting on assumign UTF-8 even though they know that will result in lots of corruption with data from the most popular producer of CSVs -- you are suggesting that will be pressure on MS to make it output in UTF-8 by default instead? If the world worked that way, it would be a very different world than the one we've got.
- out_of_protocol 2y ago
  If you think you can detect only cp-1252 i have news for you. This is 1-byte encoding, it can't fit much. So, be prepared for whole zoo of other 1-byte encodings from that era (e.g. cyrillic letters - welcome to cp-1251, where everything above first 128 chars have totally different meaning). Writing encoding detector is not easy at all. Chance of guessing wrong is high. I'm glad most of the world (but not excel) moved on away from that can of worms > You are suggesting that if software developers willfully refuse to implement it will happen anyway, sooner or later > will result in lots of corruption there are lots of corruption anyway, e.g euro sign, some european letters, whole other alphabets like cyrillic one). The most dangerous are subtle ones like german ẞ) P.S. I don't have office installed to check, but online version exports non cp-1252 chars as "?" . So nice P.P.S. Apple Numbers offers choice of encoding on export, with UTF-8 default. Google Sheets exports as UTF-8 without choice
  - jrochkind1 2y ago
    OP is in fact about doing your best to detect encoding without proper tagging. It is not perfect and you won't have 100% accuracy. But it can get pretty decent (OP is literally about the techniques used to do so, statistically etc), and is necessary because of the actual world we live in. If just refusing to do it would get MS to export in UTF-8 by default that would of course have happened a long time ago!

You are suggesting that if software developers willfully refuse to implement measures to detect CP-1252 in CSVs, instead insisting on assumign UTF-8 even though they know that will result in lots of corruption with data from the most popular producer of CSVs -- you are suggesting that will be pressure on MS to make it output in UTF-8 by default instead? If the world worked that way, it would be a very different world than the one we've got.

out_of_protocol 2y ago
If you think you can detect only cp-1252 i have news for you. This is 1-byte encoding, it can't fit much. So, be prepared for whole zoo of other 1-byte encodings from that era (e.g. cyrillic letters - welcome to cp-1251, where everything above first 128 chars have totally different meaning). Writing encoding detector is not easy at all. Chance of guessing wrong is high. I'm glad most of the world (but not excel) moved on away from that can of worms > You are suggesting that if software developers willfully refuse to implement it will happen anyway, sooner or later > will result in lots of corruption there are lots of corruption anyway, e.g euro sign, some european letters, whole other alphabets like cyrillic one). The most dangerous are subtle ones like german ẞ) P.S. I don't have office installed to check, but online version exports non cp-1252 chars as "?" . So nice P.P.S. Apple Numbers offers choice of encoding on export, with UTF-8 default. Google Sheets exports as UTF-8 without choice
- jrochkind1 2y ago
  OP is in fact about doing your best to detect encoding without proper tagging. It is not perfect and you won't have 100% accuracy. But it can get pretty decent (OP is literally about the techniques used to do so, statistically etc), and is necessary because of the actual world we live in. If just refusing to do it would get MS to export in UTF-8 by default that would of course have happened a long time ago!