4 ms·
One large source of byte order marks in utf8 is Windows. In MS DOS and later windows, 8 bit encoded files are assumed to be in the system code page, which to en
by superjan 5y ago
One large source of byte order marks in utf8 is Windows. In MS DOS and later windows, 8 bit encoded files are assumed to be in the system code page, which to enable all the worlds writing systems varies from country to country. When utf8 came along, Microsoft tools disambiguated those from the local code page by prefixing them with a byte order mark. They also do this in (for instance) the .net framework Xml libraries(by default). I don’t know what .net core does. I suppose it made sense at the time but I’m sure they regret this by now.
- superjan 5y agoAnd, I bet a significant portion of the offending CSV files are from excel. Excel is really annoying because it also silently localizes CSV files: my language uses the comma as a decimal separator, so excel wil switch to semicolon for the delimiter.