3 ms·

Converting to UTF-8 from a simple codepage like ISO is easy. The difficult part is parsing UTF-8 with all its various behaviour changing characters. Take the B

Converting to UTF-8 from a simple codepage like ISO is easy. The difficult part is parsing UTF-8 with all its various behaviour changing characters. Take the BOM as an example. I work on backend Java projects for large banks. Over the years I fought with BOM on numerous times. For some reason 95% of software that says it is UTF-8 compliant is not. My modus operandi for dealing with it is to remove BOM on ingestion and only add it when the string leaves the system when we know the outside party absolutely requires it (though it should not...)

tialaramex 7y ago
People say if the only tool you have is a hammer, everything looks like a nail - but when the only tool you have is a left-handed can opener life gets _really weird_ and that's how we got the UTF-8 BOM UTF-8 BOM is largely a Microsoft idea, they've got a bunch of code that thinks in UCS-2 (now retrofitted to more or less pretend it knows UTF-16) and so it thinks about byte order when decoding text files, and from there a Byte Order Mark in files that don't have byte ordering seems like a reasonable idea. If the files actually _mean_ something then a UTF-8 BOM just introduces confusion. Lots of code I'm responsible for processes UTF-8 just fine, but if it handles say files full of key = value pairs and your file begins with a BOM, well, OK then, that first key starts with U+FEFF, weird choice but no reason we should disallow that. And of course that isn't what you wanted and so now Windows users are complaining I'm not "compatible".
- mark-r 7y ago
  The choice of character for the BOM isn't weird or random - it's a zero-width no-break space, which means it's supposed to be invisible even when your software is capable of displaying it. If you remove whitespace from the beginning and end of your keys you'll be fine. It seems arbitrary, but I don't think they could have made a better choice.
- marcosdumay 7y ago
  Shouldn't U+FEFF be whitespace?
  - tialaramex 7y ago
    Sure. So what? Text file formats aren't magically obliged to ignore leading whitespace just because that suits Microsoft. If my format would consider U+0009 TAB or U+0020 SPACE to be part of the key if placed at the start of the key why not U+FEFF ?

tialaramex 7y ago

People say if the only tool you have is a hammer, everything looks like a nail - but when the only tool you have is a left-handed can opener life gets _really weird_ and that's how we got the UTF-8 BOM UTF-8 BOM is largely a Microsoft idea, they've got a bunch of code that thinks in UCS-2 (now retrofitted to more or less pretend it knows UTF-16) and so it thinks about byte order when decoding text files, and from there a Byte Order Mark in files that don't have byte ordering seems like a reasonable idea. If the files actually _mean_ something then a UTF-8 BOM just introduces confusion. Lots of code I'm responsible for processes UTF-8 just fine, but if it handles say files full of key = value pairs and your file begins with a BOM, well, OK then, that first key starts with U+FEFF, weird choice but no reason we should disallow that. And of course that isn't what you wanted and so now Windows users are complaining I'm not "compatible".

mark-r 7y ago
The choice of character for the BOM isn't weird or random - it's a zero-width no-break space, which means it's supposed to be invisible even when your software is capable of displaying it. If you remove whitespace from the beginning and end of your keys you'll be fine. It seems arbitrary, but I don't think they could have made a better choice.
marcosdumay 7y ago
Shouldn't U+FEFF be whitespace?
- tialaramex 7y ago
  Sure. So what? Text file formats aren't magically obliged to ignore leading whitespace just because that suits Microsoft. If my format would consider U+0009 TAB or U+0020 SPACE to be part of the key if placed at the start of the key why not U+FEFF ?