3 ms·
You basically have two approaches: 1. Write your CSV parser with an assumption that the data is ASCII compatible (this means it works with either UTF-8 or Lati
by burntsushi 9y ago
You basically have two approaches:
1. Write your CSV parser with an assumption that the data is ASCII compatible (this means it works with either UTF-8 or Latin-1 out of the box, possibly modulo non-ASCII meta characters). To support additional encodings---such as UTF-16---either the CSV library or the caller must transcode first.
2. Write your CSV parser such that it can work on multiple different encodings. For example, this means looking for `\x2C\x00` when parsing UTF-16LE data instead of just `,`. This introduces implementation complexity, and you'll be unlikely to support the full gamut of encodings that other tools support whose job it is to do that sort of thing.
(2) is kind of weird but probably quite a bit faster than (1), although I can imagine it being useful in very niche circumstances. e.g., "I have a boat load of UTF-16 encoded CSV data and transcoding it to UTF-8 to use this CSV parser isn't worth my time because ______." I can't actually fill in that blank, so solutions in (1) tend to be the way to go.
Now... If you're building a full on CSV tabular viewer, then I might understand why it should handle encoding for you automatically, but when it comes down to it, the viewer is still going to need to choose between (1) and (2). Unless they want to hand roll their own CSV library, I imagine they're just going to pick (1), and when possible, transcode the data first. In that case, it shouldn't really matter whether their underlying CSV parser supports alternative encodings or not.
- TheAceOfHearts 9y agoThe use-case in which I got to see a lot of CSVs was in a data importing tool. Personally, I think that everyone should just use UTF-8. But since a lot of people don't do that, we were forced to try guessing the encoding while providing a preview, along with a way to try manually try other encodings. I think the only other time I've seen something that hacky has been with a date parser that would try to guess the format for you. That pushed me to the firm belief that ISO8601 is the only sensible way to store dates.