3 ms·

Related to this, if you're looking for a CLI tool for handling CSVs, xsv [0] is looking promising. Based on prior experiences with CSV, one of the big problems

Related to this, if you're looking for a CLI tool for handling CSVs, xsv [0] is looking promising. Based on prior experiences with CSV, one of the big problems I've seen has been figuring out what text encoding is being used. This problem appears to be more prominent with people outside of the US. It looks like Tad is using fast-csv, which I don't think will properly handle different file encodings. Life would be so much simpler if everyone just used UTF-8. [0] https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv

nerdponx 9y ago
Csvkit [0] has been my go-to for a while now. I never heard of XSV before, but I will try just about anything with BurntSushi's name on it. Ripgrep is an amazing tool. [0]: https://github.com/wireservice/csvkit https://github.com/wireservice/csvkit
burntsushi 9y ago
You basically have two approaches: 1. Write your CSV parser with an assumption that the data is ASCII compatible (this means it works with either UTF-8 or Latin-1 out of the box, possibly modulo non-ASCII meta characters). To support additional encodings---such as UTF-16---either the CSV library or the caller must transcode first. 2. Write your CSV parser such that it can work on multiple different encodings. For example, this means looking for `\x2C\x00` when parsing UTF-16LE data instead of just `,`. This introduces implementation complexity, and you'll be unlikely to support the full gamut of encodings that other tools support whose job it is to do that sort of thing. (2) is kind of weird but probably quite a bit faster than (1), although I can imagine it being useful in very niche circumstances. e.g., "I have a boat load of UTF-16 encoded CSV data and transcoding it to UTF-8 to use this CSV parser isn't worth my time because ______." I can't actually fill in that blank, so solutions in (1) tend to be the way to go. Now... If you're building a full on CSV tabular viewer, then I might understand why it should handle encoding for you automatically, but when it comes down to it, the viewer is still going to need to choose between (1) and (2). Unless they want to hand roll their own CSV library, I imagine they're just going to pick (1), and when possible, transcode the data first. In that case, it shouldn't really matter whether their underlying CSV parser supports alternative encodings or not.
- TheAceOfHearts 9y ago
  The use-case in which I got to see a lot of CSVs was in a data importing tool. Personally, I think that everyone should just use UTF-8. But since a lot of people don't do that, we were forced to try guessing the encoding while providing a preview, along with a way to try manually try other encodings. I think the only other time I've seen something that hacky has been with a date parser that would try to guess the format for you. That pushed me to the firm belief that ISO8601 is the only sensible way to store dates.

nerdponx 9y ago

Csvkit [0] has been my go-to for a while now. I never heard of XSV before, but I will try just about anything with BurntSushi's name on it. Ripgrep is an amazing tool. [0]: https://github.com/wireservice/csvkit https://github.com/wireservice/csvkit

burntsushi 9y ago

You basically have two approaches: 1. Write your CSV parser with an assumption that the data is ASCII compatible (this means it works with either UTF-8 or Latin-1 out of the box, possibly modulo non-ASCII meta characters). To support additional encodings---such as UTF-16---either the CSV library or the caller must transcode first. 2. Write your CSV parser such that it can work on multiple different encodings. For example, this means looking for `\x2C\x00` when parsing UTF-16LE data instead of just `,`. This introduces implementation complexity, and you'll be unlikely to support the full gamut of encodings that other tools support whose job it is to do that sort of thing. (2) is kind of weird but probably quite a bit faster than (1), although I can imagine it being useful in very niche circumstances. e.g., "I have a boat load of UTF-16 encoded CSV data and transcoding it to UTF-8 to use this CSV parser isn't worth my time because ______." I can't actually fill in that blank, so solutions in (1) tend to be the way to go. Now... If you're building a full on CSV tabular viewer, then I might understand why it should handle encoding for you automatically, but when it comes down to it, the viewer is still going to need to choose between (1) and (2). Unless they want to hand roll their own CSV library, I imagine they're just going to pick (1), and when possible, transcode the data first. In that case, it shouldn't really matter whether their underlying CSV parser supports alternative encodings or not.

TheAceOfHearts 9y ago
The use-case in which I got to see a lot of CSVs was in a data importing tool. Personally, I think that everyone should just use UTF-8. But since a lot of people don't do that, we were forced to try guessing the encoding while providing a preview, along with a way to try manually try other encodings. I think the only other time I've seen something that hacky has been with a date parser that would try to guess the format for you. That pushed me to the firm belief that ISO8601 is the only sensible way to store dates.