4 ms·

The way I've dealt with it in my parser (which is influenced by how Python's standard library csv parser deals with it) is to simply not have any CSV parse erro

The way I've dealt with it in my parser (which is influenced by how Python's standard library csv parser deals with it) is to simply not have any CSV parse errors at all. That is, you always return a parse of the data, even if it's wrong. As long as it's consistently wrong, it at least gives the user of your CSV parser a chance to read the data somehow. Normally, this isn't something I'd do, since it feels icky. But it works well in practice, precisely because CSV data can be so messy. It's often much easier to hack your way around the CSV data with a sufficiently robust parser than to demand that the source of the CSV data clean up their act.

mrweasel 9y ago
The CSV parser in Python actually makes least one assumption, that should hold true, but doesn't. The delimiter isn't always a single character. I've had to deal with CSV files containing financial transactions, they used "," as a decimal separator. Fine that perfectly normal, and the same standard used here in Denmark. But what to do with the delimiter then.... that can't be a comma, so rather than making it a ; or something similar, the made the delimiter ", " that's a comma followed by a space. That doesn't actually work for most CSV parsers, they just assume that the space is part of the data, and the value is split in kroner/øre (dollar/cents). The result is that you gain an additional field and need to trim all data.... Oh and headers doesn't match.
- burntsushi 9y ago
  Sure. Given a choice between a CSV parser that supported multiple character delimiters and one that didn't, sure you'd choose the one with better support for your use case. But given the choice between a CSV parser that gives you an invalid parse and a CSV parser that completely falls over when thrown a curveball (like a multi character delimiter), which would you pick? ;-)

mrweasel 9y ago

The CSV parser in Python actually makes least one assumption, that should hold true, but doesn't. The delimiter isn't always a single character. I've had to deal with CSV files containing financial transactions, they used "," as a decimal separator. Fine that perfectly normal, and the same standard used here in Denmark. But what to do with the delimiter then.... that can't be a comma, so rather than making it a ; or something similar, the made the delimiter ", " that's a comma followed by a space. That doesn't actually work for most CSV parsers, they just assume that the space is part of the data, and the value is split in kroner/øre (dollar/cents). The result is that you gain an additional field and need to trim all data.... Oh and headers doesn't match.

burntsushi 9y ago
Sure. Given a choice between a CSV parser that supported multiple character delimiters and one that didn't, sure you'd choose the one with better support for your use case. But given the choice between a CSV parser that gives you an invalid parse and a CSV parser that completely falls over when thrown a curveball (like a multi character delimiter), which would you pick? ;-)