2 ms·
> If you don't open it in Excel, you can have as strict a parser as you want, just like any other format. No, you can not. Because the CSV format is so fuzzy y
by masklinn 5y ago
> If you don't open it in Excel, you can have as strict a parser as you want, just like any other format.
No, you can not. Because the CSV format is so fuzzy you can very easily parse incorrectly and end up with a valid parse full of garbage.
Trivially: incorrect separator, file happens to not contain that separator at all, you end up with a single column. That's a completely valid file, and might even make sense for the system. Also trivially: incorrect encoding, anything ascii-compatible will parse fine as iso-8859-*. Also trivially: incorrect quoting / escaping, might not break the parse, will likely corrupt the data (because you will not be stripping the quotes or applying the escapes and will store them instead).
It's like you people have never had to write ingestion pipelines for CSVs coming from randos.
- ribosometronome 5y agoWhy is the solution to create an entirely new format rather than try to more rigidly enforce the a single CSV standard?
- masklinn 5y agoBecause there is no way to perform that enforcement, becauae you say “CSV” and people understand “my old garbage” and you can’t fix that; and because millions of incorrect documents will yield a valid but nonsensical parse.
- romwell 5y ago>It's like you people have never had to write ingestion pipelines for CSVs coming from randos. That's because this is not what this thread is about. The comment you're responding to is not about CSVs coming from "randos". It's for the case where that rando is you, so you can make sure the problems you mention don't happen on the generation side of CSVs.