3 ms·

There are mature CSV libraries for most major languages that handle 99% of the problems of CSV. CSV should be better standardized, but ... whatever, what shoul

There are mature CSV libraries for most major languages that handle 99% of the problems of CSV. CSV should be better standardized, but ... whatever, what should be done to "fix" CSV is to advertise the proper use of the libraries and the nontrivial aspects of a superficially trivial format. A format that is trivially useful in 99% of cases is far better than many other "worse is better" things in computing.

adgjlsfhk1 5y ago
The problem is that even the fastest CSV readers are 10-100x slower than readers of better defined binary formats (like Arrow)
masklinn 5y ago
> There are mature CSV libraries for most major languages that handle 99% of the problems of CSV. They really don't. In fact I'd go further and confidently state that they really can't, because tons of mis-parsed CSVs are heuristic judgement values, and those tools don't really have the ability to make those calls. I've never seen a "mature CSV library for most major language" which'd guess encoding, separators, quoting/escaping, jaggedness, … to say nothing of being able to fix issues like mojibake.
- burntsushi 5y ago
  No, they really do. But 99% isn't 100%. Compare Python's csv parser with Go's. The former will prefer a parse for any input, and this is done in a fairly consistent way that at least gives one the opportunity to deal with "malformed" csv. The latter is a strict parser and coughs up an error upon almost any deviation from RFC 4180. I couldn't use the latter in practice because of it. It would just choke on too many csv files I had.
  - masklinn 5y ago
    > No, they really do. But 99% isn't 100%. Neither is 0. Python’s csv library does not handle any problem, it applies the separators / quoting / escapes it was given and that’s that. It gives the developer the opportunity to handle things, but if the developer isn’t aware nothing is handled.
    - burntsushi 5y ago
      If you're just going to quote me out of context and not bother responding to the actual substance of my comment (which was a high level but concrete comparison of two different CSV parsers, demonstrating an important qualitative difference with respect to csv parsers solving problems), then please, don't bother responding.

adgjlsfhk1 5y ago

The problem is that even the fastest CSV readers are 10-100x slower than readers of better defined binary formats (like Arrow)

masklinn 5y ago

> There are mature CSV libraries for most major languages that handle 99% of the problems of CSV. They really don't. In fact I'd go further and confidently state that they really can't, because tons of mis-parsed CSVs are heuristic judgement values, and those tools don't really have the ability to make those calls. I've never seen a "mature CSV library for most major language" which'd guess encoding, separators, quoting/escaping, jaggedness, … to say nothing of being able to fix issues like mojibake.

burntsushi 5y ago
No, they really do. But 99% isn't 100%. Compare Python's csv parser with Go's. The former will prefer a parse for any input, and this is done in a fairly consistent way that at least gives one the opportunity to deal with "malformed" csv. The latter is a strict parser and coughs up an error upon almost any deviation from RFC 4180. I couldn't use the latter in practice because of it. It would just choke on too many csv files I had.
- masklinn 5y ago
  > No, they really do. But 99% isn't 100%. Neither is 0. Python’s csv library does not handle any problem, it applies the separators / quoting / escapes it was given and that’s that. It gives the developer the opportunity to handle things, but if the developer isn’t aware nothing is handled.
  - burntsushi 5y ago
    If you're just going to quote me out of context and not bother responding to the actual substance of my comment (which was a high level but concrete comparison of two different CSV parsers, demonstrating an important qualitative difference with respect to csv parsers solving problems), then please, don't bother responding.