3 ms·

I work in data engineering, I have seen more ways to fuck up data than any human has any business seeing, that's not new to me. Here's some more ideas for maxim

by qsort 2y ago

I work in data engineering, I have seen more ways to fuck up data than any human has any business seeing, that's not new to me. Here's some more ideas for maximum fun with CSVs: - Make sure to write your own parser! The format is so simple! - Is the first line a header or not? The RFC says you should look at the MIME type to tell. You can't make this up. - Is the line terminator "\r\n", "\n" or "\r"? - What happens if the number of records for a line is incorrect? Well, that SHOULD'nt happen... - Make sure to involve Excel, nothing ever went wrong feeding data to Excel. - Are end-of-line spaces relevant or not? Should we trim them? There is no defending of CSV, it's a broken format and it's broken beyond repair. The point is that the separator is immaterial. Replacing commas with something else gains us exactly nothing, and results in an identically broken format.

recursive 2y ago
Instead of CSV we should use RFC 4180.
Dylan16807 2y ago
I don't think anyone's suggesting to take the CSV RFC and do a find/replace on the commas. For one, I think you'd almost certainly use Record Separator, and \r and \n would be normal text characters inside of a field. And no trimming spaces. And the behavior upon encountering corruption isn't really up to the spec, is it? Headers might be more trouble than they're worth, but if they're in I'd say they need to be mandatory. One code path. You'll find yourself with a much less broken format.
teo_zero 2y ago
I love your post, and agree to everything, except the last 2 sentences. Yes, CSV is broken beyond repair. But this doesn't make the choice of separator immaterial. It's broken for other reasons AND for the separator. A character that's already profusely found in texts and numbers... I would even argue that it would be difficult to find a WORSE separator. Probably only space and hyphen qualify.
- Ekaros 2y ago
  Full stop would be even worse. Apostrophe would be runner up... Always the seemingly simple solutions have most issues in real world... At some level still trying to edit stuff by hand might be where we have gone wrong for long time. Why don't we have any sensible agreed structured format for which tools would be on all and every device...

by qsort 2y ago

recursive 2y ago

Instead of CSV we should use RFC 4180.

Dylan16807 2y ago

I don't think anyone's suggesting to take the CSV RFC and do a find/replace on the commas. For one, I think you'd almost certainly use Record Separator, and \r and \n would be normal text characters inside of a field. And no trimming spaces. And the behavior upon encountering corruption isn't really up to the spec, is it? Headers might be more trouble than they're worth, but if they're in I'd say they need to be mandatory. One code path. You'll find yourself with a much less broken format.

teo_zero 2y ago

I love your post, and agree to everything, except the last 2 sentences. Yes, CSV is broken beyond repair. But this doesn't make the choice of separator immaterial. It's broken for other reasons AND for the separator. A character that's already profusely found in texts and numbers... I would even argue that it would be difficult to find a WORSE separator. Probably only space and hyphen qualify.

Ekaros 2y ago
Full stop would be even worse. Apostrophe would be runner up... Always the seemingly simple solutions have most issues in real world... At some level still trying to edit stuff by hand might be where we have gone wrong for long time. Why don't we have any sensible agreed structured format for which tools would be on all and every device...