3 ms·
If only there were character codes specifically meant to separate fields and records.... we wouldn't have to worry so much about quoted commas or quoted quotes.
by SoftTalker 2y ago
If only there were character codes specifically meant to separate fields and records.... we wouldn't have to worry so much about quoted commas or quoted quotes.
- mjw_byrne 2y agoThere's just no such thing as a delimiter which won't find its way into the data. Quoting and escaping really are the only robust way.
- saulpw 2y agoYou can disallow all control characters (ASCII < 32) other than CR/LF/TAB, which is reasonable. I don't know of any data besides binary blobs which uses those. I've never heard of anyone inlining a binary file (like an image) into a "CSV" anyway.
- mjw_byrne 2y agoIf you disallow control characters so that you can use them as delimiters, then CSV itself becomes a "binary" data format - or to put it another way, you lose the ability to nest CSV. It isn't good enough to say "but people don't/won't/shouldn't do that", because it will just happen regardless. I've seen nested CSV in real-life data. Compare to the zero-terminated strings used by C, one legacy of which is that PostgreSQL doesn't quite support UTF-8 properly, because it can't handle a 0 byte in a string, because 0 is "special" in C.
- saulpw 2y agoNested CSVs as you've seen in real-life data are a good counterexample, thanks for providing it.
- thayne 2y agoSo have a way to escape those control characters.
- mjw_byrne 2y agoRight, but the original point I was responding to is that control characters are disallowed in the data and therefore don't need to be escaped. If you're going to have an escaping mechanism then you can use "normal" characters like comma as delimiters, which is better because they can be read and written normally.
- thayne 2y agoBut a comma is much more likely to need to be escaped.
- mjw_byrne 2y agoIt's good for a delimiter to be uncommon in the data, so that you don't have to use your escaping mechanism too much. This is a different thing altogether from using "disallowed" control characters, which is an attempt to avoid escaping altogether - an attempt which I was arguing is doomed to fail.
- craftkiller 2y agoThat isn't solving anything, just changing the problem. If I want to store a string containing 0x1C - 0x1F in one of the columns then we're back in the exact same situation while also losing the human readable/manually typeable aspect people seem to love about CSV. The real solution is a strict spec with mandatory escaping.
- SoftTalker 2y agoNot for text data. Those values are not text characters like , or " are, and have only one meaning. It would be like arguing that 0x41 isn't always the letter "A". For binary files, yeah but you don't see CSV used there anyway.
- Hackbraten 2y agoThere are several ways how a control character might inadvertently end up inside a text corpus. Given enough millions of lines, it’s bound to happen, and you absolutely don’t want it to trip up your whole export because of that one occurrence. So yes, you have to account for it in text data, too.
- yellowapple 2y agoSure, but that's orders of magnitude less likely to happen than a comma ending up inside a text corpus.
- craftkiller 2y agoSo in addition to losing human readability, we are also throwing away the ability to nest (pseudo-)CSVs? With comma delimiters, I can take an entire CSV document and put it in 1 column, but with 0x1C-0x1F delimiters and banning non-text valid utf-8 in columns I no longer can. This continues to be a step backwards.
- thayne 2y agoNo reason you can't escape those special characters.