4 ms·
I think CSV is crappy because commas are so common in real data. For almost all scenarios I've had to work with, I'd have been perfectly happy with TSV where l
by quesera 5y ago
I think CSV is crappy because commas are so common in real data.
For almost all scenarios I've had to work with, I'd have been perfectly happy with TSV where literal Tab was a disallowed character. No escaping histrionics required.
- hermitcrab 5y agoNot being able to include Tabs and Carriage Returns in your data can be a problem though.
- deleted 5y ago[deleted]
- mjevans 5y agoDepending on the library, (double) quoted fields _may_ be able to contain carriage returns (0x13) and/or line feeds (0x10). Sometimes it works, sometimes it doesn't, can be a real pain in the butt.
- justin_oaks 5y agoThe ideal would be a format that avoids characters that are common in data, but still allows a way to encode rare characters that are part of data. I think that was the motivation for multi-character delimiters like <% %> or {{ }} in templating languages.
- pishpash 5y agoUnicode just needs a single special delimiter character that is only used as a delimiter.
- GekkePrutser 5y agoIdeally a separate row and a column delimiter
- BenjiWiebe 5y agoAs I understand it, ASCII already has this. And isn't every ASCII codepoint valid UTF-8?
- prepend 5y agoI prefer commas because I can see them over tabs. I spend zero time escaping commas because the libraries and read and write with (usually pandas but pretty much everything) escape for me. So unless I’m manually building csvs myself it’s a non-issue and certainly not histrionics.
- thayne 5y agoNewlines are pretty common in text too, depending on your data. What do you do about those?
- mbreese 5y agoI prefer tabs because my data can often have commas in it. Seeing tabs isn’t an issue as I also have invisible characters visible in all of my editors. But having your delimiter not be allowed in the record (as in \t is disallowed), makes parsing so much easier. CSV is a bear to parse because you have to read each value from a buffer to handle the quoting. line.strip().split(‘\t’) Is so handy. Batteries included are one thing, but I don’t want to pull in a library just to parse CSV files.
- tech2 5y agoGiven your example looks very much like python you could use the CSV module which is built into the standard library and handles all of this for you in a standards compliant manner. You really do want to use a library to parse CSV since there are a number of corner cases. For example, your example code may not read a whole row since rows can have newlines in them.
- mbreese 5y ago> You really do want to use a library to parse CSV Right… you absolutely need to use a library. I’ve written the parsers for CSV which handle the edge cases (at least RFC edge cases). But I don’t want to include a library to be able to do it. Most of my scripts are small, and adding libraries makes them more difficult to move around. So, I shy away from CSV as a format as a result. I’m okay with not allowing new lines and tabs in my fields. It’s a valid trade off for my data. Yes, many of my scripts are Python, but not all, so adding new libraries for different languages is more difficult for me to remember as opposed to just splitting a line by a record separator. Also, many of my files require having comments in the file’s header. Another reason why CSV can’t be easily used, as this isn’t part of the RFC.
- nerdponx 5y agoPipes are usually pretty safe too.
- qayxc 5y agoDepends entirely on context. I did consulting in wholesale and supplier data very often contained pipes as content separators (e.g. for product categories, sizes, etc.).