12 ms·

It's probably not news to anyone who works with data that CSVs are inferior to other file formats like parquet. I think the valuable insight here is that there

It's probably not news to anyone who works with data that CSVs are inferior to other file formats like parquet. I think the valuable insight here is that there needs to be a meme / movement that CSV is bad or deprecated. That's what's actually going to put the nails in its coffin, not private griping from developers when they get CSVs. I'm all for it. Down with CSV :)

drewmate 5y ago
I'm with you! But what does a hot new startup do when a rich old legacy corp sends them a data dump that isn't a custom sqlite database? Slowly but surely, the startup begrudgingly uses csv "just this once" until they grow into the legacy corp that sends out files in csv because "we've done it this way forever." And the cycle continues.
Sanzig 5y ago
The thing is, I need a hefty library to write parquet. I can write a CSV with a couple fprintf statements and a loop. A parser isn't much harder if we can make assumptions about CSV dialect ahead of time. If I'm trying to make an exportable format for a data logger with an SD card running on an ARM microcontroller, it doesn't get much easier than CSV. Sure, I could save space by rolling my own binary format, but then I have to provide a PC application to read it (and realistically, the user is probably just going to want that application to dump to a CSV anyway!). I agree that for many use cases there are much better alternatives, but one of the reasons CSV is so popular is because it's so simple. It shouldn't be used for multi-gigabyte datasets, but for many simple use cases it works great.
- dingosity 5y ago
  no. you can't write CSV with a couple of fprintf statements in a loop. If your data has a comma or a double-quote in it, the field needs to be surrounded by double quotes (and the double quote needs to be doubled). Ditto if you have leading or trailing spaces. that is... if you're following the RFC 4180 guidance for CSV. microsoft has another internal standard that occasionally gets referenced.

drewmate 5y ago

I'm with you! But what does a hot new startup do when a rich old legacy corp sends them a data dump that isn't a custom sqlite database? Slowly but surely, the startup begrudgingly uses csv "just this once" until they grow into the legacy corp that sends out files in csv because "we've done it this way forever." And the cycle continues.

Sanzig 5y ago

The thing is, I need a hefty library to write parquet. I can write a CSV with a couple fprintf statements and a loop. A parser isn't much harder if we can make assumptions about CSV dialect ahead of time. If I'm trying to make an exportable format for a data logger with an SD card running on an ARM microcontroller, it doesn't get much easier than CSV. Sure, I could save space by rolling my own binary format, but then I have to provide a PC application to read it (and realistically, the user is probably just going to want that application to dump to a CSV anyway!). I agree that for many use cases there are much better alternatives, but one of the reasons CSV is so popular is because it's so simple. It shouldn't be used for multi-gigabyte datasets, but for many simple use cases it works great.

dingosity 5y ago
no. you can't write CSV with a couple of fprintf statements in a loop. If your data has a comma or a double-quote in it, the field needs to be surrounded by double quotes (and the double quote needs to be doubled). Ditto if you have leading or trailing spaces. that is... if you're following the RFC 4180 guidance for CSV. microsoft has another internal standard that occasionally gets referenced.