3 ms·
Slightly off-topic, but maybe we need a fully standardized and unambiguous CSV dialect with its own file extension. Or maybe just use SQLite tables or Parquet?
by jkabrg 9y ago
Slightly off-topic, but maybe we need a fully standardized and unambiguous CSV dialect with its own file extension. Or maybe just use SQLite tables or Parquet?
Some things I dislike about CSV:
* No distinction between categorical data and strings. R thinks your strings are categories, and Pandas thinks your categories are strings.
* I'm not a fan of the empty field. Pandas thinks it's a floating point NaN, while R doesn't. So is it a NaN? Is it an empty string? Does it mean Not Applicable? Does it mean Don't Know? Maybe it should be removed altogether.
* No agreement about escape characters.
* No agreement about separator characters.
* No agreement about line endings.
* No agreement about encoding. Is it ASCII, or UTF-8, or UTF-16, or Latin-whatever?
* None of the choices above are made explicit in the file itself. They all have the same extension "CSV".
These use up a bit of time whenever I get a CSV from a colleague, or even when I change operating system. Sometimes I end up clobbering the file itself.
Good things:
* Human readable.
* Simple.
I think the addition of some rules, and a standard interpretation of them, could go some way to improving the format.
- kqr 9y agoSee, one of the reasons CSV managed to get so ubiquitous is precisely because all those things are unspecified. CSV is not a popular format; CSV is the name we give 960 visually similar but very different formats that as a collective are popular. The thing you use CSV for is not it's technical merit. You use CSV for its ubiquity. If you nailed down all those things you talk about, you would have a much, much smaller user base and there would be no reason to use CSV in the first place. (Hey, this reminds me of a similar situation governing s/CSV/C/g...)
- f00_ 9y ago.parquet fam, it's all about columnar data stores now
- tomc1985 9y agoXKCD covered this: https://xkcd.com/927/ https://xkcd.com/927/
- johnwilkesbooth 9y ago> No distinction between categorical data and strings. R thinks your strings are categories, and Pandas thinks your categories are strings. I think this is more of an R-ism than a standardization issue. Strings are a pretty universal data type, where as categorical data (factors) are mostly specific to the domain of statistical modeling. IMO Python is doing the correct thing here. Personally I find factors to be more trouble than they are worth, and fortunately `data.table::fread` mimics Python in this regard.