3 ms·

if you are ok with a binary format there is apache parquet or apache feather or 'jay' (https://datatable.readthedocs.io/en/latest/api/frame/to_jay.html https://

if you are ok with a binary format there is apache parquet or apache feather or 'jay' (https://datatable.readthedocs.io/en/latest/api/frame/to_jay.html https://datatable.readthedocs.io/en/latest/api/frame/to_jay....).

hermitcrab 4y ago
No doubt binary formats like Parquet are the way to go for high performance with multi-GB datasets. Seems like total overkill if you have a few hundred or thousand rows of data though. Being able to create/edit/view stuff in a text editor and easily version it is very useful.
- habitue 4y ago
  Do people really edit csvs in a text editor? It's horrific, the columns don't line up at all, empty cells are represented by a bunch of commas in a row (which, are you supposed to count all the commas?) And in terms of versioning, I have seen people commit diffs of csvs before, and they're equally unreadable. CSV is a plain text format, but that basically buys you nothing. As long as you're going to be loading it into excel or whatever anyway, might as well just pick a good binary format like parquet.
  - macintux 4y ago
    I frequently use the table syntax in org-mode. No, I wouldn’t edit CSVs because they are miserable, but a friendly syntax with some basic text editor support, absolutely.
  - hermitcrab 4y ago
    I create and edit CSVs by hand daily. To create/modify simple examples to provide technical support for a data transformation tool.
  - gnulinux 4y ago
    I frequently edit CSVs by hand. As long as you understand your data, and stakes are low (e.g. it's a one off analysis not prod code) it's pretty easy and convenient.
  - themadturk 4y ago
    Rainbow CSV in VSCode makes it a lot easier to deal with.
- twobitshifter 4y ago
  It’s strange that we pretend text is not binary. The truth is our many tools are set up to handle binary text data and these tools are not set up for alternate encodings. If you grab a text file from a Windows machine and bring it to a mac, you’ll see that txt is far from perfect. This is a long way of saying that if we develop both the format and the tooling then the distinction of text vs “binary” tabular data goes away.
  - hermitcrab 4y ago
    Fair point. But the vast majority of programming tools do already handle UFT8 text (although perhaps don't do a great job with US/RS characters). also you could write an efficient parser for US/RS separated data in 5 minutes. For parquet you would have to integrate with a library (with all it's dependencies and design choices) or spend days writing your own parser.
  - willis936 4y ago
    Text is text. It's stored as bits, typically 8 bits per character. It allows arbitrary precision but is very space (and memory throughput) inefficient. Fine, compression works but there are many more bits flying around and you're spending CPU cycles on decompression. What could you do instead? Use appropriate integers, fixed point, or floating point representations. Everybody here knows these things. Nobody is pretending anything. To the point of portability: IEEE-754 is the same everywhere.

hermitcrab 4y ago

No doubt binary formats like Parquet are the way to go for high performance with multi-GB datasets. Seems like total overkill if you have a few hundred or thousand rows of data though. Being able to create/edit/view stuff in a text editor and easily version it is very useful.

habitue 4y ago
Do people really edit csvs in a text editor? It's horrific, the columns don't line up at all, empty cells are represented by a bunch of commas in a row (which, are you supposed to count all the commas?) And in terms of versioning, I have seen people commit diffs of csvs before, and they're equally unreadable. CSV is a plain text format, but that basically buys you nothing. As long as you're going to be loading it into excel or whatever anyway, might as well just pick a good binary format like parquet.
- macintux 4y ago
  I frequently use the table syntax in org-mode. No, I wouldn’t edit CSVs because they are miserable, but a friendly syntax with some basic text editor support, absolutely.
- hermitcrab 4y ago
  I create and edit CSVs by hand daily. To create/modify simple examples to provide technical support for a data transformation tool.
- gnulinux 4y ago
  I frequently edit CSVs by hand. As long as you understand your data, and stakes are low (e.g. it's a one off analysis not prod code) it's pretty easy and convenient.
- themadturk 4y ago
  Rainbow CSV in VSCode makes it a lot easier to deal with.
twobitshifter 4y ago
It’s strange that we pretend text is not binary. The truth is our many tools are set up to handle binary text data and these tools are not set up for alternate encodings. If you grab a text file from a Windows machine and bring it to a mac, you’ll see that txt is far from perfect. This is a long way of saying that if we develop both the format and the tooling then the distinction of text vs “binary” tabular data goes away.
- hermitcrab 4y ago
  Fair point. But the vast majority of programming tools do already handle UFT8 text (although perhaps don't do a great job with US/RS characters). also you could write an efficient parser for US/RS separated data in 5 minutes. For parquet you would have to integrate with a library (with all it's dependencies and design choices) or spend days writing your own parser.
- willis936 4y ago
  Text is text. It's stored as bits, typically 8 bits per character. It allows arbitrary precision but is very space (and memory throughput) inefficient. Fine, compression works but there are many more bits flying around and you're spending CPU cycles on decompression. What could you do instead? Use appropriate integers, fixed point, or floating point representations. Everybody here knows these things. Nobody is pretending anything. To the point of portability: IEEE-754 is the same everywhere.