3 ms·
If anything you'd probably want to send it in Arrow[1] format. CSV's don't even preserve data types. [1]: https://arrow.apache.org/ https://arrow.apache.org/
by ZeroCool2u 4y ago
If anything you'd probably want to send it in Arrow[1] format. CSV's don't even preserve data types.
[1]: https://arrow.apache.org/ https://arrow.apache.org/
- alchemist1e9 4y agoarrow/feather is really the best format these days for tabular data transmission. anyone who disagrees I’d be very interested to hear your thoughts on alternatives.
- pletnes 4y agoWhat about compression - is this part of arrow itself?
- adgjlsfhk1 4y agoIt's not part of Arrow, but Arrow is columnar so just a basic LZ4/ZSTD will work pretty well.
- pletnes 4y agoOr parquet, for compression?
- FridgeSeal 4y agoArrow is meant as the “in-memory” dual to Parquet, which is meant as the “on disk serialisation format”. Many parquet supporting libs will load Parquet files into an Arrow structure in memory for example.
- pbreit 4y agoArrow looks super complicated. Are data types useful for data to/fro web/mobile clients? Encode type into the column header?
- adgjlsfhk1 4y agodata types are absolutely helpful. when you know a column stores Float64 data, you don't have to write out float to base 10 and parse it back. You just dump the bytes.