3 ms·
Yes, it's not great. Space is annoying, though compression pretty much removes that as a concern (zstd is good for this, you can even have a custom dictionary).
by IanCal 2y ago
Yes, it's not great. Space is annoying, though compression pretty much removes that as a concern (zstd is good for this, you can even have a custom dictionary). And yes, missing keys is annoying.
JSONL is handy, JSON that's in the form {data: [...hundred megs of lines]} is annoying for various parsers.
I'm quite a fan of parquet, but never expect to receive that from a client (alas).
- cogman10 2y agoParquet should get the praise. It's simply awesome. It's what I'd pick for tabular data exchange. A recent problem I solved with it and duckdb allowed me to query and share a 3M record dataset. The size? 50M. And my queries all ran subsecond. You just aren't going to get that sort of compression and query-ability with a csv.
- Izkata 2y ago> JSON that's in the form {data: [...hundred megs of lines]} is annoying for various parsers. One reason this became common was a simple protection against json hijacking: https://haacked.com/archive/2009/06/25/json-hijacking.aspx/ https://haacked.com/archive/2009/06/25/json-hijacking.aspx/