3 ms·

WRT JSON: > Headers are in each line This might be my old “space and network cost savings” reflex, which is a lot less necessary these days, kicking in, but t

WRT JSON: > Headers are in each line This might be my old “space and network cost savings” reflex, which is a lot less necessary these days, kicking in, but the feels inefficient. It also gives rise to not knowing the whole schema until you read the whole dataset (which might be multiple files), unless some form of external schema definition is provided. Having said that, I accept that JSON has advantages over CSV, even if all that is done is translating a data-table into an array of objects representing one row each. > utf8 has never been an issue for me The main problem with UTF8 isn't with CSV generally, it is usually, much like the “first column is called ID” issue, due to Excel. Unfortunately a lot of people interact with CSVs primarily with Excel, so it gets tarred with that brush by association. Unless Excel sees the BOM sequence at the start of a CSV file, which the Unicode standards recommend against for UTF8, it assumes its characters are using the Win1252 encoding (almost, but not quite, ISO-8859-1). > Csv files have all kinds of separators I've taken to calling them Character Separated Value files, rather than Comma, for this reason.

IanCal 2y ago
Yes, it's not great. Space is annoying, though compression pretty much removes that as a concern (zstd is good for this, you can even have a custom dictionary). And yes, missing keys is annoying. JSONL is handy, JSON that's in the form {data: [...hundred megs of lines]} is annoying for various parsers. I'm quite a fan of parquet, but never expect to receive that from a client (alas).
- cogman10 2y ago
  Parquet should get the praise. It's simply awesome. It's what I'd pick for tabular data exchange. A recent problem I solved with it and duckdb allowed me to query and share a 3M record dataset. The size? 50M. And my queries all ran subsecond. You just aren't going to get that sort of compression and query-ability with a csv.
- Izkata 2y ago
  > JSON that's in the form {data: [...hundred megs of lines]} is annoying for various parsers. One reason this became common was a simple protection against json hijacking: https://haacked.com/archive/2009/06/25/json-hijacking.aspx/ https://haacked.com/archive/2009/06/25/json-hijacking.aspx/

IanCal 2y ago

Yes, it's not great. Space is annoying, though compression pretty much removes that as a concern (zstd is good for this, you can even have a custom dictionary). And yes, missing keys is annoying. JSONL is handy, JSON that's in the form {data: [...hundred megs of lines]} is annoying for various parsers. I'm quite a fan of parquet, but never expect to receive that from a client (alas).

cogman10 2y ago
Parquet should get the praise. It's simply awesome. It's what I'd pick for tabular data exchange. A recent problem I solved with it and duckdb allowed me to query and share a 3M record dataset. The size? 50M. And my queries all ran subsecond. You just aren't going to get that sort of compression and query-ability with a csv.
Izkata 2y ago
> JSON that's in the form {data: [...hundred megs of lines]} is annoying for various parsers. One reason this became common was a simple protection against json hijacking: https://haacked.com/archive/2009/06/25/json-hijacking.aspx/ https://haacked.com/archive/2009/06/25/json-hijacking.aspx/