4 ms·
I routinely interface with 1GB+ csvs. The size explosion for json would be huge. Disk IO aside, I assume a json parser is going to be slower to parse than csv.
by 0cf8612b2e1e 2y ago
I routinely interface with 1GB+ csvs. The size explosion for json would be huge. Disk IO aside, I assume a json parser is going to be slower to parse than csv.
- im3w1l 2y agoWhy do you use a text-based format at all at that size?
- 0cf8612b2e1e 2y agoYou get what you get. Presumably when it started, they were a more modest size.
- imtringued 2y agoHow would JSON cause a size explosion? Nothing prevents you using ndjson where you define a header and then have an array per line.
- 0cf8612b2e1e 2y agoNobody does this currently. You have now created another bespoke format. If I am going to need a custom parser/writer, I might as well lean on a binary format that has far stronger properties than a text based one.
- ianburrell 2y agoJSONL is pretty common format. It makes sense for logs and anything else written incrementally. JSON parsers are super common. They are simpler and faster than CSV because it is more regular. JSONL is simple to implement cause write by record and read by line. The only difference with CSV are bracket characters around line and every string has quotes. The benefit is clear escaping rules including for newlines.
- 0cf8612b2e1e 2y agoJSONL is standard. Upthread said to write the header row and then make subsequent rows arrays. Of which I am not aware of anything that does this currently. My objection to JSONL was about the increase in file size owing to repeating the keys.
- ianburrell 2y agoJSON can write arrays in addition to hashes. JSON arrays are nearly identical to CSV. The only difference is brackets around li;es. There is no extra space wasted for keys.