7 ms·

Many good points are made here, but I would point out that there is a significant difference between working with JSON and CSV: data that can be operated on can

Many good points are made here, but I would point out that there is a significant difference between working with JSON and CSV: data that can be operated on can typically be extracted from a CSV by working iteratively over each line. The same can not be said about JSON files. It is normal and (almost always ?) required to parse/de-serialize an entire JSON file at once and hold the contents of it in memory, while the same is often impractical and unnecessary for a CSV (when it is millions of lines). The standard methods and idioms of doing these things in Python reflects this.

dragonwriter 6y ago
> It is normal and (almost always ?) required to parse/de-serialize an entire JSON file at once and hold the contents of it in memory It may be normal, but there's no reason it is required. Streaming JSON processing makes sense just like streaming XML processing.
tomjakubowski 6y ago
It's not uncommon to see files or streams in "JSON lines" format. For files where each line is a record and the records are sorted or indexed, it's possible to seek to a record of interest just as in CSV. http://jsonlines.org/ http://jsonlines.org/
- coldtea 6y ago
  Those files are individual JSON entries though, as per the spec. They're not read as a JSON (since they aren't) they are read per line, and each line is parsed as a JSON.
  - orf 6y ago
    Same as CSV then?
    - coldtea 6y ago
      Yes, but not the same as JSON. Which was my point. In other words, collection of JSON strings in a file != a JSON file.
moreaccountspls 6y ago
It's never required: https://stedolan.github.io/jq/manual/#Streaming https://stedolan.github.io/jq/manual/#Streaming
patrec 6y ago
I find it interesting that you are seeing one of the prime defects of python's json parsing implementation (no streaming support) as an inherent property of json parsing. Do you use any other languages that share this defect, or is this take based on your familiarity with python?

dragonwriter 6y ago

> It is normal and (almost always ?) required to parse/de-serialize an entire JSON file at once and hold the contents of it in memory It may be normal, but there's no reason it is required. Streaming JSON processing makes sense just like streaming XML processing.

tomjakubowski 6y ago

It's not uncommon to see files or streams in "JSON lines" format. For files where each line is a record and the records are sorted or indexed, it's possible to seek to a record of interest just as in CSV. http://jsonlines.org/ http://jsonlines.org/

coldtea 6y ago
Those files are individual JSON entries though, as per the spec. They're not read as a JSON (since they aren't) they are read per line, and each line is parsed as a JSON.
- orf 6y ago
  Same as CSV then?
  - coldtea 6y ago
    Yes, but not the same as JSON. Which was my point. In other words, collection of JSON strings in a file != a JSON file.

moreaccountspls 6y ago

It's never required: https://stedolan.github.io/jq/manual/#Streaming https://stedolan.github.io/jq/manual/#Streaming

patrec 6y ago

I find it interesting that you are seeing one of the prime defects of python's json parsing implementation (no streaming support) as an inherent property of json parsing. Do you use any other languages that share this defect, or is this take based on your familiarity with python?