3 ms·

> JSON’s design assumes the user can read the entire file into memory No? The design of most JSON libraries assumes that, but there are perfectly good incremen

> JSON’s design assumes the user can read the entire file into memory No? The design of most JSON libraries assumes that, but there are perfectly good incremental JSON parsers out there[1–3]. It’s just that people don’t seem to have figured out a good API for not-completely-incremental parsing (please prove me wrong here!), but this applies equally to any structured data format as soon as you want to pull out pieces of data that are nested more than one level down. The lack of length prefixes in JSON does indeed make a solid parser somewhat more difficult, but you get the ability to author and validate it manually instead. All in all a draw and not because of the incremental parsing thing. (Tabular or otherwise homogeneous data is indeed reprsented wastefully, but unless the individual records are huge json+gzip is a perfectly serviceable “worse-is-better” solution, and its self-describing nature along with support for structured cell values can at times make for a better experience than TSV. And at other times not.) [1] https://github.com/ICRAR/ijson https://github.com/ICRAR/ijson [2] https://github.com/AMDmi3/jsonslicer https://github.com/AMDmi3/jsonslicer [3] https://github.com/danielyule/naya https://github.com/danielyule/naya

1vuio0pswjnm7 5y ago
All three of those Python scripts use the same library, YAJL It comes with an example program called json_reformat which I have experimented with in the past. However the "reformatters" I write using only shell utilities work just as well. YMMV.
- mananaysiempre 5y ago
  Yes, but raw YAJL (or its Yajl-Py binding, or other incremental parsers like jsmn) is a right pain to use while these are actually interesting in terms of API design. I doubt your JSON reformatter was entirely correct (hey, we all must parse [X]HTML using regex from time to time), as you either have to pretend you can make sense of JSON using regular expressions (you can’t, no language or format supporting unbounded nesting can be regular) or write what’s essentially a standard JSON parser in shell with all the associated inefficiencies. My own needs are usually adequately served by the likes of jq -r ... | while IFS=$"\t" read -r ..., which feels much less hackish. (Even correctly handling CSV with quoting, escaping, and embedded newlines in a UNIX pipeline without a purpose-built utility is surprisingly difficult—although not impossible, as CSV is a regular language.)
  - 1vuio0pswjnm7 5y ago
    Thanks for the summary of problems with JSON. For ease of use, it will not likely become a de facto interchange format in the UNIX context. However not all JSON is the same. Simple JSON is easy to parse and one does not need jq or libraries, for example JSON DoH responses. People have even tried to standardise simpler JSON, e.g., jsonl, as another reply mentions. But, as I said, the design of JSON allows for and encourages complexity. If that's what you like, go for it. An HN commenter recently identified the "complexity fetish" that many programmers have. Not everyone suffers from that fetish, fortunately. With excessively complex JSON, I prefer to extract the specific data I want. The speed of json_reformat demonstrates how slow reformatting is when done "correctly". Every user's needs are different. jq fails to meet mine.

1vuio0pswjnm7 5y ago

All three of those Python scripts use the same library, YAJL It comes with an example program called json_reformat which I have experimented with in the past. However the "reformatters" I write using only shell utilities work just as well. YMMV.

mananaysiempre 5y ago
Yes, but raw YAJL (or its Yajl-Py binding, or other incremental parsers like jsmn) is a right pain to use while these are actually interesting in terms of API design. I doubt your JSON reformatter was entirely correct (hey, we all must parse [X]HTML using regex from time to time), as you either have to pretend you can make sense of JSON using regular expressions (you can’t, no language or format supporting unbounded nesting can be regular) or write what’s essentially a standard JSON parser in shell with all the associated inefficiencies. My own needs are usually adequately served by the likes of jq -r ... | while IFS=$"\t" read -r ..., which feels much less hackish. (Even correctly handling CSV with quoting, escaping, and embedded newlines in a UNIX pipeline without a purpose-built utility is surprisingly difficult—although not impossible, as CSV is a regular language.)
- 1vuio0pswjnm7 5y ago
  Thanks for the summary of problems with JSON. For ease of use, it will not likely become a de facto interchange format in the UNIX context. However not all JSON is the same. Simple JSON is easy to parse and one does not need jq or libraries, for example JSON DoH responses. People have even tried to standardise simpler JSON, e.g., jsonl, as another reply mentions. But, as I said, the design of JSON allows for and encourages complexity. If that's what you like, go for it. An HN commenter recently identified the "complexity fetish" that many programmers have. Not everyone suffers from that fetish, fortunately. With excessively complex JSON, I prefer to extract the specific data I want. The speed of json_reformat demonstrates how slow reformatting is when done "correctly". Every user's needs are different. jq fails to meet mine.