2 ms·
Suggestions for a more memory-efficient JSON processor for resource constrained computers? All the ones I have tried were no better than using standard line-or
by 1vuio0pswjnm7 6y ago
Suggestions for a more memory-efficient JSON processor for resource constrained computers? All the ones I have tried were no better than using standard line-oriented UNIX utilities.
Example:
189MB JSON
curl https://raw.githubusercontent.com/zemirco/sf-city-lots-json/master/citylots.json |jq
while that is running watch the system memory using vmstat, top, etc.
- kemitchell 6y agoGood Stuff: https://simdjson.org/ https://simdjson.org/
- nigeltao 6y agosimdjson is indeed good stuff, but its focus is CPU efficiency, not memory efficiency. For example, it requires the entire input to be in memory. It does not support streaming input.
- kemitchell 6y agoFor streaming applications, I've often found this approach useful: http://ndjson.org/ http://ndjson.org/ That's assuming your structure is list-large, rather than tree-large. Adjacency lists can often bridge the gap.
- nigeltao 6y agohttps://nigeltao.github.io/blog/2020/jsonptr.html https://nigeltao.github.io/blog/2020/jsonptr.html describes a JSON query tool that can use an order of magnitude less memory than jq. Its query language (JSON Pointer) is much simpler, though.