3 ms·
> JSON is a text format and interning it into proper data structures is likely going to take _less_ space, not more. ~~Yeah but parsing it can require ~2x the
by vlovich123 4y ago
> JSON is a text format and interning it into proper data structures is likely going to take _less_ space, not more.
~~Yeah but parsing it can require ~2x the RAM available and push you into swap / make it not possible.~~
> Or write a custom little program that streams data from the JSON file without buffering it all in memory. JSON parsing libraries are plentiful so this should not take a lot of code in your favorite language.
What is the state of SAX JSON parsing? I used yajl a long time ago but not sure if that’s still the state of the art (and it’s C interface was not the easiest to work with).
EDIT: Actually, I think the reason is that you typically will have pointers (8 bytes) in place of 2 byte demarcations in the text version (eg “”, {}, [] become pointers). It’s very hard to avoid that (maybe impossible? Not sure) and no surprise that Python has a problem with this.
- saidinesh5 4y agoThere's libraries like ijson. At my last job, i wrote a wrapper around this C++ library called rapidjson to parse streaming json. It worked quite well. Here's the write up about it: https://dinesh.cloud/2022/streaming-json-for-fun-and-profit/ https://dinesh.cloud/2022/streaming-json-for-fun-and-profit/ And here's the code: https://github.com/multiversal-ventures/json-buffet https://github.com/multiversal-ventures/json-buffet The API isn't the best. I'd have preferred an iterator based solution as opposed to this callback based one. But we worked with what rapidjson gave us for the proof of concept. The reason for this specific implementation was we wanted to build an index to query the server directly about it's huge json files (compressed size of 20+GB per file) using http range queries.