3 ms·

At my last job, we have had to deal with this. We had huge json files (compressed size of 20+ GB) and had to perform certain operations on the data. I found tha

At my last job, we have had to deal with this. We had huge json files (compressed size of 20+ GB) and had to perform certain operations on the data. I found that rapidjson had a better (not perfect, just better) API to do this. I quickly wrote a little wrapper around it to process json data coming in as streams. This was a writeup about it: https://dinesh.cloud/2022/streaming-json-for-fun-and-profit/ https://dinesh.cloud/2022/streaming-json-for-fun-and-profit/ . If there is any interest, I can ask if they can to open source it.

sl-dolt 4y ago
Absolutely interested, on my end at least. I wrote this to manage the transparency in coverage files: https://github.com/dolthub/data-analysis/tree/main/transparency-in-coverage/python/mrfutils https://github.com/dolthub/data-analysis/tree/main/transpare... but I'm always looking for better techniques. Edit: Oh wow, I see you used it on those exact files. How about that.
- tinselcity 4y ago
  I used the rapidjson streams with my little embedded REST HTTP(s) server library: https://github.com/Edgio/is2/ https://github.com/Edgio/is2/ We needed it for streaming large json from async server sockets. Code link: https://github.com/Edgio/is2/blob/master/include/is2/support/nbq_stream.h https://github.com/Edgio/is2/blob/master/include/is2/support... You just had to implement the interfaces like Peek/Take/Tell/etc. It worked really well for us. Probably not as fast as simdjson, but they used some simd tricks I think for skipping whitespace: https://rapidjson.org/md_doc_internals.html#SkipwhitespaceWithSIMD https://rapidjson.org/md_doc_internals.html#SkipwhitespaceWi...
- saidinesh5 4y ago
  Ha! Thanks to you, Today I found out how big those uncompressed JSON files really are (the data wasn't accessible to me, so i shared the tool with my colleague and he was the one who ran the queries on his laptop): https://www.dolthub.com/blog/2022-09-02-a-trillion-prices/ https://www.dolthub.com/blog/2022-09-02-a-trillion-prices/ . And yep, it was more or less they way you did with ijson. I found ijson just a day after I finished the prototype. Rapidjson would probably be faster. Especially after enabling SIMD. But the indexing was a one time thing. We have open sourced the codebase. Here's the link: https://github.com/multiversal-ventures/json-buffet https://github.com/multiversal-ventures/json-buffet . Since this was a quick and dirty prototype, comments were sparse. I have updated the Readme, and added a sample json-fetcher. Hope this is more useful for you. Another unwritten TODO was to nudge the data providers towards a more streaming friendly compression formats - and then just create an index to fetch the data directly from their compressed archives. That would have saved everyone a LOT of $$$.

sl-dolt 4y ago

Absolutely interested, on my end at least. I wrote this to manage the transparency in coverage files: https://github.com/dolthub/data-analysis/tree/main/transparency-in-coverage/python/mrfutils https://github.com/dolthub/data-analysis/tree/main/transpare... but I'm always looking for better techniques. Edit: Oh wow, I see you used it on those exact files. How about that.

tinselcity 4y ago
I used the rapidjson streams with my little embedded REST HTTP(s) server library: https://github.com/Edgio/is2/ https://github.com/Edgio/is2/ We needed it for streaming large json from async server sockets. Code link: https://github.com/Edgio/is2/blob/master/include/is2/support/nbq_stream.h https://github.com/Edgio/is2/blob/master/include/is2/support... You just had to implement the interfaces like Peek/Take/Tell/etc. It worked really well for us. Probably not as fast as simdjson, but they used some simd tricks I think for skipping whitespace: https://rapidjson.org/md_doc_internals.html#SkipwhitespaceWithSIMD https://rapidjson.org/md_doc_internals.html#SkipwhitespaceWi...
saidinesh5 4y ago
Ha! Thanks to you, Today I found out how big those uncompressed JSON files really are (the data wasn't accessible to me, so i shared the tool with my colleague and he was the one who ran the queries on his laptop): https://www.dolthub.com/blog/2022-09-02-a-trillion-prices/ https://www.dolthub.com/blog/2022-09-02-a-trillion-prices/ . And yep, it was more or less they way you did with ijson. I found ijson just a day after I finished the prototype. Rapidjson would probably be faster. Especially after enabling SIMD. But the indexing was a one time thing. We have open sourced the codebase. Here's the link: https://github.com/multiversal-ventures/json-buffet https://github.com/multiversal-ventures/json-buffet . Since this was a quick and dirty prototype, comments were sparse. I have updated the Readme, and added a sample json-fetcher. Hope this is more useful for you. Another unwritten TODO was to nudge the data providers towards a more streaming friendly compression formats - and then just create an index to fetch the data directly from their compressed archives. That would have saved everyone a LOT of $$$.