4 ms·

On a tangent, does anyone have code samples of parsing large JSON newline files with simdjson that don't fit into memory? I.e. where the file overall is many GB

On a tangent, does anyone have code samples of parsing large JSON newline files with simdjson that don't fit into memory? I.e. where the file overall is many GB (more than 4) but individual documents in the top-level are not that big. Most of the code samples for simdjson assume you can load the whole file in memory. I've got an idea for how you can do it with parse_many(json,window) and truncated_bytes() but it would be easier if there were just an example out there I could look at. See pages like [0] and [1] that describe it should be possible but I am just not seeing (or yet able to produce myself) working code. [0] https://github.com/simdjson/simdjson/issues/188 https://github.com/simdjson/simdjson/issues/188 [1] https://github.com/simdjson/simdjson/blob/master/doc/iterate_many.md https://github.com/simdjson/simdjson/blob/master/doc/iterate...

ashvardanian 4y ago
There is a simple way and there is a hard way. Simple - memory-map the whole file with `mmap`. We also suggest using `madvice`, to inform the kernel that you will be accessing the data just once, strictly in the sequential order. And then simply give it to SIMDJSON. With `ondemand` parser, your memory usage will be proportional to the depth of the deepest tree, not their number. Harder way, would be to parse file in chunks, locating continuous sequences that form a single document. There may be more options, but I'd have to check the docs :)
- eatonphil 4y ago
  Thanks! Yeah I've seen both general advice but since I'm not incredibly familiar with the API I've had a hard time creating code that works. If you have any code examples of either of those options I'd love to see them! Edit: Also, I'm curious: what are the downsides to doing the mmap route?
  - loeg 4y ago
    Downsides - unpredictable latency for IO when the parser crosses a page boundary and encounters an unbacked page. Unpredictable IO issue size (might be smaller IOs than ideal). Unpredictable caching behavior (OS may drop the pages before the application is done with them, only to have to reread again). You can mitigate this somewhat with explicit MAP_POPULATE and mlock.
    - Sesse__ 4y ago
      Also, generally higher overhead, more memory use for very large files (the page tables are not free), and no way of handling errors.
      - loeg 4y ago
        > more memory use for very large files (the page tables are not free), They're... kinda free (in terms of capacity). A PTE or PDE is 64 bits per page. Even with 4kB pages, this is 8/4096 = 0.2% overhead. Churning the page tables and TLB as you walk the file is expensive, or at least historically was expensive (used to be that removing an entry required flushing the entire TLB; I don't think that's true anymore). If you can use MAP_HUGE_2MB, this drops off to 0.00038% and the TLB impact goes way down. > no way of handling errors. Yeah, this aspect is really significant. Thanks for mentioning it.
        Sesse__ 4y ago
        0.2% really isn't free. If you have a data set of 10 TB (one small-ish HDD right now, or one large SSD), that's ~20 GiB in page tables alone. If you can use 2MB pages, then sure, but I would assume that also limits your page-in granularity.
        loeg 4y ago
        Oh, you're right. I wasn't remembering we needed PTEs to cover the data not paged in as well as the data in RAM. Sorry.
- roseway4 4y ago
  Regarding the hard way, this little utility does a great job of splitting larger than memory JSON documents into collections of NDJSON files: https://github.com/dolthub/jsplit https://github.com/dolthub/jsplit
- AtlasBarfed 4y ago
  Well, here's an idea. Do you HAVE to have the file as a big json file? How did it end up like that? When you source the data, can you instead output the overall doc as "part" files that are aligned along subdocument boundaries? THen you don't have to solve that "find the subdocument terminator" on stream-back. You are PROBABLY building the json doc via some DB dump, so just change the code that dumps the data. If you need to reconstruct it as a single json doc for some reason, it is about as trivial as it gets to reconstruct the data as a single document. Or programattically, a "file" object that just does the reconstruction on the fly from the source part files as it is read, that should be relatively trivial in lots of languages.
aobdev 4y ago
Just to be clear, are you talking about a file where each line is its own json object rather than the entire file being one large array? If so, it’s easy in Python to do the following: with open(“file.json”) as src: for line in src: json.loads(line)
- eatonphil 4y ago
  I'm talking about with simdjson. Lemire suggested reading line-by-line is not a good idea [0]. So I'm asking about the ideal approach using simdjson, not JSON parsers in general. [0] https://github.com/simdjson/simdjson/issues/188#issuecomment-503292225 https://github.com/simdjson/simdjson/issues/188#issuecomment...
  - MikeDelta 4y ago
    We had to parse thousands of multi GB zipped jsons for financial data. I don't have any code but it involved (in C++) boost gzip to unpack chunks into 100MB blocks and the simdjson iterate_many function to parse the block into a stream. The json files were not line-delimited but every row had a newline (json: [\n{...},\n{...}\n....\n] ), so we had to clean the commas in order for simdjson to process it as if it were newline-delimited. Whatever remained at the end of the buffer (few hundred bytes of incomplete json) we would copy just before the boost-unzip block so that there was a continuation of json data. We also reused the parser and string buffer for performance. Also try to parse the json fields in the right order for fastest performance, if possible.
    - capableweb 4y ago
      > Also try to parse the json fields in the right order for fastest performance, if possible. What do you mean with this precisely, not sure I understand? What is the right order? AFAIK, JSON attributes/keys are not ordered, so there is no "right" order, or order at all.
      - MikeDelta 4y ago
        Agreed on jsons not being ordered. The docs on simdjson mention that if you parse the fields in the order as they appear on file, then simdjson can do it in one iteration. If you get fields in random order, simdjson will have to loop the data a few times. If your json on disk is { "field1": "val1", "field2": "val2" } then it is faster to say (pseudo C++) simdjson::document_reference elem; std::string_view tmp; elem["field1"].get_string().get(tmp); elem["field2"]... than the other way around. This works only if you know what you are looking for and it is consistent on disk. Note that you are reading from an iterator that consumes the value: getting a field twice gives an error; totally different than reading from a dict. See also [1] under "Extracting Values: You can cast a JSON element to a native type..." [1] https://github.com/simdjson/simdjson/blob/master/doc/basics.md#using-the-parsed-json https://github.com/simdjson/simdjson/blob/master/doc/basics....
saidinesh5 4y ago
At my last job, we have had to deal with this. We had huge json files (compressed size of 20+ GB) and had to perform certain operations on the data. I found that rapidjson had a better (not perfect, just better) API to do this. I quickly wrote a little wrapper around it to process json data coming in as streams. This was a writeup about it: https://dinesh.cloud/2022/streaming-json-for-fun-and-profit/ https://dinesh.cloud/2022/streaming-json-for-fun-and-profit/ . If there is any interest, I can ask if they can to open source it.
- sl-dolt 4y ago
  Absolutely interested, on my end at least. I wrote this to manage the transparency in coverage files: https://github.com/dolthub/data-analysis/tree/main/transparency-in-coverage/python/mrfutils https://github.com/dolthub/data-analysis/tree/main/transpare... but I'm always looking for better techniques. Edit: Oh wow, I see you used it on those exact files. How about that.
  - tinselcity 4y ago
    I used the rapidjson streams with my little embedded REST HTTP(s) server library: https://github.com/Edgio/is2/ https://github.com/Edgio/is2/ We needed it for streaming large json from async server sockets. Code link: https://github.com/Edgio/is2/blob/master/include/is2/support/nbq_stream.h https://github.com/Edgio/is2/blob/master/include/is2/support... You just had to implement the interfaces like Peek/Take/Tell/etc. It worked really well for us. Probably not as fast as simdjson, but they used some simd tricks I think for skipping whitespace: https://rapidjson.org/md_doc_internals.html#SkipwhitespaceWithSIMD https://rapidjson.org/md_doc_internals.html#SkipwhitespaceWi...
  - saidinesh5 4y ago
    Ha! Thanks to you, Today I found out how big those uncompressed JSON files really are (the data wasn't accessible to me, so i shared the tool with my colleague and he was the one who ran the queries on his laptop): https://www.dolthub.com/blog/2022-09-02-a-trillion-prices/ https://www.dolthub.com/blog/2022-09-02-a-trillion-prices/ . And yep, it was more or less they way you did with ijson. I found ijson just a day after I finished the prototype. Rapidjson would probably be faster. Especially after enabling SIMD. But the indexing was a one time thing. We have open sourced the codebase. Here's the link: https://github.com/multiversal-ventures/json-buffet https://github.com/multiversal-ventures/json-buffet . Since this was a quick and dirty prototype, comments were sparse. I have updated the Readme, and added a sample json-fetcher. Hope this is more useful for you. Another unwritten TODO was to nudge the data providers towards a more streaming friendly compression formats - and then just create an index to fetch the data directly from their compressed archives. That would have saved everyone a LOT of $$$.

ashvardanian 4y ago

There is a simple way and there is a hard way. Simple - memory-map the whole file with `mmap`. We also suggest using `madvice`, to inform the kernel that you will be accessing the data just once, strictly in the sequential order. And then simply give it to SIMDJSON. With `ondemand` parser, your memory usage will be proportional to the depth of the deepest tree, not their number. Harder way, would be to parse file in chunks, locating continuous sequences that form a single document. There may be more options, but I'd have to check the docs :)

eatonphil 4y ago
Thanks! Yeah I've seen both general advice but since I'm not incredibly familiar with the API I've had a hard time creating code that works. If you have any code examples of either of those options I'd love to see them! Edit: Also, I'm curious: what are the downsides to doing the mmap route?
- loeg 4y ago
  Downsides - unpredictable latency for IO when the parser crosses a page boundary and encounters an unbacked page. Unpredictable IO issue size (might be smaller IOs than ideal). Unpredictable caching behavior (OS may drop the pages before the application is done with them, only to have to reread again). You can mitigate this somewhat with explicit MAP_POPULATE and mlock.
  - Sesse__ 4y ago
    Also, generally higher overhead, more memory use for very large files (the page tables are not free), and no way of handling errors.
    - loeg 4y ago
      > more memory use for very large files (the page tables are not free), They're... kinda free (in terms of capacity). A PTE or PDE is 64 bits per page. Even with 4kB pages, this is 8/4096 = 0.2% overhead. Churning the page tables and TLB as you walk the file is expensive, or at least historically was expensive (used to be that removing an entry required flushing the entire TLB; I don't think that's true anymore). If you can use MAP_HUGE_2MB, this drops off to 0.00038% and the TLB impact goes way down. > no way of handling errors. Yeah, this aspect is really significant. Thanks for mentioning it.
      - Sesse__ 4y ago
        0.2% really isn't free. If you have a data set of 10 TB (one small-ish HDD right now, or one large SSD), that's ~20 GiB in page tables alone. If you can use 2MB pages, then sure, but I would assume that also limits your page-in granularity.
        loeg 4y ago
        Oh, you're right. I wasn't remembering we needed PTEs to cover the data not paged in as well as the data in RAM. Sorry.
roseway4 4y ago
Regarding the hard way, this little utility does a great job of splitting larger than memory JSON documents into collections of NDJSON files: https://github.com/dolthub/jsplit https://github.com/dolthub/jsplit
AtlasBarfed 4y ago
Well, here's an idea. Do you HAVE to have the file as a big json file? How did it end up like that? When you source the data, can you instead output the overall doc as "part" files that are aligned along subdocument boundaries? THen you don't have to solve that "find the subdocument terminator" on stream-back. You are PROBABLY building the json doc via some DB dump, so just change the code that dumps the data. If you need to reconstruct it as a single json doc for some reason, it is about as trivial as it gets to reconstruct the data as a single document. Or programattically, a "file" object that just does the reconstruction on the fly from the source part files as it is read, that should be relatively trivial in lots of languages.

aobdev 4y ago

Just to be clear, are you talking about a file where each line is its own json object rather than the entire file being one large array? If so, it’s easy in Python to do the following: with open(“file.json”) as src: for line in src: json.loads(line)

eatonphil 4y ago
I'm talking about with simdjson. Lemire suggested reading line-by-line is not a good idea [0]. So I'm asking about the ideal approach using simdjson, not JSON parsers in general. [0] https://github.com/simdjson/simdjson/issues/188#issuecomment-503292225 https://github.com/simdjson/simdjson/issues/188#issuecomment...
- MikeDelta 4y ago
  We had to parse thousands of multi GB zipped jsons for financial data. I don't have any code but it involved (in C++) boost gzip to unpack chunks into 100MB blocks and the simdjson iterate_many function to parse the block into a stream. The json files were not line-delimited but every row had a newline (json: [\n{...},\n{...}\n....\n] ), so we had to clean the commas in order for simdjson to process it as if it were newline-delimited. Whatever remained at the end of the buffer (few hundred bytes of incomplete json) we would copy just before the boost-unzip block so that there was a continuation of json data. We also reused the parser and string buffer for performance. Also try to parse the json fields in the right order for fastest performance, if possible.
  - capableweb 4y ago
    > Also try to parse the json fields in the right order for fastest performance, if possible. What do you mean with this precisely, not sure I understand? What is the right order? AFAIK, JSON attributes/keys are not ordered, so there is no "right" order, or order at all.
    - MikeDelta 4y ago
      Agreed on jsons not being ordered. The docs on simdjson mention that if you parse the fields in the order as they appear on file, then simdjson can do it in one iteration. If you get fields in random order, simdjson will have to loop the data a few times. If your json on disk is { "field1": "val1", "field2": "val2" } then it is faster to say (pseudo C++) simdjson::document_reference elem; std::string_view tmp; elem["field1"].get_string().get(tmp); elem["field2"]... than the other way around. This works only if you know what you are looking for and it is consistent on disk. Note that you are reading from an iterator that consumes the value: getting a field twice gives an error; totally different than reading from a dict. See also [1] under "Extracting Values: You can cast a JSON element to a native type..." [1] https://github.com/simdjson/simdjson/blob/master/doc/basics.md#using-the-parsed-json https://github.com/simdjson/simdjson/blob/master/doc/basics....

saidinesh5 4y ago

At my last job, we have had to deal with this. We had huge json files (compressed size of 20+ GB) and had to perform certain operations on the data. I found that rapidjson had a better (not perfect, just better) API to do this. I quickly wrote a little wrapper around it to process json data coming in as streams. This was a writeup about it: https://dinesh.cloud/2022/streaming-json-for-fun-and-profit/ https://dinesh.cloud/2022/streaming-json-for-fun-and-profit/ . If there is any interest, I can ask if they can to open source it.

sl-dolt 4y ago
Absolutely interested, on my end at least. I wrote this to manage the transparency in coverage files: https://github.com/dolthub/data-analysis/tree/main/transparency-in-coverage/python/mrfutils https://github.com/dolthub/data-analysis/tree/main/transpare... but I'm always looking for better techniques. Edit: Oh wow, I see you used it on those exact files. How about that.
- tinselcity 4y ago
  I used the rapidjson streams with my little embedded REST HTTP(s) server library: https://github.com/Edgio/is2/ https://github.com/Edgio/is2/ We needed it for streaming large json from async server sockets. Code link: https://github.com/Edgio/is2/blob/master/include/is2/support/nbq_stream.h https://github.com/Edgio/is2/blob/master/include/is2/support... You just had to implement the interfaces like Peek/Take/Tell/etc. It worked really well for us. Probably not as fast as simdjson, but they used some simd tricks I think for skipping whitespace: https://rapidjson.org/md_doc_internals.html#SkipwhitespaceWithSIMD https://rapidjson.org/md_doc_internals.html#SkipwhitespaceWi...
- saidinesh5 4y ago
  Ha! Thanks to you, Today I found out how big those uncompressed JSON files really are (the data wasn't accessible to me, so i shared the tool with my colleague and he was the one who ran the queries on his laptop): https://www.dolthub.com/blog/2022-09-02-a-trillion-prices/ https://www.dolthub.com/blog/2022-09-02-a-trillion-prices/ . And yep, it was more or less they way you did with ijson. I found ijson just a day after I finished the prototype. Rapidjson would probably be faster. Especially after enabling SIMD. But the indexing was a one time thing. We have open sourced the codebase. Here's the link: https://github.com/multiversal-ventures/json-buffet https://github.com/multiversal-ventures/json-buffet . Since this was a quick and dirty prototype, comments were sparse. I have updated the Readme, and added a sample json-fetcher. Hope this is more useful for you. Another unwritten TODO was to nudge the data providers towards a more streaming friendly compression formats - and then just create an index to fetch the data directly from their compressed archives. That would have saved everyone a LOT of $$$.