5 ms·
Just noting that this library requires that you are able to hold your expanded document in memory. I needed to parse a very very large JSON document and pull o
by grandinj 6y ago
Just noting that this library requires that you are able to hold your expanded document in memory.
I needed to parse a very very large JSON document and pull out a subset of data, which didn't work, because it exceeded available RAM.
- deleted 6y ago[deleted]
- wenc 6y agoAny possibility of using mmap? https://en.wikipedia.org/wiki/Mmap https://en.wikipedia.org/wiki/Mmap
- grandinj 6y agonot when you exceed the size of the pagefile/swapfile :-) But it's a good project, otherwise.
- cma 6y agommap can exceed pagefile/swapfile+ram if you are mapping a file and not anonymous pages.
- grandinj 6y agoyeah. In my situation (a) the file was 10x RAM (b) when this library parses a JSON it creates an in-memory tree representing the file which is approx 16x the size of the JSON file. Consequently it swapped so hard I was never going to finish processing. So I used a streaming parser, and it finished in minutes.
- gnaman 6y agoWhat did you use instead?
- grandinj 6y agoSorry, don't remember, was a throwawy project
- sroussey 6y agoYeah, have this problem reading gig+ files on mobile in JS. Have to stream them.
- TkTech 6y agoDocument streaming is on the roadmap, and jkeiser is actively working on a SAX-style interface. Once these are done you'll be able to control memory usage.