3 ms·
There is a simple way and there is a hard way. Simple - memory-map the whole file with `mmap`. We also suggest using `madvice`, to inform the kernel that you w
by ashvardanian 4y ago
There is a simple way and there is a hard way.
Simple - memory-map the whole file with `mmap`. We also suggest using `madvice`, to inform the kernel that you will be accessing the data just once, strictly in the sequential order. And then simply give it to SIMDJSON. With `ondemand` parser, your memory usage will be proportional to the depth of the deepest tree, not their number.
Harder way, would be to parse file in chunks, locating continuous sequences that form a single document.
There may be more options, but I'd have to check the docs :)
- eatonphil 4y agoThanks! Yeah I've seen both general advice but since I'm not incredibly familiar with the API I've had a hard time creating code that works. If you have any code examples of either of those options I'd love to see them! Edit: Also, I'm curious: what are the downsides to doing the mmap route?
- loeg 4y agoDownsides - unpredictable latency for IO when the parser crosses a page boundary and encounters an unbacked page. Unpredictable IO issue size (might be smaller IOs than ideal). Unpredictable caching behavior (OS may drop the pages before the application is done with them, only to have to reread again). You can mitigate this somewhat with explicit MAP_POPULATE and mlock.
- Sesse__ 4y agoAlso, generally higher overhead, more memory use for very large files (the page tables are not free), and no way of handling errors.
- loeg 4y ago> more memory use for very large files (the page tables are not free), They're... kinda free (in terms of capacity). A PTE or PDE is 64 bits per page. Even with 4kB pages, this is 8/4096 = 0.2% overhead. Churning the page tables and TLB as you walk the file is expensive, or at least historically was expensive (used to be that removing an entry required flushing the entire TLB; I don't think that's true anymore). If you can use MAP_HUGE_2MB, this drops off to 0.00038% and the TLB impact goes way down. > no way of handling errors. Yeah, this aspect is really significant. Thanks for mentioning it.
- Sesse__ 4y ago0.2% really isn't free. If you have a data set of 10 TB (one small-ish HDD right now, or one large SSD), that's ~20 GiB in page tables alone. If you can use 2MB pages, then sure, but I would assume that also limits your page-in granularity.
- loeg 4y agoOh, you're right. I wasn't remembering we needed PTEs to cover the data not paged in as well as the data in RAM. Sorry.
- roseway4 4y agoRegarding the hard way, this little utility does a great job of splitting larger than memory JSON documents into collections of NDJSON files: https://github.com/dolthub/jsplit https://github.com/dolthub/jsplit
- AtlasBarfed 4y agoWell, here's an idea. Do you HAVE to have the file as a big json file? How did it end up like that? When you source the data, can you instead output the overall doc as "part" files that are aligned along subdocument boundaries? THen you don't have to solve that "find the subdocument terminator" on stream-back. You are PROBABLY building the json doc via some DB dump, so just change the code that dumps the data. If you need to reconstruct it as a single json doc for some reason, it is about as trivial as it gets to reconstruct the data as a single document. Or programattically, a "file" object that just does the reconstruction on the fly from the source part files as it is read, that should be relatively trivial in lots of languages.