3 ms·
I had to try working with a 100GB single json file. Couldn’t find anything that could work with that sized file.
by asadlionpk 5y ago
I had to try working with a 100GB single json file. Couldn’t find anything that could work with that sized file.
- onion2k 5y agoAt that point you might as well ignore the fact it's JSON and just treat it as a stream.
- iainmerrick 5y agoOut of curiosity, what does "working with" mean here? What operations did you need to perform on it? Streaming reads, transforms, indexed reads, appends, edits? I'm thinking that any general-purpose JSON loader is likely to perform badly for a 100GB file, purely because it'll use 2x (or more likely 10x) as much memory for the parsed representation. So you'd want some kind of special-case reader for huge files -- maybe it just builds some kind of sparse index with pointers back into an mmap of the raw data.