2 ms·
ipython, plus a bit of magic to json.loads to get it to intern the object keys? (Assuming you have enough repetition in the keys to result in enough memory savi
by deathanatos 4y ago
ipython, plus a bit of magic to json.loads to get it to intern the object keys? (Assuming you have enough repetition in the keys to result in enough memory savings to get you within your budget.)
I had to do this once for a large piece of JSON that ballooned quite large when loaded into RAM. But in my case I could wait until after json.loads to do the interning; but I think you can do it on the fly with object_hook or object_pairs_hook.
Alternatively, Rust+Serde & doing as bare-bones a computation to get it working, again with an eye out to whether the decoded form will fit. (Hopefully, since Serde'ing into a struct should make the keys 0 bytes, essentially, but there would be some memory lost for pointers and the like.)
Alternatively, a 20 GiB swap file.