3 ms·
(1) Try a language with fast allocations (C, C++, Rust, maybe Go or Java) -- anything except Python or Ruby or (2) Try using streaming API (I don't know Ruby,
by theamk 8y ago
(1) Try a language with fast allocations (C, C++, Rust, maybe Go or Java) -- anything except Python or Ruby
or
(2) Try using streaming API (I don't know Ruby, but quick google found https://github.com/dgraham/json-stream https://github.com/dgraham/json-stream ).
Note that this method will require you to massively restructure your program -- you want to avoid having all of the data in memory at once.
The streaming API might work better with jq-based preprocessing -- for example, if you want to compare two unsorted sets, it may be faster to sort them using jq, then compare line-by-line using streaming API.
- dith3r 8y agoPython is fast at parsing JSON, Go had hard time to match parsing speed of it. Additionally you have PyPy to help.
- mattbillenstein 8y agoNod. https://gist.github.com/mattbillenstein/34cf2907390102ffbabd982a3662b204 https://gist.github.com/mattbillenstein/34cf2907390102ffbabd...
- Filligree 8y agoPython is fast at doing anything that doesn't involve running Python. That's an important caveat. Python's C JSON parser library is super-fast, but if you want to use the data for anything but a simple equality check afterwards, it'll be slow as molasses. Or you'll write a C extension for it...