2 ms·
Working with large json files is hard to parallelize. Just filtering the objects in a root array can take very long. jqr and gojq both die with OOM when running
by oever 4y ago
Working with large json files is hard to parallelize. Just filtering the objects in a root array can take very long. jqr and gojq both die with OOM when running on large files like
https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.gz https://dumps.wikimedia.org/wikidatawiki/entities/latest-all...
A fast tool to split a json file like that into a format with one json file per line would already help a lot.
- brundolf 4y agoMine can :)