3 ms·
Loading 1GB json files is still slow.
by visarga 4y ago
Loading 1GB json files is still slow.
- mrkeen 4y agoIf you specify the type of the data, not just the size, then yep: it wasn't really about the number of bytes read, but what the CPU had to do to process it.
- flohofwoe 4y agoThat's usually not because of IO but because of dumb memory management in the JSON parser implementation. Some JSON parsers build a huge tree out of individually allocated items for each JSON property, or in higher level languages, build an object/property tree out of the JSON structure. That's guaranteed to be slow for large trees.
- krapp 4y agoI don't want to be that guy but having 1GB json files to begin with seems like the bigger problem.
- omgtehlion 4y agoNot if you use SAX-like (or "evented") parser