3 ms·
Will this work on JSON files that are larger than the available system memory? Firebase backups are huge JSON files and we haven’t found a good way to deal wit
by yeldarb 8y ago
Will this work on JSON files that are larger than the available system memory?
Firebase backups are huge JSON files and we haven’t found a good way to deal with them.
There are some “streaming JSON parsers” that we have wrestled with but they are buggy.
- nojvek 8y agoProbably not. I requires a memory allocation the size of the file for parsing. However they have the ability to build a tape out of the json and find the interesting marks. Perhaps it can be adapted to make a fast parser than only parses the relevant stuff but zooms through the large file in blocks.
- glangdale 8y agoSadly it will not. Arguably we could 'stream' things, but we don't have an API or a use case for it. If you could capture your requirements and put them on an issue on Github, it would be helpful. We're not against the streaming use case, we just don't understand it very well.