3 ms·
"JSON Lines" isn't "streamable JSON", it's just JSON that's easier to use with existing tools in a streaming fashion. Just like you can stream an HTML/XML file
by hombre_fatal 6y ago
"JSON Lines" isn't "streamable JSON", it's just JSON that's easier to use with existing tools in a streaming fashion. Just like you can stream an HTML/XML file with a SAX parser, you can do the same with JSON.
I don't think there's any reason it's necessary to have an 190gb json file, nor anything stopping one from incrementally dumping it into Sqlite. Though it would depend on the format of the proprietary file.
But I will add that there's an obvious benefit to dumping the data into the database: it has indexes and querying capacity that doesn't involve full 190GB file scans. The I/O of a 190gb scan alone takes time.
- smt88 6y agoThe DB can also provide transparent compression, turning that 190GB file into something hopefully a small fraction of the size.