3 ms·
> Is it really better than just using an array, though? In many cases, absolutely yes it's better. Otherwise you can end up being forced to cram several GB or
by metadat 4y ago
> Is it really better than just using an array, though?
In many cases, absolutely yes it's better. Otherwise you can end up being forced to cram several GB or even 10's of GB of JSON into working memory. Not ideal.
Usually each ndjson file is generated by a single process, so there isn't much concern about invalid formatting making it's way in. It's one of those rare win-wins.
- e12e 4y agoI could see how a language might have a streaming json parser, but where a naïve "for each line in file" might read the file as a string - allocating all that memory anyway?
- metadat 4y agoUsed properly, "For each line in file" constructs in the major languages (Python, Go, Java, C#, Javascript, and so on) are well tested and really do read only one single line at a time. Just be sure to avoid the usual pitfalls, e.g. don't turn all the lines into a list before starting iteration. I actually recently tried out a python based steaming JSON parser I found on SO and it was actually even more inefficient (for both CPU and memory) than the built-in JSON package (probably because Python and not a C-module). It was also quite a complex state machine, tricky to catch all corner cases. The SO module had some bugs that I fixed, but I ended up abandoning the entire approach because it wasn't saving any memory. NDJSON certainly is a much simpler approach, which I like and appreciate.