4 ms·
One particular downside to using an actual array instead of something like JSON lines is that you can't really stream the data. If you have so much data that y
by Arch485 3y ago
One particular downside to using an actual array instead of something like JSON lines is that you can't really stream the data.
If you have so much data that you need to stream it, you probably shouldn't be using JSON, though...
- fermisea 3y agoShopify, for instance, uses this format if you request bulk downloads of records, for this exact reason.
- vsnf 3y ago>If you have so much data that you need to stream it, you probably shouldn't be using JSON, though... Why not? I have a bunch of large/real-time files I'd like to read piece by piece and json is a convenient format for parsing it to something useful on the client. I understand using other, more efficient formats, but I think just wanting to stream the data isn't enough of a reason to move to them.
- Arch485 3y agoWell, I said _need_ to stream it. I think that's slightly different: I _want_ to stream data almost all the time, but I _need_ to stream data (e.g. due to RAM limitations) fairly infrequently. If you have so much data (e.g. 8GB+, yes my workstation "only" has 8GB of RAM) that it _must_ be streamed, in my personal opinion you should opt for a better format than JSON. It's likely, too, that with a more specialized/space-efficient format, you may no longer need to stream the data!
- baq 3y agoYou can, this has been done since xml was a thing (yes some of us are old). The api to do that is necessarily horrible, so no one sane does, though.