5 ms·
So you save some space on top level keys which are stored once in the header...but you waste space by having to flatten data structures into a table to start wi
by smegel 10y ago
So you save some space on top level keys which are stored once in the header...but you waste space by having to flatten data structures into a table to start with, with key values being repeated over lines, the more deeply nested, the worse it gets.
Stick with JSON document per line.
- thewisenerd 10y agois there necessarily a need to repeat key values over lines? why not have all possible object definitions in the header and data as a single huge array?
- athenot 10y agoYou're describing the type-length-value format, used in some places in digital video or in Open Sound Control. Basically you have a TYPE (fixed width, eg. 16 bits), a LENGTH (again fixed) and a VALUE which is simply a stream of bytes that goes on for LENGTH bytes and is interpreted based on what TYPE it is. You can nest your fields and get very creative in quite a compact way. One could represent a JSON document with TLV by adding an associative map between the field names and their type numbers, then stream away. I always loved the simplicity of TLV. Though nowadays, I'm not sure this has much of an advantage over gzipped JSON.
- ubertaco 10y agoOff-topic: Howdy, Mr. Thénot! Crazy reading the comments and seeing your username here. Hope life is goin' well for ya with the HealthNinja stuff -- haven't seen you since I left Wellcentive (around the same time as mbonnette, if that gives you an idea of who I am).
- proteusguy 10y agoBecause to parse it legally you'd have to store the entire file in memory. That's not actually streaming. The CSJ format puts the object schema and order up front and just applies it to each line afterwards. Stream away...
- mrits 10y agoThere are some good arguments against this format but your points were invalid. Key data in many different types of data is much larger than the actual values. E.g ("average_cpu_cylces_per_min": 60)
- athenot 10y agoGzip or some other compression scheme takes care of that pretty well. It's not rare to do JSON over gzipped HTTP requests.
- proteusguy 10y agoYou gzip csv and csj as well so this is orthogonal to the point.
- MyNameIsFred 10y agoIs it? I think his point is that key repition may not be a relevant factor when a gzip can be presumed.