3 ms·

Not if you're streaming input data where you cannot know the size ahead of time, and you want to pipeline the processing so that output is written in lockstep w

Not if you're streaming input data where you cannot know the size ahead of time, and you want to pipeline the processing so that output is written in lockstep with the input. It might not be the entire dataset that's streamed. For example, consider serializing something like [fetch(url1), join(fetch(url2), fetch(url3))]. The outer count is knowable, but the inner isn't. Even if the size of fetch(url2) and fetch(url3) are known, evaluating a join function may produce an unknown number of matches in its (streaming) output. JSON, Protobuf, etc. can be very efficiently streamed, but it sounds like MessagePack is not designed for this. So processing the above would require pre-rendering the data in memory and then serializing it, which may require too much memory.

baobun 2y ago
> JSON, Protobuf, etc. can be very efficiently streamed Protobuf yes, JSON no: you can't properly deserialize a JSON collection until it is fully consumed. The same issue you're highlighting for serializing MessagePack occurs when deserializing JSON. I think MessagePack is very much written with streaming in mind. It makes sense to trade write-efficiency for read-efficiency. Especially as the entity primarily affected by the tradeoff is the one making the cut, in case of msgpack. It all depends on your workloads but Ive done benchmarks for past work where msgpack came up on top. It can often be a good fit for when you need to do stuff in Redis. (If anyone thinks to counter with JSONL, well, there's no reason you can't do the same with msgpack).
- atombender 2y ago
  Sorry, I was mentally thinking of writing mostly. With JSON the main problem is, as you say, read efficiency.
- laurencerowe 2y ago
  The advantage of JSON for streaming is on serialization. A server can begin streaming the response to the client before the length of the data is known. JSON Lines is particularly helpful for JavaScript clients where streaming JSON parsers tend to be much slower than JSON.parse.

baobun 2y ago

> JSON, Protobuf, etc. can be very efficiently streamed Protobuf yes, JSON no: you can't properly deserialize a JSON collection until it is fully consumed. The same issue you're highlighting for serializing MessagePack occurs when deserializing JSON. I think MessagePack is very much written with streaming in mind. It makes sense to trade write-efficiency for read-efficiency. Especially as the entity primarily affected by the tradeoff is the one making the cut, in case of msgpack. It all depends on your workloads but Ive done benchmarks for past work where msgpack came up on top. It can often be a good fit for when you need to do stuff in Redis. (If anyone thinks to counter with JSONL, well, there's no reason you can't do the same with msgpack).

atombender 2y ago
Sorry, I was mentally thinking of writing mostly. With JSON the main problem is, as you say, read efficiency.
laurencerowe 2y ago
The advantage of JSON for streaming is on serialization. A server can begin streaming the response to the client before the length of the data is known. JSON Lines is particularly helpful for JavaScript clients where streaming JSON parsers tend to be much slower than JSON.parse.