4 ms·

I think this quote explains the efficient streaming: > There is an interesting observation: when a stream does not contain removal entries it can be safely reo

I think this quote explains the efficient streaming: > There is an interesting observation: when a stream does not contain removal entries it can be safely reordered. So if I'm understanding, the example in the readme could be sent in reverse, allowing the client to immediately use root:0 and then string:2 while the rest streams in. I was looking for something like this, but my use case exceeds the 65k key limit for objects.

pshirshov 1y ago
> 65k key limit for objects The limit comes from 2-byte element pointer size. That can be adjusted. We don't have an implementation with larger pointers but it can be done easily. > while the rest streams in Yes, there are many usecases where you can use some chunks of the data/rebuild some parts of the structures immediately without any accumulation. The problem is that we don't have a nice streaming abstraction which would suit anyone for any usecase. SICK as a library is an efficient indexed binary storage for JSON with listed limitations. SICK as a concept is much more but you might need your own implementation tailored to your usecase.
gethly 1y ago
If the stream contains removal entries, then that is not a data stream but command stream and, again, has nothing to do with JSON itself. We can extrapolate this to some kind of json-rpc-ish stream if we want to keep this aligned with JSON. Overall, I think that mentioning JSON here at all is simply a mistake. It would be better to just introduce this as streaming protocol/framework for data structures. But then we can do the same thing with literally any format and syntax.
- pshirshov 1y ago
  > has nothing to do with JSON itself. It's literally a deduplicated indexed binary storage for JSON (plus an approach to JSON representation more suitable for streaming than serialized JSON). > we can do the same thing I would highly encourage doing same things! For some reason people love to fully parse a 50MiB long JSON document when they need just 0.1% of the data in it.

pshirshov 1y ago

> 65k key limit for objects The limit comes from 2-byte element pointer size. That can be adjusted. We don't have an implementation with larger pointers but it can be done easily. > while the rest streams in Yes, there are many usecases where you can use some chunks of the data/rebuild some parts of the structures immediately without any accumulation. The problem is that we don't have a nice streaming abstraction which would suit anyone for any usecase. SICK as a library is an efficient indexed binary storage for JSON with listed limitations. SICK as a concept is much more but you might need your own implementation tailored to your usecase.

gethly 1y ago

If the stream contains removal entries, then that is not a data stream but command stream and, again, has nothing to do with JSON itself. We can extrapolate this to some kind of json-rpc-ish stream if we want to keep this aligned with JSON. Overall, I think that mentioning JSON here at all is simply a mistake. It would be better to just introduce this as streaming protocol/framework for data structures. But then we can do the same thing with literally any format and syntax.

pshirshov 1y ago
> has nothing to do with JSON itself. It's literally a deduplicated indexed binary storage for JSON (plus an approach to JSON representation more suitable for streaming than serialized JSON). > we can do the same thing I would highly encourage doing same things! For some reason people love to fully parse a 50MiB long JSON document when they need just 0.1% of the data in it.