3 ms·
You could stream incrementally like this without explicitly demarcating the "holes". You can simply send the unfinished JSON (with empty arrays as the holes), t
by atombender 1y ago
You could stream incrementally like this without explicitly demarcating the "holes". You can simply send the unfinished JSON (with empty arrays as the holes), then compute the next iteration and send a delta, then compute the next and send a delta, and so on.
A good delta format is Mendoza [1] (full disclosure: I work at Sanity where we developed this), which has Go and JS/TypeScript [2] implementations. It expresses diffs and patches as very compact operations.
Another way is to use binary digging. For example, zstd has some nifty built-in support for diffing where you can use the previous version as a dictionary and then produce a diff that can be applied to that version, although we found Mendoza to often be as small as zstd. This approach also requires treating the JSON as bytes and keeping the previous binary snapshot in memory for the next delta, whereas a Mendoza patch can be applied to a JavaScript value, so you only need the deserialized data.
This scheme would force you to compare the new version for what's changed rather than plug in exactly what's changed, but I believe React already needs to do that? Also, I suppose the Mendoza applier could be extended to return a list of keys that were affected by a patch application.
[1] https://github.com/sanity-io/mendoza https://github.com/sanity-io/mendoza
[2] https://github.com/sanity-io/mendoza-js https://github.com/sanity-io/mendoza-js
- __mattya 1y agoThey want to know where the holes are so that they can show a loading state.
- atombender 1y agoYou don't need templating ($1 etc.) for that as long as you can describe the holes somehow, which can be done out-of-band. If we imagine a streaming protocol of key/value pairs that are either snapshots or deltas: event: snapshot data: {"topPost":[], "user": {"comments": []}} pending: topPosts,user.comments event: delta data: [17,{"comments":[{"body":"hello world"}]},"user"] pending: topPosts
- andrewingram 1y agoFor the use case of streaming data for UI, I don’t think empty arrays and nulls are sufficient information. At any moment during the stream, you need the ability to tell what data is pending. If pending arrays are just returned as empty arrays, how do I know if it’s empty because it’s actually empty, or empty because it’s pending? GraphQL’s streaming payloads try to get the best of both worlds, at any point in time you have a valid payload according the GraphQL schema - so it’s possible to render some valid UI, but it also communicates what paths contain pending data, and then subsequent payloads act as patches (though not as sophisticated as Mendoza’s).
- atombender 1y agoAs I commented in https://news.ycombinator.com/item?id=44150238 https://news.ycombinator.com/item?id=44150238, all you need is a way to express what is pending, which can be done using JSON key paths. Of course, you could do it in-band, too: {"comments": {"state": "pending", "values": []}} …at the cost of needing your data model to be explicit about it. But this has the benefit of being diffable, of course, so once the data is available, the diff is just the new state and the new values.
- andrewingram 1y agoYes, hence the last paragraph in my comment :)
- kiitos 1y agoMendoza doesn't appear to provide a spec?