3 ms·

Though you can't avoid seeing an item just before it is deleted and skipping an item just before it is created. This isn't a huge problem but does mean that the

Though you can't avoid seeing an item just before it is deleted and skipping an item just before it is created. This isn't a huge problem but does mean that the list you end up with may not be a coherent view of the database. But if you require guarantees like that then you might be better of just using SQL directly. Or just include everything in a single response, HTTPS was designed to handle arbitrary length payloads if I recall correctly. You might want to use a more streamable format than JSON though.

dharmab 6y ago
If you care about that kind of guarantee you might want to keep a connection open and stream new updates.
- rualca 6y ago
  > If you care about that kind of guarantee you might want to keep a connection open and stream new updates. You don't need that. You just need to get the collection resource to be cacheable and subsequently track your resources' state with conditional requests. https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditional_requests https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditiona...
ajitid 6y ago
> use a more streamable format than JSON Did you mean we should stream data over WebSocket or use HTTP/2 or do we need to do something different altogether?
- alexeldeib 6y ago
  AFAIK chunked transfer works specifically with HTTP 1.1, think Kubernetes for example uses this with JSON in a few prominent places for things like WATCH. But that setup is hardly ideal for the use case.
- cheschire 6y ago
  Check out MessagePack. https://msgpack.org/ https://msgpack.org/
- contravariant 6y ago
  I don't think you need to do anything difficult you can just treat it as one big file, pretty sure online radio streams have been using this technique for ages. Though I'm not too sure about the particulars. This is annoying to do with JSON though because you need to remember to close all your brackets, something like CSV is a lot easier because you can just write the header once and then just stream the data. Edit: Looks like Python supports this using chunked transfer by simply providing the HTTP request data as an iterator.

dharmab 6y ago

If you care about that kind of guarantee you might want to keep a connection open and stream new updates.

rualca 6y ago
> If you care about that kind of guarantee you might want to keep a connection open and stream new updates. You don't need that. You just need to get the collection resource to be cacheable and subsequently track your resources' state with conditional requests. https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditional_requests https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditiona...

ajitid 6y ago

> use a more streamable format than JSON Did you mean we should stream data over WebSocket or use HTTP/2 or do we need to do something different altogether?

alexeldeib 6y ago
AFAIK chunked transfer works specifically with HTTP 1.1, think Kubernetes for example uses this with JSON in a few prominent places for things like WATCH. But that setup is hardly ideal for the use case.
cheschire 6y ago
Check out MessagePack. https://msgpack.org/ https://msgpack.org/
contravariant 6y ago
I don't think you need to do anything difficult you can just treat it as one big file, pretty sure online radio streams have been using this technique for ages. Though I'm not too sure about the particulars. This is annoying to do with JSON though because you need to remember to close all your brackets, something like CSV is a lot easier because you can just write the header once and then just stream the data. Edit: Looks like Python supports this using chunked transfer by simply providing the HTTP request data as an iterator.