3 ms·

I'd suggest caching the result of a query that could return multiple pages, if consistency is truly important. Otherwise, I don't think it really matters which

by cmckn 6y ago

I'd suggest caching the result of a query that could return multiple pages, if consistency is truly important. Otherwise, I don't think it really matters which option you choose, as long as you document the behavior. Edit: obviously the proper impl totally depends on your app. If the result set is enormous, caching doesn't make sense. If it's rapidly changing, pagination probably doesn't make sense. etc, etc.

ngrilly 6y ago
But then you need to provide some kind of query ID in the the HTTP request? And if you many concurrent clients, that can be expensive in terms of RAM? If we don't care about added items (they can be deduped client-side), and only care about removed items, maybe the backend can maintain a tombstone timestamp on each deleted item, instead of deleting them, and then the client can provide a "snapshot" timestamp in the query that can be compared with the tombstone timestamp.
- cmckn 6y ago
  > But then you need to provide some kind of query ID in the the HTTP request? And if you many concurrent clients, that can be expensive in terms of RAM? A query ID is super easy to implement, and yes, for any application you run you have to be aware of the resource requirements. Tune your eviction policy, and this is totally feasible. Deduping and tombstones is messier to implement IMO, and hitting a cached result for the next page (e.g. a redis LIST) is probably less expensive than a "query" (whatever that means for your backend).
dragonwriter 6y ago
If consistency is really important, don't paginate on the backend at all. The only way to win is not to play the game. That said, using “page of results preceding <first id if following page>” and “page of results following <last id of preceding page>” reduces obvious pagination artifacts compared to <page number n>. Whether it's better to beat consumers over the head with inconsistency due to concurrent changes probably depends on the application or provide a nearer illusion of consistency probably depends on application domain and use case.

by cmckn 6y ago

ngrilly 6y ago

But then you need to provide some kind of query ID in the the HTTP request? And if you many concurrent clients, that can be expensive in terms of RAM? If we don't care about added items (they can be deduped client-side), and only care about removed items, maybe the backend can maintain a tombstone timestamp on each deleted item, instead of deleting them, and then the client can provide a "snapshot" timestamp in the query that can be compared with the tombstone timestamp.

cmckn 6y ago
> But then you need to provide some kind of query ID in the the HTTP request? And if you many concurrent clients, that can be expensive in terms of RAM? A query ID is super easy to implement, and yes, for any application you run you have to be aware of the resource requirements. Tune your eviction policy, and this is totally feasible. Deduping and tombstones is messier to implement IMO, and hitting a cached result for the next page (e.g. a redis LIST) is probably less expensive than a "query" (whatever that means for your backend).

dragonwriter 6y ago

If consistency is really important, don't paginate on the backend at all. The only way to win is not to play the game. That said, using “page of results preceding <first id if following page>” and “page of results following <last id of preceding page>” reduces obvious pagination artifacts compared to <page number n>. Whether it's better to beat consumers over the head with inconsistency due to concurrent changes probably depends on the application or provide a nearer illusion of consistency probably depends on application domain and use case.