4 ms·

> If items get returned twice while iterating, then I know something got added. Fair point. I agree, that's a nice side effect. But what if you're looking at p

> If items get returned twice while iterating, then I know something got added. Fair point. I agree, that's a nice side effect. But what if you're looking at page 1, and an item is removed from that page? Then you'll never see the first item at page 2, because it's now the last on page 1. > Then if I want a slice from 300-8000th items, I can type exactly that in the URL. That's a nice feature. But it can put a lot of load on your backend if you paginate over 10 of thousands of items.

lhorie 6y ago
> an item is removed How common are concurrent removals in practice though? I can't think of a single instance off the top of my head where this is problematic and you can't just go back to the previous page if you're really paranoid or confused that something is "missing" > it can put a lot of load on your backend if you paginate over 10 of thousands of items Look at it in aggregate. Fiddling w/ URL params to paginate (instead of clicking pagination links) is a power user move. If I want the date of first commit in a repo, I'll only look at the last page (vs paging through the entire history). For guessing, I can click a page, and if I went too far, binary search from there (again, vs linear search). Etc. Even for the most degenerate use case (e.g. some jerk trying to crawl over the entire dataset), the load is smaller with a single request than the overhead of multiple requests. Paginating is not an appropriate mitigation strategy against this type of traffic, and you arguably can implement detection/caching/blocking mechanisms much more easily for naive huge queries than if you need to differentiate regular traffic from bots.
- pc86 6y ago
  > I can't think of a single instance off the top of my head where this is problematic and you can't just go back to the previous page if you're really paranoid or confused that something is "missing" The whole issue is that you're never going to know about it. Sure, you can write some convoluted automated process to double-check previous page(s) but most people aren't going to do that.
  - lhorie 6y ago
    Sure, that's why I'm asking whether this is at all common. If the use case is programmatic, then just provide the ability to get an arbitrarily large set atomically (rather than forcing pagination size limits) and the problem goes away. Another thing to consider when paginating via id cursors: if the deleted item is the one your cursor is sitting on, then you no longer have a frame of reference at all. For example, what happens to commit history pagination in github's implementation if I rewrite git history? Chibicc for example deliberately rewrites its history for didactic purposes.
    - ngrilly 6y ago
      > just provide the ability to get an arbitrarily large set atomically It's difficult to do in most HTTP deployments where we try to limit HTTP response duration. > if the deleted item is the one your cursor is sitting on, then you no longer have a frame of reference at all Not a problem. If your order your results by timestamp and ID, then you provide the timestamp and ID of the last item returned, and take anything after. Doesn't matter if the item is still there or not.
      - lhorie 6y ago
        Not sure what you mean by timestamp, but I don't think that works for the example I posted. If I rewrite the first commit in history, every single entry will have different sha's and commit dates. With offsets, clicking "next" will at best feel like nothing happened, or at worst run into the mild off-by-one confusion. With ids, there's no way for the system to know where to go unless there's custom logic to somehow map old sha's to the new ones (assuming that is even possible).
pawelmi 6y ago
> That's a nice feature. But it can put a lot of load on your backend if you paginate over 10 of thousands of items. To prevent that, usually maximum page size is enforced on the server anyway and the client is informed about the actual page size in the metadata in the reply.

lhorie 6y ago

> an item is removed How common are concurrent removals in practice though? I can't think of a single instance off the top of my head where this is problematic and you can't just go back to the previous page if you're really paranoid or confused that something is "missing" > it can put a lot of load on your backend if you paginate over 10 of thousands of items Look at it in aggregate. Fiddling w/ URL params to paginate (instead of clicking pagination links) is a power user move. If I want the date of first commit in a repo, I'll only look at the last page (vs paging through the entire history). For guessing, I can click a page, and if I went too far, binary search from there (again, vs linear search). Etc. Even for the most degenerate use case (e.g. some jerk trying to crawl over the entire dataset), the load is smaller with a single request than the overhead of multiple requests. Paginating is not an appropriate mitigation strategy against this type of traffic, and you arguably can implement detection/caching/blocking mechanisms much more easily for naive huge queries than if you need to differentiate regular traffic from bots.

pc86 6y ago
> I can't think of a single instance off the top of my head where this is problematic and you can't just go back to the previous page if you're really paranoid or confused that something is "missing" The whole issue is that you're never going to know about it. Sure, you can write some convoluted automated process to double-check previous page(s) but most people aren't going to do that.
- lhorie 6y ago
  Sure, that's why I'm asking whether this is at all common. If the use case is programmatic, then just provide the ability to get an arbitrarily large set atomically (rather than forcing pagination size limits) and the problem goes away. Another thing to consider when paginating via id cursors: if the deleted item is the one your cursor is sitting on, then you no longer have a frame of reference at all. For example, what happens to commit history pagination in github's implementation if I rewrite git history? Chibicc for example deliberately rewrites its history for didactic purposes.
  - ngrilly 6y ago
    > just provide the ability to get an arbitrarily large set atomically It's difficult to do in most HTTP deployments where we try to limit HTTP response duration. > if the deleted item is the one your cursor is sitting on, then you no longer have a frame of reference at all Not a problem. If your order your results by timestamp and ID, then you provide the timestamp and ID of the last item returned, and take anything after. Doesn't matter if the item is still there or not.
    - lhorie 6y ago
      Not sure what you mean by timestamp, but I don't think that works for the example I posted. If I rewrite the first commit in history, every single entry will have different sha's and commit dates. With offsets, clicking "next" will at best feel like nothing happened, or at worst run into the mild off-by-one confusion. With ids, there's no way for the system to know where to go unless there's custom logic to somehow map old sha's to the new ones (assuming that is even possible).

pawelmi 6y ago

> That's a nice feature. But it can put a lot of load on your backend if you paginate over 10 of thousands of items. To prevent that, usually maximum page size is enforced on the server anyway and the client is informed about the actual page size in the metadata in the reply.