3 ms·

> “There’s absolutely no use case out there, where I search for something, and then I say, hey, I believe my search result will be item #3175 in the current sor

by mlthoughts2018 7y ago

> “There’s absolutely no use case out there, where I search for something, and then I say, hey, I believe my search result will be item #3175 in the current sort order.” I worked for a large ecommerce company that sold digital assets previously (music, photos and other things). It was absolutely a huge use case that people would submit deeply paginated queries directly, such as from saved search result links to resume later, or known good bots that we allowed to crawl and index the search results for customers that integrated our inventory to their APIs. It was such a high priority use case that we actually built a detector system to recognize these queries and divert them to a different pool of query shards to keep the heavy deep paginating load and cache misses off of the pool of shards that served main traffic. Very much a huge part of our pagination design.

wrs 7y ago
Doesn’t that have an even worse problem, in that if you use a query for an extended period, the results are going to change out from under you while you’re scanning them? Seems like you can’t build a stable crawler on that — you’d need to store the actual result list, not redo the query each time.
- mlthoughts2018 7y ago
  This system used pagination keying, not just a link to a page of the results. The link would take you back to the most recently visited item from your pagination key. Results could have changed in that sort order, but that is a risk in all pagination, depending on the freshness of results required by the user.
lukaseder 7y ago
> It was absolutely a huge use case that people would submit deeply paginated queries directly, such as from saved search result links to resume later How would offset pagination matter to these folks? They would have probably had an even better experience with keyset pagination, resuming later after the exact item they left with, not the page which may have random other items on it by the time the resume. > or known good bots that we allowed to crawl and index the search results for customers that integrated our inventory to their APIs. Why not offer an API to those bots? And how does this relate to offset pagination, in the first place?
- mlthoughts2018 7y ago
  > “How would offset pagination matter to these folks? They would have probably had an even better experience with keyset pagination,” we did use pagination keying, see the sibling comment > “Why not offer an API to those bots?” We did offer an API, these calls were from the API. The use case of the bots was to reproduce our search result orderings inside of partner integrations. Inside the partner app, the partner’s own search index of our stuff would utilize our own sort order ranking for a variety of queries that were dynamically changing at the partner’s discretion. Everything they consumed, including paginated items, was through our developer API.

wrs 7y ago

Doesn’t that have an even worse problem, in that if you use a query for an extended period, the results are going to change out from under you while you’re scanning them? Seems like you can’t build a stable crawler on that — you’d need to store the actual result list, not redo the query each time.

mlthoughts2018 7y ago
This system used pagination keying, not just a link to a page of the results. The link would take you back to the most recently visited item from your pagination key. Results could have changed in that sort order, but that is a risk in all pagination, depending on the freshness of results required by the user.

lukaseder 7y ago

> It was absolutely a huge use case that people would submit deeply paginated queries directly, such as from saved search result links to resume later How would offset pagination matter to these folks? They would have probably had an even better experience with keyset pagination, resuming later after the exact item they left with, not the page which may have random other items on it by the time the resume. > or known good bots that we allowed to crawl and index the search results for customers that integrated our inventory to their APIs. Why not offer an API to those bots? And how does this relate to offset pagination, in the first place?

mlthoughts2018 7y ago
> “How would offset pagination matter to these folks? They would have probably had an even better experience with keyset pagination,” we did use pagination keying, see the sibling comment > “Why not offer an API to those bots?” We did offer an API, these calls were from the API. The use case of the bots was to reproduce our search result orderings inside of partner integrations. Inside the partner app, the partner’s own search index of our stuff would utilize our own sort order ranking for a variety of queries that were dynamically changing at the partner’s discretion. Everything they consumed, including paginated items, was through our developer API.