5 ms·
Library of congress is quite explicit about this on their first search page, rather than showing an irrelevant number. So, having limits isn’t unusual, but not
by Retric 4y ago
Library of congress is quite explicit about this on their first search page, rather than showing an irrelevant number. So, having limits isn’t unusual, but not everyone is deceptive about it.
You Searched: ALL: God
Your search retrieved more records than can be displayed. Only the first 10,000 will be shown.
Titles List: 1-25 of 10000 https://catalog.loc.gov/vwebv/search?searchArg=God&searchCode=GKEY%5E*&searchType=0&recCount=25 https://catalog.loc.gov/vwebv/search?searchArg=God&searchCod...
- kweingar 4y agoI guess they have multiple search interfaces. I navigated to loc.gov and searched directly from the home page. I was sent to this page: https://www.loc.gov/search/?in=&q=god&new=true&st= https://www.loc.gov/search/?in=&q=god&new=true&st= At the top, it says “Results 1-25 of 324,782” and does not mention the limit of 10,000 anywhere.
- Retric 4y agoAhh, looks like the limit on results is 100,000 items on that page. https://www.loc.gov/search/?q=god&sp=4000 https://www.loc.gov/search/?q=god&sp=4000 But you can subdivide the search results by date and get every item on your original list: https://www.loc.gov/search/?dates=1890/1899&q=god&sp=1808 https://www.loc.gov/search/?dates=1890/1899&q=god&sp=1808 So, they have and can show you every single one of those 324,782.
- mattkrause 4y agoIt does once you get to the 4,001st page of the results (which is actually 100k items): "Sorry! We can't process this request. This request exceeds the maximum search results depth." The error page even includes a link to "LC for Robots" (https://labs.loc.gov/lc-for-robots/ https://labs.loc.gov/lc-for-robots/), a list of APIs, since few humans are going get that far on their own. The number here also seems less misleading because all 324,782 results do seem to exist. It doesn't want to generate a pagination for the entire set, but you could get to them by choosing different formats, date ranges, collections etc. The number Google repors, as far as I can tell, needs to be taken on faith.
- deleted 4y ago[deleted]
- sailfast 4y agoYeah - as someone that has run production search clusters before on technologies like Elastic / open search, deep pagination is rarely used and an extremely annoying edge case that takes your cluster memory to zero. I found it best to optimize for whatever is a reasonable but useful for users while also preventing any really seriously resource intensive but low value queries (mostly bots / folks trying to mess with your site) to some number that will work with your server main node memory limits.