12 ms·

Google’s “million’s of search results” are not being served

265 points by vincent_s 4y ago

kazinator 4y ago
TL;DR: "You know that Google problem everyone knows about, how it reports millions or even billions of 'results', but actually serves a couple hundred at most? Well, shocker of shockers, it turns out that the API has the same problem!" Google really needs to stop using the word 'result' for whatever it supposedly denotes, which is probably something like the estimated number of pages crawled by Google which contain hits for those terms, the majority of which will not be a search 'result' (something delivered by the search function) under any circumstances.
pilgrimfff 4y ago
I was expecting the number of search results here to be much higher - like who cares if Google only serves the first million results out of a billion? Very interesting to see that Google will only serve a few hundred links when they claim to have hundreds of thousands of relevant results indexed. I'm very curious where Google is getting that count and why the reality is so different. Systematic overcounting? Suppressing hundreds of thousands of results?
- kevincox 4y ago
  I used to work at Google but not in search, these are just my own guesses. > where Google is getting that count This is very likely a fairly accurate of the number of pages in Google's index that "match" the search query. Basically exactly what you would expect when you see the number. > why the reality is so different Cost reasons. Most search engines are more or less scanning down a sorted list of pages. The further you need to scan the more expensive it is. Just like running "OFFSET 1000" is usually slow in SQL. At some point the quality of results is generally very low and the cost is growing so it makes sense overall for Google to just cut it off to prevent it becoming an abuse vector (imagine just asking Google for the 10 millionth page of results for "cat"). The fact that few people realize that Google has a page limit shows how rarely people actually want more pages.
  - yamtaddle 4y ago
    > The fact that few people realize that Google has a page limit shows how rarely people actually want more pages. I used to (years and years ago) go past the first page pretty often, but results are so bad now that it rarely helps, so I almost never even click "2", let alone later pages. It's all gonna be obviously-irrelevant crap google "helpfully" found for me or the auto-generated spam that google used to try to fight (circa 2008 and earlier) but no longer seems to, just letting it gunk up and dominate up any results you get that aren't from a handful of top sites. So this is in part one of those "we broke a thing and now no-one uses it, guess they didn't want it!"
    - kevincox 4y ago
      The thing has always been "broken". Google has had a page limit for at least a decade.
      - yamtaddle 4y ago
        No, by "broken" I mean "let lazy auto-generated spam take over the results almost completely". So now those of us who did used to browse past page one (which, to be fair, may not have been many people) don't bother anymore. [EDIT] For those who weren't around for it, Google used to play cat-n-mouse with spam-site operators. It'd go through cycles where results would slowly get worse, then suddenly a ton better, though never as bad as they are today. Around '08 or '09 they (evidently, I'm just judging from the search engine's behavior starting around then and continuing to this day) seemed to give up and just boosted a relatively small set of sites way up the results, abandoning the rest to the spammers.
        Melatonic 4y ago
        Kagi has been working very well for me as an alternative
        pbadams 4y ago
        Part of the difficulty is, if very few people are browsing to page 2, deciding what to put on page 2 becomes harder and harder. Google has a lot of user behavior signals to decide what should be in results 1-10. Deciding if a page should be ranked 20, 200, or 2000 without any user clicks to check if you're right is really difficult. I would bet that since 2008/9, the relative numbers of spam site operators, Google engineers, second-page searches have changed significantly.
- yazaddaruvala 4y ago
  The problem is generally called "deep pagination". It's extremely inefficient to compute. Specifically, counting requires very low memory. When data is spread across 10,000 computers, all of them counting returns just 10,000 numbers i.e. 4 bytes * 10,000 = 40KB. It's easy for 1 computer to count those 10,000. Even at 100,000 computers 400KB. Merging sorted search results is extremely memory intensive. Even with just the Id+Score pair, let's say 8 bytes. To get the 10,000th search result, each computer needs to create a List of 10,000 results, thats 10,000 * 10,000 * 8 bytes = 800 MB. For the 100,000th search result 10,000 * 100,000 * 8 bytes = 8 GB. OR if your data grows to 100,000 computers, thats 100,000 * 100,000 * 8 bytes = 80 GB of intermediate results to process at the end. As you can see this doesn't scale well. You're required to retain context (i.e. sessions) of the search in memory instead, and get the search engine to better coordinate across all 100,000 computers. This also has scaling limitations based on memory of the session, the number of computers, the number of sessions, and their TTL (someone can leave the search page open for day and hit "next page" - should the sessions still be open? Thats an answer each search engine has to decide). The reality is, if a customer wants deep pagination, they are better suited to a full data dump (i.e. full table scan) or using an async search API, rather than a sync search API.
  - generalizations 4y ago
    Well at that point, who really cares if the content of the 1001s page is deterministic, or in perfect order? Get the first 100 or so pages right, and thereafter just request the nth results from each of those m computers. No merge and no memory explosion, you'll just get them slightly out of order.
    - kevincox 4y ago
      You still need to filter based on the other indexes. If you search for [bitcoin mining] you don't want to find pages related to coal mining. So this data still needs to be joined.
      - dekhn 4y ago
        the search term for this is intersection. The posting lists for the two terms are intersected, then the results are ranked. But there are a lot more steps in a production search engine. The long and short of it is if you really want the full results, just join google, join the search team, and then get enough experience so that you can do full queries over the docjoins directly. This was part of Norvig's pitch to attract researchers a while ago. For a research project, I built a regular expression that matched DNA sequences and spat out the list of all pages containing what looked like DNA and then annotated the pages so in principle you could have done dna:<whatever sequence> but obviously that was not a goal for the search team.
kweingar 4y ago
I was curious, so I tried to do this with GitHub search. The search “Golang” returns 225k+ repositories. Each page of results has 10 repositories. I queried page 1000 to see the 10001st-10010th results and got a 404. I searched Wikipedia (using its own search page) for “America” and was told there were 1.9M results. I requested the 10001st result and it failed with this message: “An error has occurred while searching: Could not retrieve results. Up to 10000 search results are supported, but results starting at 10000 were requested.” There was an HN thread along these lines about web search somewhat recently, with people making confident accusations about false advertising etc. But it really seems that everyone does this. I don’t know, maybe it’s still wrong or misleading. But personally I’m ok knowing “there are X hits and we will serve you some of them”. EDIT: I just searched the Library of Congress for “god” and was told I was viewing results 1-25 of 324,782. Sure enough when I asked for page 10,000 I was rebuffed. I really don’t think this is a Google thing
- Retric 4y ago
  Library of congress is quite explicit about this on their first search page, rather than showing an irrelevant number. So, having limits isn’t unusual, but not everyone is deceptive about it. You Searched: ALL: God Your search retrieved more records than can be displayed. Only the first 10,000 will be shown. Titles List: 1-25 of 10000 https://catalog.loc.gov/vwebv/search?searchArg=God&searchCode=GKEY%5E*&searchType=0&recCount=25 https://catalog.loc.gov/vwebv/search?searchArg=God&searchCod...
  - kweingar 4y ago
    I guess they have multiple search interfaces. I navigated to loc.gov and searched directly from the home page. I was sent to this page: https://www.loc.gov/search/?in=&q=god&new=true&st= https://www.loc.gov/search/?in=&q=god&new=true&st= At the top, it says “Results 1-25 of 324,782” and does not mention the limit of 10,000 anywhere.
    - Retric 4y ago
      Ahh, looks like the limit on results is 100,000 items on that page. https://www.loc.gov/search/?q=god&sp=4000 https://www.loc.gov/search/?q=god&sp=4000 But you can subdivide the search results by date and get every item on your original list: https://www.loc.gov/search/?dates=1890/1899&q=god&sp=1808 https://www.loc.gov/search/?dates=1890/1899&q=god&sp=1808 So, they have and can show you every single one of those 324,782.
ravenstine 4y ago
I don't know where that quote is, but I remember reading one from someone at The Google who, in response to a question about DuckDuckGo, said something along the lines of "We're not competing on search." Turns out what they were saying was pretty truthful, assuming I'm not misremembering. Their "search", last I looked, shoves a bunch of crap above the fold that aren't exactly the results per se. The actual results usually aren't plentiful like I remember them once being. YouTube search has become astonishingly worse to such an extent that I get maybe 3 relevant results and the rest being videos I've already watched or stuff I'm clearly not interested in.
pessimizer 4y ago
Has anybody noticed that a Duckduckgo search (at least with infinite scroll) will serve you the same first page of search results over and over again? I've been trying to figure out whether that's a bug or if they're really fluffing one page into the appearance of multiple pages. I think the internet is closing up.
- 42365767567 4y ago
  Same with Google Images. If you keep hitting more and more results, it'll just recycle the top ones.
  - pessimizer 4y ago
    How can it be anything but intentional to give you the same image over and over again in an image search? That's not something you could possibly miss when checking for the quality of results.
    - propogandist 4y ago
      another search means another opportunity to serve you more ads for different keywords
- turminal 4y ago
  Maybe they used a char to store the array index and you hit an integer overflow?
- omega3 4y ago
  I've noticed plenty of websites now do this.
- Liquix 4y ago
  It seems DDG has recently started ignoring entire keywords in a 3 or 4 word search phrase without informing the user a keyword has been excluded. Something like "2015 camry leaking exhaust" could return nothing but "2015 camry exhaust" results until "leaking" is put in quotes (not a working example). Google search ignoring keywords was the straw that broke the camel's back - is DDG following suit?
ValleZ 4y ago
It shows not the number of shown results which is pretty much same for any query, but number of all pages that contain the keywords. It is not really feasible to rank and show millions of pages, but is possible to do that for few hundred pages. This is why search engines work fast.
- Theodores 4y ago
  It was once possible to trawl through results pages 'near indefinitely'. How I see it is that it is like how the music search works with Google Assistant. It can do it for 90% of the songs you are likely to hear without an online connection. For anything else it can do the search on the server. When you search something it is not as if a billion Google servers are consulted with the results delivered accordingly. The nearest box just knows 90% of the 'answers' and you get whatever that is, rather than a 'real search'. If you enter in a specific code, e.g. a ISBN number, then that is like the 'unusual song' that needs actual looking up. However it works, it is a good hustle. Actually nearer to 'the hitch hikers guide to the galaxy' in having a compiled book of answers.
gkoberger 4y ago
I actually like this feature. I don't want to see the 1,000th results (who would?), but it does help with perspective. If I'm searching for something that should be common, having an order of magnitude helps me realize I'm going in the wrong direction.
crazygringo 4y ago
I can't even tell what this blog post is complaining about, it seems so badly organized and written. But it seems obvious that if "coffee" gives 2 billion results that, no, you're not going to be able to browse to page 187,398,384 to get those results. There's no use case for that for any normal consumer (as opposed to competitor, researcher, etc.). If you're capped at browsing the first 10 pages or whatever, that's entirely reasonable. > A misconception regarding Google’s search results is that all of the results are being served to the user conducting that particular search. That misconception lies only with the author. Nobody's being "served" 2 billion results, I don't even know what that would mean. The number of results being reported is quite obviously in order to allow users to judge the breadth of search queries. If it says 2 billion, you might want to refine. If it says 15 and they're all useless, go broader. (It's useful for researching item popularity too, although that's been superseded by Google Trends which is built specifically for that.)
- Grustaf 4y ago
  I agree that it's very poorly written. The heading is especially confusing, it took me a long time to understand that "SerpAPI" was the search term, not a Google service. It is also the name of the company he's promoting, which makes it even more confusing. But I don't agree that it's "obvious" that "you're not going to be able to browse to page 187,398,384 to get those results. I would argue that a search result is only a search result if you can actually view it. If not, it's just a marketing, or statistics.
- admax88qqq 4y ago
  Feels like SEO blogspam trying to capitalize on Google's currently negative public profile after the Stadia cancellation, and general negative google sentiment. Really easy to get clicks (and upvotes on HN apparently) by complaining about literally anything Google related.
  - adamrezich 4y ago
    why do you think this? nothing in the article indicates this, and Google's questionable search results (compared to what they once were) has been a nearly constant topic of discussion for years now
the_third_wave 4y ago
Try searching for "january 6" on bing for some insight in how results are both manipulated as well as censored: https://www.bing.com/search?q=january+6 https://www.bing.com/search?q=january+6 Once you get past page 6 of the 96 200 000 results they keep on repeating the same results, page after page, with either a Guardian (January 6 committee postpones Wednesday hearing over …) or (every now and then) Yahoo (Jan. 6 hearings to resume following bombshell revelation about …) article on top of the page. The rest of the page is largely identical, page upon page until it comes to page 32 (307-316 of 96 200 000 results). Whatever you tell it to do beyond that page it will always serve page 32 with that Guardian article on top and the other - similarly slanted - results below it. I don't know whether this is just another example of typical Microsoft incompetence in that they make their meddling with the results so incredibly obvious or whether they're just telling visitors this is what they should read and nothing else but it does show these search engines are as unreliable when it comes to politically sensitive topics as e.g. Wikipedia is.
- Melatonic 4y ago
  That is not how it is behaving at all for me
  - the_third_wave 4y ago
    What do you see for that query? Are you logged in? Which country? Which browser? Here: Not logged in, Sweden, Firefox on Linux/Firefox on Android. Same results every time, on both platforms. I took screenshots for pages 6-15, compare these to what you see and let us know in what way your results are different: https://imgur.com/a/QGT044T https://imgur.com/a/QGT044T
- pessimizer 4y ago
  DDG draws at least the vast majority of its results from Bing, and does the same thing. Also, all of their news results seem to be links to MSN and Yahoo versions of stories that were published on other websites. I don't know that what you're seeing has anything to do with politics. For example, I just searched for "dog food nutrition," (https://duckduckgo.com/?q=dog+food+nutrition https://duckduckgo.com/?q=dog+food+nutrition) and https://www.akc.org/expert-advice/nutrition/soy-in-dog-food-what-you-need-to-know/ https://www.akc.org/expert-advice/nutrition/soy-in-dog-food-... Was the 14th result, and also the 36th, 68th, and 87th. It is also the 108th result, where it first starts topping the page, and from then on it's at the top of the rest of the pages, which are repetitions, so 128th, 148th, 168th, unto infinity. I just counted this result because it was topping the repeated page of results, but all of the other results were repeating arbitrarily until it was down to the same 20 results repeating. I'm tempted to sum the number of appearances of every result. I'm not sure there are more than 40. Why "dog food nutrition?" I figured there would be a lot of hits. I was wrong. ----- edit: https://www.bing.com/search?q=dog+food+nutrition https://www.bing.com/search?q=dog+food+nutrition https://www.akc.org/expert-advice/nutrition/soy-in-dog-food-what-you-need-to-know/ https://www.akc.org/expert-advice/nutrition/soy-in-dog-food-... 22nd result, 31st, 45th, 59th, 73rd, and plenty from there on out. https://www.petmd.com/dog/nutrition/can-dogs-eat-peaches https://www.petmd.com/dog/nutrition/can-dogs-eat-peaches 10th result, 20th, 34th, 48th, 51st, 66th, 70th, 88th, 93rd... ... and the pages start repeating from page 9 on with our old friend "Soy in Dog Food?" on top.
tethys 4y ago
Related comment from someone from Google from not too long ago: > It's because the counts are very fast, rough estimates. And when you go into additional pages, we start to refine them. https://news.ycombinator.com/item?id=32354785 https://news.ycombinator.com/item?id=32354785
verisimi 4y ago
Isn't it great that google has got rid of so many bugs? After all, more than 1 result is a bug: https://www.youtube.com/watch?v=XeIIpLqsOe4 https://www.youtube.com/watch?v=XeIIpLqsOe4
pessimizer 4y ago
With: https://www.bing.com/search?q=dog+food+nutrition https://www.bing.com/search?q=dog+food+nutrition ----- https://www.akc.org/expert-advice/nutrition/soy-in-dog-food-what-you-need-to-know/ https://www.akc.org/expert-advice/nutrition/soy-in-dog-food-... is the 22nd result, 31st, 45th, 59th, 73rd, and plenty from there on out. ----- https://www.petmd.com/dog/nutrition/can-dogs-eat-peaches https://www.petmd.com/dog/nutrition/can-dogs-eat-peaches is the 10th result, 20th, 34th, 48th, 51st, 66th, 70th, 88th, 93rd... ... and the pages start repeating from page 9 on with the previous "Soy in Dog Food?" on top. ----- Have the search engines given up on search? Also, can they really say they're being gamed if they're serving the same results up multiple times? Seems more like they're picking winners. This search is pretending like <40 results is hundreds of results, and someone had to make the effort to make sure that the same link wouldn't show up in the same page of results. That strikes me as a deliberate falsification. edit: As far as I can tell from https://www.google.com/search?q=dog+food+nutrition https://www.google.com/search?q=dog+food+nutrition it seems like google are doing a hell of a lot better. Google is also banned from setting cookies or using localstorage on my machine, so the result isn't search bubbled. Nothing about soy or peaches, and no repeats are jumping out at me even 100 results in.
jeffbee 4y ago
Imagine believing there's a sensible discrimination between the 200000th result and the 200001st result. The relative ranking among the top results is supportable but below the top 10 and especially the top 100 the relative ranking has no signal.
- bitcurious 4y ago
  I might not be able to discern between the 1000 and 1001 but there’s no reason I can’t discern between 1001 and 100001. Things can be fuzzy nearby while still being coherent overall.
  - saalweachter 4y ago
    I used to believe that, until I played https://semantle.com/ https://semantle.com/.
lousken 4y ago
yea, google has been terrible at this for a while now, the most annoying thing is that you can't get to those results, i haven't found a way which is quite frustrating if you can't find what you're searching for in those few results
evancox100 4y ago
It makes sense if you separate the indexing process from the result ranking/retrieval process. Building up the index it has encountered the word “dog” millions (billions?) of times, but the Google search API’s job is NOT to operate as a catalogue that will list every single web page on the internet with the word “dog”. Instead its job is to show you what it thinks are the most relevant dog results. That said I think this could be communicated more clearly when you get to the end of the results. The fact that the results number _changes_ is an own goal that makes it seem like they have something to hide. Instead it should just say “Google Search shows a selection of the most relevant occurrences of your search term. If you didn’t find what you need, try XYZ.”
dredmorbius 4y ago
Google is hardly the only search engine which exhibits this behaviour, and the information of total possible matches is useful --- it indicates a grossly generic search phrase --- even where all results are not presented. As I noted recently in a similar thread (<https://news.ycombinator.com/item?id=32923468 https://news.ycombinator.com/item?id=32923468>, HN's search through Algolia is similar. An unqualified search reports (at this writing) somewhat north of 30 million results (30,121,402). It will display only 34 pages' worth of results, I believe 1,000 in total. See: <https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...> Final page: <https://hn.algolia.com/?dateRange=all&page=33&prefix=false&query=&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=33&prefix=false&q...> As discussed recently on HN, the match count is useful information even where all matching results are not displayed as it indicates whether or not a query is generic. Contrast search: <https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=Squeamish%20Ossifrage&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...> (one result as of this writing). I can understand some confusion on this point, but implying some sort of gross fraud or deception suggests far more wrong with the complainant, not their subject.
spullara 4y ago
These posts are just stupid. Search is for finding what you are looking for. I feel like most of the posts in here think that it should be for browsing content with the word in it. There is nothing to see here.
- sleepymoose 4y ago
  >Search is for finding what you are looking for. And at that, it performs miserably.
- kuschku 4y ago
  I want an index, like in a library, with the ability to filter on it. I want to know all content, add many different filters, sort by different properties, and get all results. Just like I'd do with an auto sort/filter table in excel
  - spullara 4y ago
    Oh, then you want to use this instead of a search engine: https://medium.com/@brevityinmotion/search-the-html-across-25-billion-websites-for-passive-reconnaissance-using-common-crawl-7fe109250b83 https://medium.com/@brevityinmotion/search-the-html-across-2...
RajT88 4y ago
Incidentally, conspiracy theorists have used this behavior as evidence of the "Dead Internet" theory. https://www.reddit.com/r/conspiracy/comments/xt8jzj/complete_proof_of_dead_internet_theory_you_can/ https://www.reddit.com/r/conspiracy/comments/xt8jzj/complete... For those who are behind the times on the "Dead Internet" theory: https://www.theatlantic.com/technology/archive/2021/08/dead-internet-theory-wrong-but-feels-true/619937/ https://www.theatlantic.com/technology/archive/2021/08/dead-...
bhartzer 4y ago
What I have an issue with is that Google's numbers change if you click on a few pages of results. Search for a keyword. Go to the bottom and click on the last page of results. Then keep clicking as deep as you can... notice that the number of results for that keyword will change. For one keyword search query, first it's 9,480, then 11,800 and then 105. Changes if you click on the next page of results (page 2, 3, 4, ... 10, etc.)
superkuh 4y ago
Google will never return more than 400 search results. You may think you are getting more because you use the default 10 per page and so that's ~38 pages of results. But if you really look it will never be more than 400. This pissed me off so much I actually registered and made http://googlesearchonlyreturns400results.lol/ http://googlesearchonlyreturns400results.lol/ And as I say on the page, the only way to approach this is to send feedback to google. We have to absolutely spam them with it so they know this is not acceptable. Bing, btw, only ever returns 900 results.
Barrin92 4y ago
Of course they aren't because it makes zero technical, economic or product sense to literally serve hundreds of thousands of pages to people. Everyone understands the number to mean something like the rough popularity of the query. I don't think I've ever seen an angry protester in front of the Googlplex complaining they couldn't actually get to page 250k undertaking some sort of Jules Verne-esque journey to the center of the earth. The post is basically a one year old ad for an alternative search product
jaimex2 4y ago
Take note anyone looking to make any kind of search. Just do Math.rand on the number of total results.
kderbyma 4y ago
Google is garbage. Bing is a bitch. DuckDuckGo doesn't do great, Yandex is yawn. search sucks.
jasfi 4y ago
Perhaps it's meant to limit what people can scrape from search results? A human user shouldn't need to see every result.
nonameking2026 4y ago
Very interesting - tell us more please.
Khelavaster 4y ago
The FCC needs to step in...

265 points by vincent_s 4y ago

TL;DR: "You know that Google problem everyone knows about, how it reports millions or even billions of 'results', but actually serves a couple hundred at most? Well, shocker of shockers, it turns out that the API has the same problem!" Google really needs to stop using the word 'result' for whatever it supposedly denotes, which is probably something like the estimated number of pages crawled by Google which contain hits for those terms, the majority of which will not be a search 'result' (something delivered by the search function) under any circumstances.

pilgrimfff 4y ago

I was expecting the number of search results here to be much higher - like who cares if Google only serves the first million results out of a billion? Very interesting to see that Google will only serve a few hundred links when they claim to have hundreds of thousands of relevant results indexed. I'm very curious where Google is getting that count and why the reality is so different. Systematic overcounting? Suppressing hundreds of thousands of results?

kevincox 4y ago
I used to work at Google but not in search, these are just my own guesses. > where Google is getting that count This is very likely a fairly accurate of the number of pages in Google's index that "match" the search query. Basically exactly what you would expect when you see the number. > why the reality is so different Cost reasons. Most search engines are more or less scanning down a sorted list of pages. The further you need to scan the more expensive it is. Just like running "OFFSET 1000" is usually slow in SQL. At some point the quality of results is generally very low and the cost is growing so it makes sense overall for Google to just cut it off to prevent it becoming an abuse vector (imagine just asking Google for the 10 millionth page of results for "cat"). The fact that few people realize that Google has a page limit shows how rarely people actually want more pages.
- yamtaddle 4y ago
  > The fact that few people realize that Google has a page limit shows how rarely people actually want more pages. I used to (years and years ago) go past the first page pretty often, but results are so bad now that it rarely helps, so I almost never even click "2", let alone later pages. It's all gonna be obviously-irrelevant crap google "helpfully" found for me or the auto-generated spam that google used to try to fight (circa 2008 and earlier) but no longer seems to, just letting it gunk up and dominate up any results you get that aren't from a handful of top sites. So this is in part one of those "we broke a thing and now no-one uses it, guess they didn't want it!"
  - kevincox 4y ago
    The thing has always been "broken". Google has had a page limit for at least a decade.
    - yamtaddle 4y ago
      No, by "broken" I mean "let lazy auto-generated spam take over the results almost completely". So now those of us who did used to browse past page one (which, to be fair, may not have been many people) don't bother anymore. [EDIT] For those who weren't around for it, Google used to play cat-n-mouse with spam-site operators. It'd go through cycles where results would slowly get worse, then suddenly a ton better, though never as bad as they are today. Around '08 or '09 they (evidently, I'm just judging from the search engine's behavior starting around then and continuing to this day) seemed to give up and just boosted a relatively small set of sites way up the results, abandoning the rest to the spammers.
      - Melatonic 4y ago
        Kagi has been working very well for me as an alternative
      - pbadams 4y ago
        Part of the difficulty is, if very few people are browsing to page 2, deciding what to put on page 2 becomes harder and harder. Google has a lot of user behavior signals to decide what should be in results 1-10. Deciding if a page should be ranked 20, 200, or 2000 without any user clicks to check if you're right is really difficult. I would bet that since 2008/9, the relative numbers of spam site operators, Google engineers, second-page searches have changed significantly.
yazaddaruvala 4y ago
The problem is generally called "deep pagination". It's extremely inefficient to compute. Specifically, counting requires very low memory. When data is spread across 10,000 computers, all of them counting returns just 10,000 numbers i.e. 4 bytes * 10,000 = 40KB. It's easy for 1 computer to count those 10,000. Even at 100,000 computers 400KB. Merging sorted search results is extremely memory intensive. Even with just the Id+Score pair, let's say 8 bytes. To get the 10,000th search result, each computer needs to create a List of 10,000 results, thats 10,000 * 10,000 * 8 bytes = 800 MB. For the 100,000th search result 10,000 * 100,000 * 8 bytes = 8 GB. OR if your data grows to 100,000 computers, thats 100,000 * 100,000 * 8 bytes = 80 GB of intermediate results to process at the end. As you can see this doesn't scale well. You're required to retain context (i.e. sessions) of the search in memory instead, and get the search engine to better coordinate across all 100,000 computers. This also has scaling limitations based on memory of the session, the number of computers, the number of sessions, and their TTL (someone can leave the search page open for day and hit "next page" - should the sessions still be open? Thats an answer each search engine has to decide). The reality is, if a customer wants deep pagination, they are better suited to a full data dump (i.e. full table scan) or using an async search API, rather than a sync search API.
- generalizations 4y ago
  Well at that point, who really cares if the content of the 1001s page is deterministic, or in perfect order? Get the first 100 or so pages right, and thereafter just request the nth results from each of those m computers. No merge and no memory explosion, you'll just get them slightly out of order.
  - kevincox 4y ago
    You still need to filter based on the other indexes. If you search for [bitcoin mining] you don't want to find pages related to coal mining. So this data still needs to be joined.
    - dekhn 4y ago
      the search term for this is intersection. The posting lists for the two terms are intersected, then the results are ranked. But there are a lot more steps in a production search engine. The long and short of it is if you really want the full results, just join google, join the search team, and then get enough experience so that you can do full queries over the docjoins directly. This was part of Norvig's pitch to attract researchers a while ago. For a research project, I built a regular expression that matched DNA sequences and spat out the list of all pages containing what looked like DNA and then annotated the pages so in principle you could have done dna:<whatever sequence> but obviously that was not a goal for the search team.

kweingar 4y ago

I was curious, so I tried to do this with GitHub search. The search “Golang” returns 225k+ repositories. Each page of results has 10 repositories. I queried page 1000 to see the 10001st-10010th results and got a 404. I searched Wikipedia (using its own search page) for “America” and was told there were 1.9M results. I requested the 10001st result and it failed with this message: “An error has occurred while searching: Could not retrieve results. Up to 10000 search results are supported, but results starting at 10000 were requested.” There was an HN thread along these lines about web search somewhat recently, with people making confident accusations about false advertising etc. But it really seems that everyone does this. I don’t know, maybe it’s still wrong or misleading. But personally I’m ok knowing “there are X hits and we will serve you some of them”. EDIT: I just searched the Library of Congress for “god” and was told I was viewing results 1-25 of 324,782. Sure enough when I asked for page 10,000 I was rebuffed. I really don’t think this is a Google thing

Retric 4y ago
Library of congress is quite explicit about this on their first search page, rather than showing an irrelevant number. So, having limits isn’t unusual, but not everyone is deceptive about it. You Searched: ALL: God Your search retrieved more records than can be displayed. Only the first 10,000 will be shown. Titles List: 1-25 of 10000 https://catalog.loc.gov/vwebv/search?searchArg=God&searchCode=GKEY%5E*&searchType=0&recCount=25 https://catalog.loc.gov/vwebv/search?searchArg=God&searchCod...
- kweingar 4y ago
  I guess they have multiple search interfaces. I navigated to loc.gov and searched directly from the home page. I was sent to this page: https://www.loc.gov/search/?in=&q=god&new=true&st= https://www.loc.gov/search/?in=&q=god&new=true&st= At the top, it says “Results 1-25 of 324,782” and does not mention the limit of 10,000 anywhere.
  - Retric 4y ago
    Ahh, looks like the limit on results is 100,000 items on that page. https://www.loc.gov/search/?q=god&sp=4000 https://www.loc.gov/search/?q=god&sp=4000 But you can subdivide the search results by date and get every item on your original list: https://www.loc.gov/search/?dates=1890/1899&q=god&sp=1808 https://www.loc.gov/search/?dates=1890/1899&q=god&sp=1808 So, they have and can show you every single one of those 324,782.

ravenstine 4y ago

I don't know where that quote is, but I remember reading one from someone at The Google who, in response to a question about DuckDuckGo, said something along the lines of "We're not competing on search." Turns out what they were saying was pretty truthful, assuming I'm not misremembering. Their "search", last I looked, shoves a bunch of crap above the fold that aren't exactly the results per se. The actual results usually aren't plentiful like I remember them once being. YouTube search has become astonishingly worse to such an extent that I get maybe 3 relevant results and the rest being videos I've already watched or stuff I'm clearly not interested in.

pessimizer 4y ago

Has anybody noticed that a Duckduckgo search (at least with infinite scroll) will serve you the same first page of search results over and over again? I've been trying to figure out whether that's a bug or if they're really fluffing one page into the appearance of multiple pages. I think the internet is closing up.

42365767567 4y ago
Same with Google Images. If you keep hitting more and more results, it'll just recycle the top ones.
- pessimizer 4y ago
  How can it be anything but intentional to give you the same image over and over again in an image search? That's not something you could possibly miss when checking for the quality of results.
  - propogandist 4y ago
    another search means another opportunity to serve you more ads for different keywords
turminal 4y ago
Maybe they used a char to store the array index and you hit an integer overflow?
omega3 4y ago
I've noticed plenty of websites now do this.
Liquix 4y ago
It seems DDG has recently started ignoring entire keywords in a 3 or 4 word search phrase without informing the user a keyword has been excluded. Something like "2015 camry leaking exhaust" could return nothing but "2015 camry exhaust" results until "leaking" is put in quotes (not a working example). Google search ignoring keywords was the straw that broke the camel's back - is DDG following suit?

ValleZ 4y ago

It shows not the number of shown results which is pretty much same for any query, but number of all pages that contain the keywords. It is not really feasible to rank and show millions of pages, but is possible to do that for few hundred pages. This is why search engines work fast.

Theodores 4y ago
It was once possible to trawl through results pages 'near indefinitely'. How I see it is that it is like how the music search works with Google Assistant. It can do it for 90% of the songs you are likely to hear without an online connection. For anything else it can do the search on the server. When you search something it is not as if a billion Google servers are consulted with the results delivered accordingly. The nearest box just knows 90% of the 'answers' and you get whatever that is, rather than a 'real search'. If you enter in a specific code, e.g. a ISBN number, then that is like the 'unusual song' that needs actual looking up. However it works, it is a good hustle. Actually nearer to 'the hitch hikers guide to the galaxy' in having a compiled book of answers.

gkoberger 4y ago

I actually like this feature. I don't want to see the 1,000th results (who would?), but it does help with perspective. If I'm searching for something that should be common, having an order of magnitude helps me realize I'm going in the wrong direction.

crazygringo 4y ago

I can't even tell what this blog post is complaining about, it seems so badly organized and written. But it seems obvious that if "coffee" gives 2 billion results that, no, you're not going to be able to browse to page 187,398,384 to get those results. There's no use case for that for any normal consumer (as opposed to competitor, researcher, etc.). If you're capped at browsing the first 10 pages or whatever, that's entirely reasonable. > A misconception regarding Google’s search results is that all of the results are being served to the user conducting that particular search. That misconception lies only with the author. Nobody's being "served" 2 billion results, I don't even know what that would mean. The number of results being reported is quite obviously in order to allow users to judge the breadth of search queries. If it says 2 billion, you might want to refine. If it says 15 and they're all useless, go broader. (It's useful for researching item popularity too, although that's been superseded by Google Trends which is built specifically for that.)

Grustaf 4y ago
I agree that it's very poorly written. The heading is especially confusing, it took me a long time to understand that "SerpAPI" was the search term, not a Google service. It is also the name of the company he's promoting, which makes it even more confusing. But I don't agree that it's "obvious" that "you're not going to be able to browse to page 187,398,384 to get those results. I would argue that a search result is only a search result if you can actually view it. If not, it's just a marketing, or statistics.
admax88qqq 4y ago
Feels like SEO blogspam trying to capitalize on Google's currently negative public profile after the Stadia cancellation, and general negative google sentiment. Really easy to get clicks (and upvotes on HN apparently) by complaining about literally anything Google related.
- adamrezich 4y ago
  why do you think this? nothing in the article indicates this, and Google's questionable search results (compared to what they once were) has been a nearly constant topic of discussion for years now

the_third_wave 4y ago

Try searching for "january 6" on bing for some insight in how results are both manipulated as well as censored: https://www.bing.com/search?q=january+6 https://www.bing.com/search?q=january+6 Once you get past page 6 of the 96 200 000 results they keep on repeating the same results, page after page, with either a Guardian (January 6 committee postpones Wednesday hearing over …) or (every now and then) Yahoo (Jan. 6 hearings to resume following bombshell revelation about …) article on top of the page. The rest of the page is largely identical, page upon page until it comes to page 32 (307-316 of 96 200 000 results). Whatever you tell it to do beyond that page it will always serve page 32 with that Guardian article on top and the other - similarly slanted - results below it. I don't know whether this is just another example of typical Microsoft incompetence in that they make their meddling with the results so incredibly obvious or whether they're just telling visitors this is what they should read and nothing else but it does show these search engines are as unreliable when it comes to politically sensitive topics as e.g. Wikipedia is.

Melatonic 4y ago
That is not how it is behaving at all for me
- the_third_wave 4y ago
  What do you see for that query? Are you logged in? Which country? Which browser? Here: Not logged in, Sweden, Firefox on Linux/Firefox on Android. Same results every time, on both platforms. I took screenshots for pages 6-15, compare these to what you see and let us know in what way your results are different: https://imgur.com/a/QGT044T https://imgur.com/a/QGT044T
pessimizer 4y ago
DDG draws at least the vast majority of its results from Bing, and does the same thing. Also, all of their news results seem to be links to MSN and Yahoo versions of stories that were published on other websites. I don't know that what you're seeing has anything to do with politics. For example, I just searched for "dog food nutrition," (https://duckduckgo.com/?q=dog+food+nutrition https://duckduckgo.com/?q=dog+food+nutrition) and https://www.akc.org/expert-advice/nutrition/soy-in-dog-food-what-you-need-to-know/ https://www.akc.org/expert-advice/nutrition/soy-in-dog-food-... Was the 14th result, and also the 36th, 68th, and 87th. It is also the 108th result, where it first starts topping the page, and from then on it's at the top of the rest of the pages, which are repetitions, so 128th, 148th, 168th, unto infinity. I just counted this result because it was topping the repeated page of results, but all of the other results were repeating arbitrarily until it was down to the same 20 results repeating. I'm tempted to sum the number of appearances of every result. I'm not sure there are more than 40. Why "dog food nutrition?" I figured there would be a lot of hits. I was wrong. ----- edit: https://www.bing.com/search?q=dog+food+nutrition https://www.bing.com/search?q=dog+food+nutrition https://www.akc.org/expert-advice/nutrition/soy-in-dog-food-what-you-need-to-know/ https://www.akc.org/expert-advice/nutrition/soy-in-dog-food-... 22nd result, 31st, 45th, 59th, 73rd, and plenty from there on out. https://www.petmd.com/dog/nutrition/can-dogs-eat-peaches https://www.petmd.com/dog/nutrition/can-dogs-eat-peaches 10th result, 20th, 34th, 48th, 51st, 66th, 70th, 88th, 93rd... ... and the pages start repeating from page 9 on with our old friend "Soy in Dog Food?" on top.

tethys 4y ago

Related comment from someone from Google from not too long ago: > It's because the counts are very fast, rough estimates. And when you go into additional pages, we start to refine them. https://news.ycombinator.com/item?id=32354785 https://news.ycombinator.com/item?id=32354785

verisimi 4y ago

Isn't it great that google has got rid of so many bugs? After all, more than 1 result is a bug: https://www.youtube.com/watch?v=XeIIpLqsOe4 https://www.youtube.com/watch?v=XeIIpLqsOe4

pessimizer 4y ago

With: https://www.bing.com/search?q=dog+food+nutrition https://www.bing.com/search?q=dog+food+nutrition ----- https://www.akc.org/expert-advice/nutrition/soy-in-dog-food-what-you-need-to-know/ https://www.akc.org/expert-advice/nutrition/soy-in-dog-food-... is the 22nd result, 31st, 45th, 59th, 73rd, and plenty from there on out. ----- https://www.petmd.com/dog/nutrition/can-dogs-eat-peaches https://www.petmd.com/dog/nutrition/can-dogs-eat-peaches is the 10th result, 20th, 34th, 48th, 51st, 66th, 70th, 88th, 93rd... ... and the pages start repeating from page 9 on with the previous "Soy in Dog Food?" on top. ----- Have the search engines given up on search? Also, can they really say they're being gamed if they're serving the same results up multiple times? Seems more like they're picking winners. This search is pretending like <40 results is hundreds of results, and someone had to make the effort to make sure that the same link wouldn't show up in the same page of results. That strikes me as a deliberate falsification. edit: As far as I can tell from https://www.google.com/search?q=dog+food+nutrition https://www.google.com/search?q=dog+food+nutrition it seems like google are doing a hell of a lot better. Google is also banned from setting cookies or using localstorage on my machine, so the result isn't search bubbled. Nothing about soy or peaches, and no repeats are jumping out at me even 100 results in.

jeffbee 4y ago

Imagine believing there's a sensible discrimination between the 200000th result and the 200001st result. The relative ranking among the top results is supportable but below the top 10 and especially the top 100 the relative ranking has no signal.

bitcurious 4y ago
I might not be able to discern between the 1000 and 1001 but there’s no reason I can’t discern between 1001 and 100001. Things can be fuzzy nearby while still being coherent overall.
- saalweachter 4y ago
  I used to believe that, until I played https://semantle.com/ https://semantle.com/.

lousken 4y ago

yea, google has been terrible at this for a while now, the most annoying thing is that you can't get to those results, i haven't found a way which is quite frustrating if you can't find what you're searching for in those few results

evancox100 4y ago

It makes sense if you separate the indexing process from the result ranking/retrieval process. Building up the index it has encountered the word “dog” millions (billions?) of times, but the Google search API’s job is NOT to operate as a catalogue that will list every single web page on the internet with the word “dog”. Instead its job is to show you what it thinks are the most relevant dog results. That said I think this could be communicated more clearly when you get to the end of the results. The fact that the results number _changes_ is an own goal that makes it seem like they have something to hide. Instead it should just say “Google Search shows a selection of the most relevant occurrences of your search term. If you didn’t find what you need, try XYZ.”

dredmorbius 4y ago

Google is hardly the only search engine which exhibits this behaviour, and the information of total possible matches is useful --- it indicates a grossly generic search phrase --- even where all results are not presented. As I noted recently in a similar thread (<https://news.ycombinator.com/item?id=32923468 https://news.ycombinator.com/item?id=32923468>, HN's search through Algolia is similar. An unqualified search reports (at this writing) somewhat north of 30 million results (30,121,402). It will display only 34 pages' worth of results, I believe 1,000 in total. See: <https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...> Final page: <https://hn.algolia.com/?dateRange=all&page=33&prefix=false&query=&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=33&prefix=false&q...> As discussed recently on HN, the match count is useful information even where all matching results are not displayed as it indicates whether or not a query is generic. Contrast search: <https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=Squeamish%20Ossifrage&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...> (one result as of this writing). I can understand some confusion on this point, but implying some sort of gross fraud or deception suggests far more wrong with the complainant, not their subject.

spullara 4y ago

These posts are just stupid. Search is for finding what you are looking for. I feel like most of the posts in here think that it should be for browsing content with the word in it. There is nothing to see here.

sleepymoose 4y ago
>Search is for finding what you are looking for. And at that, it performs miserably.
kuschku 4y ago
I want an index, like in a library, with the ability to filter on it. I want to know all content, add many different filters, sort by different properties, and get all results. Just like I'd do with an auto sort/filter table in excel
- spullara 4y ago
  Oh, then you want to use this instead of a search engine: https://medium.com/@brevityinmotion/search-the-html-across-25-billion-websites-for-passive-reconnaissance-using-common-crawl-7fe109250b83 https://medium.com/@brevityinmotion/search-the-html-across-2...

RajT88 4y ago

Incidentally, conspiracy theorists have used this behavior as evidence of the "Dead Internet" theory. https://www.reddit.com/r/conspiracy/comments/xt8jzj/complete_proof_of_dead_internet_theory_you_can/ https://www.reddit.com/r/conspiracy/comments/xt8jzj/complete... For those who are behind the times on the "Dead Internet" theory: https://www.theatlantic.com/technology/archive/2021/08/dead-internet-theory-wrong-but-feels-true/619937/ https://www.theatlantic.com/technology/archive/2021/08/dead-...

bhartzer 4y ago

What I have an issue with is that Google's numbers change if you click on a few pages of results. Search for a keyword. Go to the bottom and click on the last page of results. Then keep clicking as deep as you can... notice that the number of results for that keyword will change. For one keyword search query, first it's 9,480, then 11,800 and then 105. Changes if you click on the next page of results (page 2, 3, 4, ... 10, etc.)

superkuh 4y ago

Google will never return more than 400 search results. You may think you are getting more because you use the default 10 per page and so that's ~38 pages of results. But if you really look it will never be more than 400. This pissed me off so much I actually registered and made http://googlesearchonlyreturns400results.lol/ http://googlesearchonlyreturns400results.lol/ And as I say on the page, the only way to approach this is to send feedback to google. We have to absolutely spam them with it so they know this is not acceptable. Bing, btw, only ever returns 900 results.

Barrin92 4y ago

Of course they aren't because it makes zero technical, economic or product sense to literally serve hundreds of thousands of pages to people. Everyone understands the number to mean something like the rough popularity of the query. I don't think I've ever seen an angry protester in front of the Googlplex complaining they couldn't actually get to page 250k undertaking some sort of Jules Verne-esque journey to the center of the earth. The post is basically a one year old ad for an alternative search product

jaimex2 4y ago

Take note anyone looking to make any kind of search. Just do Math.rand on the number of total results.

kderbyma 4y ago

Google is garbage. Bing is a bitch. DuckDuckGo doesn't do great, Yandex is yawn. search sucks.

jasfi 4y ago

Perhaps it's meant to limit what people can scrape from search results? A human user shouldn't need to see every result.

nonameking2026 4y ago

Very interesting - tell us more please.

Khelavaster 4y ago

The FCC needs to step in...