4 ms·

I used to work at Google but not in search, these are just my own guesses. > where Google is getting that count This is very likely a fairly accurate of the n

I used to work at Google but not in search, these are just my own guesses. > where Google is getting that count This is very likely a fairly accurate of the number of pages in Google's index that "match" the search query. Basically exactly what you would expect when you see the number. > why the reality is so different Cost reasons. Most search engines are more or less scanning down a sorted list of pages. The further you need to scan the more expensive it is. Just like running "OFFSET 1000" is usually slow in SQL. At some point the quality of results is generally very low and the cost is growing so it makes sense overall for Google to just cut it off to prevent it becoming an abuse vector (imagine just asking Google for the 10 millionth page of results for "cat"). The fact that few people realize that Google has a page limit shows how rarely people actually want more pages.

yamtaddle 4y ago
> The fact that few people realize that Google has a page limit shows how rarely people actually want more pages. I used to (years and years ago) go past the first page pretty often, but results are so bad now that it rarely helps, so I almost never even click "2", let alone later pages. It's all gonna be obviously-irrelevant crap google "helpfully" found for me or the auto-generated spam that google used to try to fight (circa 2008 and earlier) but no longer seems to, just letting it gunk up and dominate up any results you get that aren't from a handful of top sites. So this is in part one of those "we broke a thing and now no-one uses it, guess they didn't want it!"
- kevincox 4y ago
  The thing has always been "broken". Google has had a page limit for at least a decade.
  - yamtaddle 4y ago
    No, by "broken" I mean "let lazy auto-generated spam take over the results almost completely". So now those of us who did used to browse past page one (which, to be fair, may not have been many people) don't bother anymore. [EDIT] For those who weren't around for it, Google used to play cat-n-mouse with spam-site operators. It'd go through cycles where results would slowly get worse, then suddenly a ton better, though never as bad as they are today. Around '08 or '09 they (evidently, I'm just judging from the search engine's behavior starting around then and continuing to this day) seemed to give up and just boosted a relatively small set of sites way up the results, abandoning the rest to the spammers.
    - Melatonic 4y ago
      Kagi has been working very well for me as an alternative
    - pbadams 4y ago
      Part of the difficulty is, if very few people are browsing to page 2, deciding what to put on page 2 becomes harder and harder. Google has a lot of user behavior signals to decide what should be in results 1-10. Deciding if a page should be ranked 20, 200, or 2000 without any user clicks to check if you're right is really difficult. I would bet that since 2008/9, the relative numbers of spam site operators, Google engineers, second-page searches have changed significantly.
- samatman 4y ago
  I find search results are frequently even worse than this, in that the first page will have nothing useful, with about three good links split between the second and third page. If I'm lucky.
- int_19h 4y ago
  What's really weird is that sometimes you get results that are outright repeating on those first N pages. Sometimes, more than once. It's almost as if it tries to pad the output to be long enough that you'd lose patience before you reach the end of "effective pagination".
bryanrasmussen 4y ago
>The fact that few people realize that Google has a page limit shows how rarely people actually want more pages. the fact is I just want the long tail or weird results to escape content farms, but I guess if it were possible for google to serve those content farms would spring up to game the long tail or weird results market.
- bmicraft 4y ago
  Google tries to ignore them already, so the long tail is probably littered with old and mitigated content farms because they "match" but have a low page rank
salawat 4y ago
If you've ever read Larry Niven's Fleet of Worlds series, there's a Bussard Ramjet with an AI programmed to hide any information that could help a hostile enemy/force find their way back to Earth. A small cadre of humans who were raised by an Alien Race who came across a human seed ship cross paths with this Ramjet, and one of the protagonists realizes something is off when they do a query on the size of presentable search results in the astrographic/navigational dataset, and realizes that the number of starmaps the AI will produce is far smaller than the amount of space the system actually dedicates to storing said maps. Point being, you can't trust any system that restricts results to a subset to not actually being designed to leave out results. and it furthermore makes a great, plausibly deniable way to drop search results... Force ranking to 10001+. You'll forgive me, I'm sure, if I question a company well known for cooperating with an anti-humanitarian regime (Project Dragonfly) and that regularly black holes other undesirable datapoints, of engaging in less than up front search result presentation, I hope?
- notriddle 4y ago
  This isn't the revelation you act like it is. Because of course Google hides results. They don't pretend not to, and they even inform webmasters when it happens. The Search Console calls it a "Manual action" when they do so. More importantly, the people asking for a "censorship-free search engine" are expressing an incoherent desire. The whole point of a search engine is to take the zillions of web pages that have matching keywords, push the crap to the bottom, and leave the gold on top. A system that does this is inherently censorious. We're just quibbling over what the criteria should be. What our world lacks is a reasonably-quick way to hold Google accountable when they fail to represent the interests of the public who searches with them. The real-world consenquences of their filtering decisions need to filter back to the people making these decisions. Because "just don't make any filtering decisions" isn't going to result in a usable information retrieval system.
  - kevincox 4y ago
    > "censorship-free search engine" are expressing an incoherent desire That's not really true. `grep` is a censorship-free search engine. It just reports every matching result. Of course that wouldn't generally be useful over the web, however even with sorting it is possible to be censorship free. You just need to include every matching result eventually. Of course you would find that generating later pages likely also becomes expensive, so you may also add a page limit and ask the user to refine the query instead. Of course then you are back to this problem of it can be very difficult to find every result because you need to guess what words are on the page. But all of this is basically moot because Google doesn't claim to be censorship-free so they have much simpler way of hiding results.

yamtaddle 4y ago

> The fact that few people realize that Google has a page limit shows how rarely people actually want more pages. I used to (years and years ago) go past the first page pretty often, but results are so bad now that it rarely helps, so I almost never even click "2", let alone later pages. It's all gonna be obviously-irrelevant crap google "helpfully" found for me or the auto-generated spam that google used to try to fight (circa 2008 and earlier) but no longer seems to, just letting it gunk up and dominate up any results you get that aren't from a handful of top sites. So this is in part one of those "we broke a thing and now no-one uses it, guess they didn't want it!"

kevincox 4y ago
The thing has always been "broken". Google has had a page limit for at least a decade.
- yamtaddle 4y ago
  No, by "broken" I mean "let lazy auto-generated spam take over the results almost completely". So now those of us who did used to browse past page one (which, to be fair, may not have been many people) don't bother anymore. [EDIT] For those who weren't around for it, Google used to play cat-n-mouse with spam-site operators. It'd go through cycles where results would slowly get worse, then suddenly a ton better, though never as bad as they are today. Around '08 or '09 they (evidently, I'm just judging from the search engine's behavior starting around then and continuing to this day) seemed to give up and just boosted a relatively small set of sites way up the results, abandoning the rest to the spammers.
  - Melatonic 4y ago
    Kagi has been working very well for me as an alternative
  - pbadams 4y ago
    Part of the difficulty is, if very few people are browsing to page 2, deciding what to put on page 2 becomes harder and harder. Google has a lot of user behavior signals to decide what should be in results 1-10. Deciding if a page should be ranked 20, 200, or 2000 without any user clicks to check if you're right is really difficult. I would bet that since 2008/9, the relative numbers of spam site operators, Google engineers, second-page searches have changed significantly.
samatman 4y ago
I find search results are frequently even worse than this, in that the first page will have nothing useful, with about three good links split between the second and third page. If I'm lucky.
int_19h 4y ago
What's really weird is that sometimes you get results that are outright repeating on those first N pages. Sometimes, more than once. It's almost as if it tries to pad the output to be long enough that you'd lose patience before you reach the end of "effective pagination".

bryanrasmussen 4y ago

>The fact that few people realize that Google has a page limit shows how rarely people actually want more pages. the fact is I just want the long tail or weird results to escape content farms, but I guess if it were possible for google to serve those content farms would spring up to game the long tail or weird results market.

bmicraft 4y ago
Google tries to ignore them already, so the long tail is probably littered with old and mitigated content farms because they "match" but have a low page rank

salawat 4y ago

If you've ever read Larry Niven's Fleet of Worlds series, there's a Bussard Ramjet with an AI programmed to hide any information that could help a hostile enemy/force find their way back to Earth. A small cadre of humans who were raised by an Alien Race who came across a human seed ship cross paths with this Ramjet, and one of the protagonists realizes something is off when they do a query on the size of presentable search results in the astrographic/navigational dataset, and realizes that the number of starmaps the AI will produce is far smaller than the amount of space the system actually dedicates to storing said maps. Point being, you can't trust any system that restricts results to a subset to not actually being designed to leave out results. and it furthermore makes a great, plausibly deniable way to drop search results... Force ranking to 10001+. You'll forgive me, I'm sure, if I question a company well known for cooperating with an anti-humanitarian regime (Project Dragonfly) and that regularly black holes other undesirable datapoints, of engaging in less than up front search result presentation, I hope?

notriddle 4y ago
This isn't the revelation you act like it is. Because of course Google hides results. They don't pretend not to, and they even inform webmasters when it happens. The Search Console calls it a "Manual action" when they do so. More importantly, the people asking for a "censorship-free search engine" are expressing an incoherent desire. The whole point of a search engine is to take the zillions of web pages that have matching keywords, push the crap to the bottom, and leave the gold on top. A system that does this is inherently censorious. We're just quibbling over what the criteria should be. What our world lacks is a reasonably-quick way to hold Google accountable when they fail to represent the interests of the public who searches with them. The real-world consenquences of their filtering decisions need to filter back to the people making these decisions. Because "just don't make any filtering decisions" isn't going to result in a usable information retrieval system.
- kevincox 4y ago
  > "censorship-free search engine" are expressing an incoherent desire That's not really true. `grep` is a censorship-free search engine. It just reports every matching result. Of course that wouldn't generally be useful over the web, however even with sorting it is possible to be censorship free. You just need to include every matching result eventually. Of course you would find that generating later pages likely also becomes expensive, so you may also add a page limit and ask the user to refine the query instead. Of course then you are back to this problem of it can be very difficult to find every result because you need to guess what words are on the page. But all of this is basically moot because Google doesn't claim to be censorship-free so they have much simpler way of hiding results.