11 ms·
On the other side, I've also noticed it appears to be aggressively pruning its index in the past few years, so the fact that it's crawled your site doesn't mean
by userbinator 3y ago
On the other side, I've also noticed it appears to be aggressively pruning its index in the past few years, so the fact that it's crawled your site doesn't mean it's necessarily searchable either.
Another "bug" that seems to manifest quite often: if I search for a specific phrase or unique word on a page that I found in a SERP, so I know it's crawled that page, it often doesn't return that page either.
Add to that the automatic CAPTCHA-hellban you get if you use "site:" in anything more than a tiny amount (and the one you still get if you search "too much"), and I realise that there's increasingly huge amounts of information out there on sites that Google may have crawled before and knows about, but doesn't want to show me for some reason. I remember it used to be much easier to find information about obscure topics even if it meant wading through dozens of pages of SEO spam; now it's nearly impossible for anything but the most vapid of queries.
- nohuck13 3y ago"Add to that the automatic CAPTCHA-hellban you get if you use "site:" in anything more than a tiny amount" Source? This would be worrisome.
- josephcsible 3y agoFWIW, I've personally experienced exactly that happening too.
- pteraspidomorph 3y agoI personally have been surprised to find myself CAPTCHA'd out of google search recently. No idea what's up with that. Regular commercial ISP, no VPNs.
- tyingq 3y agoAnecdotal, but this happens to me a lot, and not just with the "site:" operator. Generally using any of the advanced operators seems to set it off. Things like inurl:, intitle:, etc, trigger it also. Not every time, but after a few times. From a normal ISP connection, no VPN, even while logged into Google, etc.
- Hard_Space 3y agoAnecdata, but I can confirm a uniform and long-standing experience that adding colon-based operators to a search query results in a CAPTCHA challenge every single time on a subsequent search, even if the subsequent search is 'vanilla' (i.e., no operators). Has been like this more years now than I can remember. Apparently this kind of 'advanced' usage is indication of bot activity.
- raverbashing 3y agoSo now anyone displaying slightly more intelligence than an eggplant while doing a search in Google is a "bot"? Appalling
- freedomben 3y agowelcome to the machine learning future, where anything you do that is a statistical outlier gets you algorithmed by a machine that is incapable of reason but knows when you're different. As a person who has been a statistical outlier most of my life, I am dreading this. It's bad enough dealing with human impressions and mis-judgment, but now we get it from our computers now, which used to be logical, deterministic havens.
- pixl97 3y ago>As a person who has been a statistical outlier most of my life Anomaly detected. Termination authorized.
- CatWChainsaw 3y agoAll humans must be reduced to sameness so machine successors can flourish.
- brazzledazzle 3y agoFor what it’s worth this never, ever happens to me. These days I only get captcha’d when someone’s laptop on the same network gets owned and is being used to hit google.
- 3y ago
- mavhc 3y agobots use it to find websites with security flaws I assume
- EvanAnderson 3y agoI've never gotten a CAPTCHA-hellban that I know if, but I absolutely get a CAPTCHA when I use "site:" for more than just a couple searches. (It sounds par-for-the-course w/ Google, though...)
- metalforever 3y agoYeah, I get these. The problem is that the Captchas take forever to fill out (like 5 minutes of challenges). But the worse part is that the captchas are asking for wrong answers. It tells you to select scooter and there's no scooter in the photo but it thinks there is. So you just end up stuck in a captcha loop for a long time. I am not sure why I get them but it might be due to using anti-fingerprinting tools.
- freedomben 3y agoI've wondered if it isn't intentionally impossible to solve, because "the algorithm" decided that you're a bot or malicious and they want to spin your cycles endlessly. The affect on me know is I won't even try anymore, I'll just take a different route. That may even reinforcement teach the system that I was a bot that couldn't solve it
- metalforever 3y agoI think it's more malicious than that. They know I use privacy tools and can't be tracked -> they can't make money on me -> bully me into not using their service.
- Atlas22 3y agoIt may also be part of their anticompetitive war on other browsers. I get captchas constantly in a new default Firefox profile, but not in a new default chrome profile. Spoofing user agent to recent chrome agent in Firefox makes the captchas happen far less often for me.
- privacyking 3y agoI sometimes get multiple captchas in a row that I fill correctly but they keep on showing more..I then just do the audio one which works.
- 20after4 3y agoThis is probably the common thread among all the people reporting this. As an alternate date point, I haven't experienced the captcha from using advanced search queries.
- baremetal 3y ago>and I realise that there's increasingly huge amounts of information out there on sites that Google may have crawled before and knows about, but doesn't want to show me for some reason. I wonder what that reason could be.
- mistermann 3y agoI've always wondered why we got rid of curated directories and changed to search for almost everything (and yes I do realize that volume of sites is problematic).
- pixl97 3y agoIt's expensive? Why provide the best product when you only have to have a product slightly better than your competition. After that everything is profit. Couple that with a huge portion of new sites seem to be bot generated shit that's copied from other places on the internet it seems Google has given up on the open web.
- moffkalast 3y agoNot sure why they'd care, they have effectively infinite money from adsense.
- marcosdumay 3y agoAs long as people don't abandon search, yeah, they do. If they lose their absolute dominance in search, they will automatically have competition on adsense too. Or maybe Google disagrees with my assessment, but I can't imagine what kind of inside information would make them do that. It looks like a very clear and inescapable reality to me.
- pixl97 3y agoYou should know that's not how capitalism works. They have to keep making more money per dollar every year or they get punished in the market. They've tapped out on their limits of growth and now actual costs are increasing due to floods of automated crap at levels far beyond what we had in the past.
- 77pt77 3y ago>CAPTCHA-hellban you get if you use "site:" in anything more than a tiny amount Please explain this point.
- kroltan 3y agoIf you use advanced search features say 10 times in 10 minutes or whatever (a reasonable amount when refining a search if you ask me), you're quite liable to be elected to have a trial of endurance against the "prove you are a human" feature, having to solve multiple (my record is 16) consecutive "select all images that contain BLAH" tests.
- shaky-carrousel 3y agoI had to solve thousands of captchas as part of the yahoo groups archiving project. You only have to choose four of the images, whatever the test is, and it's not really precise, so you can make small mistakes and it still will let you pass.
- pixl97 3y ago> having to solve multiple Do you use adblock? I find if adblock is enabled when doing captchas I have to keep clicking pictures over and over.
- Avamander 3y agoYou have to solve a bunch of captchas if your searches are obscure or frequent enough.
- isaacfrond 3y agoI recognise this as well. I write for a living. So I'll do lots of searches to cross check stuff. But if you search to quickly, or to 'weirdly', or whatever you'll have to pick out bridges or zebra's or whatever is the current fashion in Captcha.
- walthamstow 3y agoFWIW I have found audio captchas much less annoying and time-consuming. For Google's captchas, click the headphones symbol.
- sph 3y ago> On the other side, I've also noticed it appears to be aggressively pruning its index in the past few years, so the fact that it's crawled your site doesn't mean it's necessarily searchable either. I've noticed this as well. I have a crappy website for my app I need to do better marketing for (not my priority just now), but I've noticed that, for however crap it is, I have received ZERO incoming hits from Google, apart from a couple people that have literally just googled my domain name. I do not believe for a second there's not a single query done in the 2 months the page has been up, globally, for which my website wasn't a bit relevant. Either that, or the spam problem Google has is much bigger than anyone thinks. Yet another data point in favour of the Dead Internet theory.
- gilleain 3y agoYou could try the google search console - it gives you a view on what hits/clicks have come in over time. edit: Hah. I notice it suggests using it at the top of the page if you use 'site:..." - and I only get 5 results for my site when the console claims to have indexed 10 times that many! edit2: Also duckduckgo returns more like 15 hits ...
- soared 3y agoSilly to see people complaining about search results and indexing without backing those claims with data from search console. It’s like devs turn off their brains when it comes to marketing because they don’t like it.
- sph 3y agoGoogle's bloody Search Console says I got 16 impressions in 2 months for literal searches of my domain name, and nothing else. Funny seeing people thinking I got those figures by reading tea leaves. Who's the silly one now?
- rideontime 3y agoDead Internet theory?
- dontupvoteme 3y ago>Add to that the automatic CAPTCHA-hellban you get if you use "site:" in anything more than a tiny amount Is it more expensive, or do they just wish to prevent people from being able to cache their own results locally?
- bombcar 3y agoI suspect it bypasses some advertising metric and they don't like it.
- Gasp0de 3y agoI assume it is so that websites don't abuse it to build search boxes for their own sites without showing ads? E.g. I can build a searchbox on mywebsite.com, and if you type "hamster" I'll just query google for "site:mywebsite.com hamster" and return the results to you. That way, my site can be static but still have a search box, and google has all the work but gets no money.
- gniv 3y agoIt should actually be cheaper to run. Much cheaper, since the site operator acts as a restrict on URLs.
- privacyking 3y agoStartpage does the same thing when I use sitebut with no captcha to bypass the hellban. Sometimes it just shows no results intentionally. Refreshing the page fixes that one
- topicseed 3y agoVery true. Some client websites have had half their keywords gone from position 1-3 to deindexed, then back, then gone, then back, and that's been since February 2023.
- aaron695 3y ago[dead]
- burnte 3y agoAnother bug I'm noticing lately is it'll flat out ignore things sometimes, even if you put a term in quotes or try to exclude it with a -leadingdash. About 30% of the time if I use those operators, they'll have no effect on the results. I don't understand why they'd make things worse on purpose, but I don't know how it could be just a "mistake" no one noticed.
- cmeacham98 3y agoI notice this happening when the actual query would have returned 0 results, Google ""helpfully"" will modify your query (such as dropping quotes) to generate more results. This is super annoying because it doesn't appear to inform you of this anywhere in the UI, until you click through to page 2 and see what it modified your query to be.
- winternett 3y agoSites like Twitter and Instagram also frequently completely change the search term now to something else for certain queries. This practice is anti-competitive in the highest order. The very foundation of having a text search is to have an exact query match to begin with... The alternate spelling item should only be a suggestion in results at the most, but they've flipped this now, and that's outright deceptive.
- QuantumGood 3y agoOver the years, I've found the frequency of "0 result" queries has gone way up. Subjective anecdata from me, but it's a pretty big difference. There must be some large areas of their index that have been dropped over time. From what Google's hinted at and probably your own experiences (which I reckon are like mine), it's pretty clear that most folks aren't great at Google searches. This might be why Google has leaned on AI to "guess" the best results. They figure their AI can predict what you want better than you are able to specify via your search query.
- morkalork 3y agoAlso 1 page results. Around 10-20 links. Like, that's it? Really? No more? That's not what I see when I try the same query on Bing!
- roncesvalles 3y agoAnother thing I've noticed: Google only indexes what people search. Meaning, sometimes if you search for something obscure and you don't get good results, come back a week later and you'll get much better results because your query is now a part of their indexed search terms.
- masswerk 3y agoThis, I have noticed some years ago. It seems much like, if the number of returned results doesn't meet a given threshold, some kind of optimizer runs over night on these searches in order to provide a more extensive result set.
- kccqzy 3y agoSuper interesting discovery! I wonder if whatever algorithm Google is using has reached its scalability limit on today's Internet, and it takes some kind of an over-night batch job to do obscure searches usefully. Maybe all Google Search is doing is just a giant cache of slow search results.
- masswerk 3y agoThis has been some years ago. Notably, I observed this in relation to search suggestions. You could enter a search and get zero results, but a day later or two, you'd get at least a suggested search term (regardless how accurate or meaningless this may have been). So I guessed, these were built up, at least partly, retroactively. With results now happily including these "sympathetically adjusted search terms" without presenting this as an explicit option, I'd guess, this may now apply automatically.
- deleted 3y ago[deleted]
- dstroot 3y agoSERP: Search engine results page. I asked ChatGPT. "SERP stands for Search Engine Results Page. It refers to the page displayed by a search engine in response to a user's query. When a user enters a search term or keyword, the search engine generates a list of relevant web pages and presents them in the form of a SERP. The SERP typically includes a combination of organic search results, which are the regular listings based on relevance to the query, and paid advertisements, which are sponsored listings that advertisers pay for to appear prominently on the page. SERPs often contain additional elements such as featured snippets, knowledge graphs, image or video results, local map results, and other specialized features, depending on the specific search engine and query."
- donkeybeer 3y agoWhat I fucking hate is writing a query, sometimes even with parts in double quotes to clarify, and google "helpfully" correcting it to something unwanted, and then putting up the damn captcha when I click the link to search exactly what I want.
- bhartzer 3y agoJust fyi, the database that is used for the site:domain.com is actually not the same database that they use for live searches. So you may see a certain number of pages using the site: command but not or less pages may be indexed. If you want pages indexed, out then in an xml sitemap file, make sure there are internal links to them on your site, and external links from other sites really helps. Third party indexer tools help as well.
- Eumenes 3y agoAlso anything past the first page, will just show you crap on the first page. I use to be a power user of operators like 'site:' but agreed, it results in a captcha every other page sometimes.
- jeffbee 3y ago> so I know it's crawled that page, it often doesn't return that page either. This is your misunderstanding. The fact that a thing was in the index does not ensure it will always be there. Things disappear from the web all the time. Serving fresh docs means not only crawling the new stuff but also deleting the unreachable stuff promptly.
- userbinator 3y agoNo, I literally did a site: search seconds after visiting the site to see if it had any other pages with what I was looking for, and it found zero results --- not even the original page I found.
- jeffbee 3y agoAnd you simply refuse to believe that online index updates exist?
- 0xcde4c3db 3y ago> Add to that the automatic CAPTCHA-hellban you get if you use "site:" in anything more than a tiny amount Pretty much any advanced operators seem to do it for me, notably "intitle:" and "inurl:". I'd wager that there are a lot of automated searches using these to look for exposed admin interfaces, but I find them extremely useful for filtering out the crap that clogs up results when a ton of news sites all regurgitate the same viral press release or wire article.
- deleted 3y ago[deleted]
- narrator 3y agoThey are probably trying to reduce "misinformation" by removing most of the web from their index. With AI, they could just ask bard, "does this website contain any information that would be considered misinformation?" and then just ban it. If you want "misinformation," or to just search the web like it's the mid 2000s, you can use http://Yandex.com http://Yandex.com. They do a pretty good job on controversial queries. Google has gotten so political that they even have this "results are changing rapidly" page they return when there's been some new political hot topic that they haven't gotten the commissars at headquarters to weigh in on yet as to what's going to be the official narrative.[1] [1]https://www.theverge.com/2021/6/25/22550430/google-search-results-changing-quickly-warning-breaking-news https://www.theverge.com/2021/6/25/22550430/google-search-re...
- Atlas22 3y agoNice try FBI. In all seriousness though, has it actually gotten so bad that yandex of all search engines is less censored? Or is it just less censored when it comes to topics controversial to the US (and not russia)? The fact that so much censoring is going on that google has a "hold on while we censor this" page is insane.
- philsnow 3y ago> google has a "hold on while we censor this" page a what now?
- account42 3y agoYandex is also censored but in the other direction. Probably does censor less than Google but enough that you shouldn't rely on it alone for topics that involve Russia. Their index is also limited in general when it comes to non-russian content. But it does return many things that Google would rather you not see so it is invaluable if you want to get the full picture. Yandex image search also has an infinitely better interface, linking your directly to image sources and not being full of links to sites that want you to sign up before showing anything like Google is. It's still not perfect and IME often groups images to agressively which effectively hides "similar" results.
- ren_engineer 3y ago> there's increasingly huge amounts of information out there on sites that Google may have crawled before and knows about, but doesn't want to show me for some reason this is some machine learning stuff they are doing, instead of indexing all the specific keywords they are creating vector embeddings and basically summarizing what's on the page and going on similarity to your query rather than specific keywords. Good for casual searches, but extremely annoying for power users
- jxramos 3y agoThis is one of the reasons I make notes off websites that piqued my interest in something. It’s just too hard to re-search from scratch.
- fasterik 3y agoGoogle results have become so bad that I use "site:" for a majority of my searches these days. I have a bunch of Chrome search engine keywords set up so that I can go straight to results on Wikipedia, Economist, Reddit, Stack Overflow, Cppreference, etc. It's concerning that they're even nerfing site search, which seems like a core feature for a search engine. You could argue that Google isn't really a search engine any more, but rather a general knowledge engine and advertising platform. I hope somebody can build an alternative to Google that does what a search engine is supposed to do, i.e. index the web without all the extra garbage. But maybe SEO has killed that dream at this point.
- jamesfinlayson 3y agoI'd never noticed any issues with Google until a few months ago where I was googling an exact phrase that I knew appeared on one site. Google gave me nothing but DuckDuckGo found it. The site is probably 20 years old and has no SSL, but still... giving me no results is worse then giving me the one correct result.
- oilchange 3y ago> On the other side, I've also noticed it appears to be aggressively pruning its index in the past few years In terms of breadth and depth, the quality of google search has declined noticeably. They don't have any real competitors in search so they can do whatever they want. > now it's nearly impossible for anything but the most vapid of queries. Rather than getting us want we want, they want to gives us what they want. A narrow band of approved results. Youtube is like this as well, but then again, youtube and google are both part of Alphabet. It's like google news was a test run and they slowly exported it to search, youtube, etc.
- fomine3 3y agoI noticed it in January 2018. I thought that it's a temporary degradation so it will be fixed soon. It have never been fixed ;/
- GolfPopper 3y ago>now it's nearly impossible for anything but the most vapid of queries I've noticed that myself, looking for very precise content, which I know is out there but failed to bookmark. (Most recently, for amateur astronomy and roleplaying.) The solution to finding niche stuff now seems to be digging through relevant reddit or forum threads, hoping someone posted a link to it.