5 ms·
Problems caused by Google crawlers
- dymk 8y agoSome of these mitigation tactics seem a little much, given the situation; did the author consider a robots.txt first?
- Jyaif 8y ago"making my web site less usable is the only way I can find to stop the Google from indexing infinite trees of it's own creation" So apparently not.
- dymk 8y agoAre you claiming that Google doesn't respect robots.txt? He can just disallow *.
- ancarda 8y agoPresumably you'd want Google to index pages, just not try to discover pages by brute forcing the URL? Is there any way to do that in robots.txt? Someone did mention adjusting the crawl rate in Webmaster Tools -- would the author need to do this for every search engine?
- dymk 8y agoHe can submit a sitemap to Google if he’d like a particular structure crawled. Sounds like his issue is mainly with Google, not other search engines, so it’s not an O(n) problem he’s got to solve.
- greglindahl 8y agoSubmitting a sitemap might cause Google to crawl those pages, but it certainly won't stop google from crawling additional pages like the ones mentioned in the blog post. Lots of sitemaps are incomplete or outdated or completely invalid or whatever.
- dymk 8y agoI feel like you're still misunderstanding just how simple the fix is. If he wants to, he can just disallow the portions of his site that he claims Google is scraping via a robots.txt, and he can also submit a sitemap to the portions that he does want crawled.
- greglindahl 8y agoOK, so let's look at the top example of unwanted crawling from the article: /blog/index.php?/categories/14-Forensics (wanted) /blog/index.php?/categories/14-Forensic (unwanted) /blog/index.php?/categories/14-Forensi (unwanted) /blog/index.php?/categories/14-Forens (unwanted) How do you propose blocking the last 3, without having an enormous robots.txt?
- pas 8y agoI think if you do a Disallow: * and also submit an explicit sitemap, then the sitemap would 'win' for those explicit URLs, no?
- _asummers 8y agoYou can return a `X-Robots-Tag: none` header for the inner URLs. It's a thing I throw into APIs I write, just in case Google manages to find it.
- greglindahl 8y agorobots.txt can certainly keep google away from your search box, but it won't defend you against the lower-case probes, the extra letter probes, the delete-a-character probes, etc.
- dwild 8y agoYet he didn't try it. That's seriously weird that he doesn't mentions trying that (he updated his post to say they doesn't support regex, yet does support wildcard, which is all what's needed for a search). He removed the search before using a simple robots.txt...
- userbinator 8y agoThe first wave of the attack began dropping off letters. I wouldn't call it an attack, but it seems more like some sort of autocompletion checking given how similar it is to the queries generated to Google if you use their search autocomplete feature --- one request for each character entered. That said, Google's search quality has already taken a steep decline and I've been getting blocked very often for searching more obscure things now, so whatever "bot mitigations" they put in, I hope they don't make that even worse.
- wybiral 8y ago> I've been getting blocked very often for searching more obscure things Me too! It seems like nearly any time I go past the second page for results they think I'm a bot.
- _asummers 8y agoThe claim was that it started with the full URL and progressively removed them. If it were reversed I would buy the autocomplete bug. I fixed something like that a few weeks ago at work.
- mynameisvlad 8y agoIf Chrome address bar autocomplete generates queries (which I'm not sure if it does), then the following scenario could easily explain what's going on: - As a user, I type "foo.com" into my address bar. Having already gone to a specific page of that site before, autocomplete "helpfully" expands it to that page's URL "https://foo.com/this/is/a/page" https://foo.com/this/is/a/page" - It turns out I don't actually want that, I want the main page, so I start deleting characters until I'm left with "https://foo.com" https://foo.com" If Google took each of those deletions as a separate URL (either by bug or on purpose) and tried to verify said URL is valid, I could see a GoogleBot request being generated for each iteration between the initial URL and the full URL, starting with that initial one.
- estebank 8y agoGiven what I've read on the article, I'm fairly confident that what's happening here is a conjunction of Chrome autocomplete suggestions causing the crawl of every URL permutation and the website returning the same content to every permutation, instead of using 403 redirects or including a `rel=canonical tag`.
- deleted 8y ago[deleted]
- ryanlol 8y agoThis guy has had so many issues with various crawlers "attacking" his (boring!) sites, I'm starting to think he might just be a bit paranoid. see: https://www.hackerfactor.com/blog/index.php?/archives/678-Bot-Spotting.html https://www.hackerfactor.com/blog/index.php?/archives/678-Bo... https://www.hackerfactor.com/blog/index.php?/archives/762-Attacked-Over-Tor.html https://www.hackerfactor.com/blog/index.php?/archives/762-At... https://www.hackerfactor.com/blog/index.php?/archives/775-Scans-and-Attacks.html https://www.hackerfactor.com/blog/index.php?/archives/775-Sc... https://www.hackerfactor.com/blog/index.php?/archives/777-Stopping-Tor-Attacks.html https://www.hackerfactor.com/blog/index.php?/archives/777-St... https://www.hackerfactor.com/blog/index.php?/archives/779-Behind-the-Tor-Attacks.html https://www.hackerfactor.com/blog/index.php?/archives/779-Be... I've done lots of portscanning of the entire internet, this always generates vast amounts of abuse emails. Many of these these emails are xenophobic rants from obviously distressed people staring at their access.logs, thinking they're being targeted by the Russian government or something. I can't help but wonder if this might be a similar case.
- hnabz 8y agoThe imgur thing looks like Google was following links from another webpages. i.e. I submit a pic to fotoforensics, I post the link here (something like fotoforensics.com/analyze?url=imgur.com/blabla) and Google ended up crawling all those links in batches. The fact that he says something like "a Google administrator tracked down a rogue employee and made him stop" makes it clear he's paranoid (or malicious).
- wybiral 8y ago> The fact that he says something like "a Google administrator tracked down a rogue employee and made him stop" makes it clear he's paranoid (or malicious). Or sensationalism. (it did land a top spot on HN)
- mintplant 8y agoIf you read the original post about that issue [0], the imgur.com URLs submitted were for imgur web pages containing an image, rather than direct image URLs. This caused the FotoForensics service to emit an error message. Users wouldn't have any reason to link to those error pages. [0] https://www.hackerfactor.com/blog/index.php?/archives/484-Google-Abuse.html https://www.hackerfactor.com/blog/index.php?/archives/484-Go...
- deleted 8y ago[deleted]
- partiallypro 8y agoYou can adjust the crawl rate in webmaster tools
- Kiro 8y ago> Unfortunately, I had to remove that feature because Google began submitting random dictionary words. Hundreds of thousands of them. That sounds insane. Is that actually true or is there an exaplanation of some sort?
- dlubarov 8y agoGooglebot does sometimes submit search forms in an attempt to discover new pages, though I'm surprised to hear "hundreds of thousands". Either that's an exaggeration or it was over a very long time period. As other comments mentioned, he could have easily prevented this with robots.txt: User-agent: * Disallow: /search
- mynameisvlad 8y agoWould that work if the search box was on the main page, though? Presumably, GoogleBot wouldn't index the results, but is it smart enough to know that the form redirects to the disallowed page?
- greglindahl 8y agoYes, Googlebot (and everyone else, really) is smart enough to figure out if a form action is disallowed in robots.txt.
- T3RMINATED 8y agoGoogle's engineers have no clue what they are doing. They should be fired, honestly.
- WolfRazu 8y agoThis sounds to me like someone is simply using GoogleBot in their request headers whilst probing.
- crazygringo 8y agoWhat a weird... rant. There are so many questions here: - Is he sure it's Googlebot and not somebody spoofing headers? - Is he sure Google is generating the queries, as opposed to Google just following URL's coming from elsewhere? - Is he sure it's the search crawler, and not some browser service trying to cache autocomplete combinations or results? - Has he tried simple potential fixes like changing GET to POST for searches, or robots.txt to set a crawl interval to something like 30s? It just seems so weird. Google crawlers have finite resources so this seems like a misunderstanding or really bizarre edge case or both.
- puzzle 8y agoHe doesn't mention robots.txt once, I checked. GoogleBot not only obeys that, it also tries to detect if a site is getting overloaded, so it doesn't run it into the ground. It's not clear if he was getting hit by GoogleBot or e.g. FeedFetcher, which does NOT obey robots.txt, because it's triggered by human actions: https://support.google.com/webmasters/answer/178852?hl=en https://support.google.com/webmasters/answer/178852?hl=en There's also the possibility, as you say, that traffic was spoofed. Google publishes its netblocks over DNS, so it's easy to double check that.
- mintplant 8y ago> Is he sure it's Googlebot and not somebody spoofing headers? In the author's previous post on the subject [0], he verified that the request were coming from Google IPs. > Is he sure Google is generating the queries, as opposed to Google just following URL's coming from elsewhere? From the URLs quoted in the article, it looks more like some sort of (machine learning-y?) system trying to infer his site's canonical URL scheme. [0] https://www.hackerfactor.com/blog/index.php?/archives/484-Google-Abuse.html https://www.hackerfactor.com/blog/index.php?/archives/484-Go...
- emayljames 8y agoIs it possible that it is Google Cloud (console) IP?.
- dawnerd 8y agoA good start: https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en Then use robots.txt on any dynamic route like search. A solid site map could help too.
- burtonator 8y agoI've ran a large scale web crawler for a decade: http://www.datastreamer.io/ http://www.datastreamer.io/ The main issues and complaints we run into are mostly unusual requests for indexing content. People see the total number of requests and think it's a significant burden and just have an irrational need to protect their content. It's almost a form of paranoia at some point. I mean we have NO problem removing someone that doesn't want to be indexed and 99/100 we do so without a problem. The only times we push back are when government organizations contact us as we feel we have a legal right to index the content. We haven't had to really fight over this because they usually agree and that's the end of the discussion.
- zzzcpan 8y agoI see hundreds of thousands of requests from search engines daily without any burden, but I do block all other bots. It has nothing to do with paranoia, a lot of bots are just evil making it impossible to act towards bots in good faith. For example there are forensic bots, copyright protection bots, brand protections bots and similar bots with the whole purpose of using crawled data to send you threats and "abuse" reports or drag you to court. There are bots that scan for vulnerabilities to exploit and bots that scan pages to find something bad to blacklist somewhere. There are bots that generate spammy links or even grab content for black hat seo, something search engines punish you for. There are bots that belong to companies that sell your data or try to gain competitive advantage through this without giving you anything in return, something you may or may not be comfortable with. There are even bots that gather data for political and intelligence purposes, literally causing death and destruction.
- brennebeck 8y agoGiven the wide range of bots you mentioned, I’m very curious how you’re just blanket blocking all other bots? Could you elaborate a bit? Edit: add other; ambiguous.
- AndrewStephens 8y agoI had no idea that googlebot (and others) will sometimes POST to urls[0] - this seems like a terrible idea. I imagine what happened to this guy is that somebody coded up a site that used his as a service and google spidered it and went crazy trying to follow the links. It sucks, but that is what robots.txt is for. Anything that requires more than just serving a page should be blocked. You can drive yourself insane reading your logs trying to guess what bots are trying to achieve. They seem to excel at spidering exactly what you don't want while ignoring the content that would actually be useful. The one time I did have a problem with a misbehaving crawler was with something called 80legs[1] [0] https://webmasters.googleblog.com/2011/11/get-post-and-safely-surfacing-more-of.html https://webmasters.googleblog.com/2011/11/get-post-and-safel... [1] https://sheep.horse/2012/8/80legs_is_a_pain_in_the_neck.html https://sheep.horse/2012/8/80legs_is_a_pain_in_the_neck.html
- crazygringo 8y agoThat is pretty crazy, I'm kind of shocked. I'm sure their ML is pretty decent... but I do have to wonder if it's ever wound up leaving hundreds of comments on an anonymous forum or something, thinking it was a search box.
- emilfihlman 8y agoRelated thread: https://news.ycombinator.com/item?id=16302821 https://news.ycombinator.com/item?id=16302821 Google bot is stupid af and their support is even less helpful. TL;DR: Google just scans you site even if it always gets 404s for millions of requests.
- xg15 8y agoI'm surprised by almost all comments in this thread dismissing the article completely. Google does field trials all the way in Chrome. Why would it be so unrealistic that they did similar trials with Googlebot?
- kordlessagain 8y agoHere's the robots.txt file for hackerfactor.com: User-agent: ia_archiver Allow: / User-agent: ScoutJet Disallow: /blog User-Agent: Googlebot Allow: / Disallow: /blog/index.php?/archives/2018/ Disallow: /blog/index.php?/archives/2017/ Disallow: /blog/index.php?/archives/2016/ Disallow: /blog/index.php?/archives/2015/ Disallow: /blog/index.php?/archives/2014/ Disallow: /blog/index.php?/archives/2013/ Disallow: /blog/index.php?/archives/2012/ Disallow: /blog/index.php?/archives/2011/ Disallow: /blog/index.php?/archives/2010/ Disallow: /blog/index.php?/archives/2009/ Disallow: /blog/index.php?/archives/2008/ Disallow: /blog/index.php?/archives/2007/ Disallow: /blog/index.php?/archives/2006/ Disallow: /blog/index.php?/archives/2005/ Disallow: /blog/index.php?/archives/2004/ Disallow: /blog/index.php?/archives/2003/ Disallow: /blog/index.php?/archives/2002/ Disallow: /blog/index.php?/archives/2001/ Disallow: /blog/index.php?/archives/2000/ #Disallow: /blog/index.php?/categories/ User-agent: * Disallow: /blog Disallow: /badbot-a2d0ac98abcaf3cafd9eff83b3cffa98fec7a390a6c5b9 I'm not 100% certain about that ? in there, but I think you can do this instead: Disallow: /blog/index.php?/archives/*/
- jrnichols 8y agoWhat's surprising to me is how many hoops people have to jump through just to tell Google to go away.
- OnlyRepliesToBS 8y agooh wow, probe intensity can vary based on will