5 ms·
The big news here is that Googlebot will be blocked from September 15th onwards by one the "block training" policies, because Google use the same crawler infras
by simonw 3mo ago
The big news here is that Googlebot will be blocked from September 15th onwards by one the "block training" policies, because Google use the same crawler infrastructure for their search index AND for training Gemini:
> Another change that will apply on September 15 is that multi-purpose crawlers (specifically those that combine Search with Training) will be allowed/blocked according to all of their behaviors, in line with our call for transparency for website owners. Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service).
- jofzar 3mo agoWe had googlebot blast a random customer system and almost cause an outage, this is when I first learnt that google will use it for AI training also. It's honestly kind of frustrating also because you then search on it and theres (was) nothing on how you are meant to "correctly" tell google to fuck off, and not use it like that.
- 20k 3mo agoGoogle's web scraping functionality has been acting as a ddos for more than two decades. I've seen literally hundreds of reports of them attacking websites and taking them down, where there's nothing you can do but accept the traffic, or get delisted This is unfortunately nothing new. There's no correct way to tell them to fuck off, they do not care, and they never will do. People have even taken them to court over this
- weird-eye-issue 3mo agoIf a site cannot handle traffic from the real Googlebot that is a serious issue with the site itself since it's actually pretty conservative Also I should note there are lots of fake Googlebots...
- remus 3mo agoIndeed, I've got a site which gets a lot of bot traffic and google bot is pretty sensible compared to a lot of other mainstream bots.
- weird-eye-issue 3mo agoYes, it really does not make that many requests. In fact lots of site owners struggle with having it not crawl and index their site enough
- 20k 3mo agoIt is mostly, but it doesn't take a lot of googling to find sites getting ridiculous amounts of traffic from googlebot on google IPs. Its one of the most common complaints about google's search indexing
- weird-eye-issue 3mo agoLots of people abuse Google Cloud to get a "Google IP" for a fake Googlebot. Why don't you show me a single screenshot from Google Search Console showing a high number of requests to a site that would be counted as a DoS? All requests from the official Googlebot are logged there so if it's such a common problem it must be very easy for you to show me this.
- weird-eye-issue 3mo agoStill waiting for a single shred of evidence ;)
- bobbiechen 3mo agoThe vast majority of Googlebot user agents are lying. Real Googlebot is pretty well behaved in my experience. You should use reverse DNS or IP lists to check: https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests https://developers.google.com/crawling/docs/crawlers-fetcher...
- simondotau 3mo agoIf you're using Cloudflare, set up a security rule to block requests that have "Googlebot" in the UA and are not recognised by CF as a real bot.
- inigyou 3mo ago[flagged]
- dannyw 3mo agoGood. Google's approach here is manifestly predator, unfair, and IMO illegal. They deserve to be in court for this behaviour, and mandating owners give consent for AI training or drop out of Google; which is just a non-starter because they're a search monopoly. That's exactly what antitrust laws are supposed to do, and I hope at least EU regulators take action. Every single Googlebot crawl in your access logs is a trace for damages.
- troyvit 3mo agoI think it's bad, because everybody is desperate to hold onto every last bit of google search traffic they can, so they're going to allow training to do so. Google's predatory, unfair and illegal actions will continue as they have with a few $100 million slaps in the wrist from the EU and a few more white house dinners for their CEO.
- PunchyHamster 3mo agoI don't think it's the Google bot DDOSing people's infrastructure for AI training...
- deleted 3mo ago[deleted]
- xhi 3mo ago[dead]
- AnthonyMouse 3mo agoThe irony here is that the people blocking all other crawlers are the ones shoring up their monopoly. If you can't block Googlebot because you need the search traffic but you block everybody else so that nobody other than Google can index your site, how do you expect to ever get any search traffic that isn't from Google?
- tempest_ 3mo agoSearch traffic is nose diving due to LLM use. The fundamental calculus with google is you install GA and it helps your SEO has changed and google is riding out what will eventually wither as people start to reevaluate the trade off.
- miohtama 3mo agoPeople will use something for search and something needs to index pages, either for LLM or old school search engine.
- motbus3 3mo agoDon't that feel like a threat to businesses who dare to avoid their content being stolen?