5 ms·
Cloudflare: We Will Get Google to Provide a Way to Block AI Overviews
- deleted 1y ago[deleted]
- kmeisthax 1y ago[dead]
- thisislife2 1y agoI didn't understand what Cloudflare is saying, so perhaps somebody else could clarify - Google already crawls sites to index them for its search. So why would Google need a separate AI crawler? Cloudflare (according to the article) claims they already "block" Gemini - does that mean Google does operate a separate AI crawler? Why though, when they already have a huge index? What does an AI crawler do different from a search engine (indexing) crawler?
- npollock 1y agoit could be the "grounding with google search" feature when using Gemini models
- dawnerd 1y agoGoogle does crawl with a different crawler. Makes no sense other than clearly two different teams with different goals.
- eddythompson80 1y agoThe assumption is that "Google" is a good player, not a malicious one. Same idea as robots.txt. It's only there to configure the "good guys" behavior not to prevent malicious actors. So I'm assuming Cloudflare is basically asking Google to split its crawler's UA and distinguish between search and AI overview and respect something akin ot robots.txt
- Palomides 1y agocloudflare wants to sell a "AI bots have to pay to access your site" service, so they need to argue it out with google one way or another the reality of the tech is irrelevant
- beejiu 1y agoGoogle does not use a separate crawler, but they do use the robots.txt semantics to control what's allowed in the AI "index" separately from the search index. From the docs: > Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity. https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers https://developers.google.com/search/docs/crawling-indexing/...
- imilk 1y agoGoogle has made it very difficult to completely block their AI crawling by also using the standard googlebot search crawlers to feed data into their AI overviews and other AI features within Google search. Google says there is a workaround but it also blocks your site from fully indexing in Google search. This also also all covered in the article though. > What does an AI crawler do different from a search engine (indexing) crawler? Many people don't want the extra bot traffic hitting their site that comes from AI, especially when AI chat & AI overviews in Google provide such a small amount of traffic in return and that traffic pretty much always has horrendous conversion rates (personally seen across multiple industries).
- Spivak 1y agoIn the specific case of Google would there be any additional traffic that isn't just the normal googlebot? I can't imagine they would bother crawling twice for every site on the internet.
- imilk 1y agoThere are about a dozen Google crawlers that can hit your website for different reasons: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers https://developers.google.com/search/docs/crawling-indexing/... Google-Extended is what is associated with AI crawling, but GoogleBot also crawls to produce AI overviews in addition to indexing your website in Google search. While the number of crawlers and their overlapping responsibilities makes it difficult to know which ones you can safely block, I should also say that pure AI company bots behave 1000x worse than Google crawlers when it comes to not flooding your site with scraping requests.
- fathomdeez 1y agoIt doesn't seem like the extra traffic is the issue. People don't want Google's AI from reading and summarizing their data and thus preventing clickthroughs. Why would I click on your site if google did all the work of giving me the answer ahead of time?
- 1y ago
- Snoozus 1y agoIs this how the democratic process works now? Cloudflare threatens Google to pass a law?
- thisislife2 1y agoDo you have any more doubts who run the US and makes laws there?
- handfuloflight 1y agoHard to believe Cloudflare has more sway than Google.
- pjc50 1y agoTech regulation, or lack thereof, tends to be "biggest pile of money wins", but in this case there's already large anti-Google and anti-AI constituencies which CF may be able to mobilize. Especially in the EU.
- eviks 1y ago"Always has been"
- imilk 1y agoWhat has been democratic about how the internet has evolved over the last 2 decades? Because as far as I can see, the internet has undergone a massive centralization into the hands of a few players with practically no regulation. Especially Google, which can make decisions such as adding AI Overviews to search results leading to millions of websites seeing a ~25% drop in organic traffic in the last few months.
- kingstnap 1y agoI don't understand how AI scrapers make up such a large percentage of traffic to websites, as people claim it does. In principle, if you post a webpage, presumably, it's going to be viewed at least a few dozen times. If it's an actually good article, it might be viewed a few hundred or even thousands of times. If each of the 20 or so large AI labs visit it as well, does it just become N+20? Or am I getting this wrong somehow?
- wat10000 1y agoMaybe they're vibe-coding the scrapers.
- csomar 1y agoI don't understand it either. I track requests and AI crawlers are there but not as abusive as people claim. Most annoying requests are from hackers who are trying to find my ".git" directory. But I highly doubt these guys will respect any rules anyway.
- imilk 1y agoVastly varies on what type of website you have, how many pages you have, and how often they are updated. We routinely see 1000's of requests per minute coming from AI bots and the scraping lasts for hours. Enough to make up 20-30% of overall requests to the server.
- delfinom 1y agoI have a symbol server hosting a few thousand PDBs for a foss package. Amazonbot is every day trying to scan every single PDB directory listed. For no real reason. This is something causing 10k+ requests each day when legitimate traffic sits at maybe 50 requests a day.
- dsign 1y agoIf you run a website, you'll realize it's very difficult to get human traffic. Worse, trying to understand what those eyeballs are doing is a swamp; there are legitimate privacy concerns for example. Maybe all you care about is if your articles about sewing machines are getting more traction than your articles about computing Pi, but you can't get that without navigating all the legal complications of your analytics platform of choice, who wants to make sure you suffer for not letting them collect private information on your visitors to sell to third parties and to dump ads onto your visitors. Were it not for the bots, you would be fine just by running grep on your access logs. But no, bot traffic leaves noise everywhere; and for small websites that noise is more than enough to bury the signal and to be most of the traffic bill.
- ninetyninenine 1y agoThey are going to have to use the law. Are you kidding? You block ai for Google while every other agent is able to bypass that block and crawl your site? How will Google stay competitive for AI? Of course Google would rather have you do it by law. You block it for them you better do it for every single one of their competitors too. Also guaranteed even if you do block by law there will be actors who will ignore the law. For example people outside of the US. Those people will as a result likely build better AI because they have access to more training data. First example of this is of course China. One thing with China is there’s no holy sanctity behind data. Whatever is made is copy-able and goes to the public domain regardless of anything. It both causes China to both exceed the US and to be slightly less innovative at the same time. If these laws come to pass you bet your ass China will be exceeding the US in AI like they already have with stem cell research.
- delfinom 1y agoAh ok, yet the megacorps continue to reap profits off stealing labor and work from the peasants. I propose we just remove the charades and require us peasants pay these megacorps 20% of our annual income yearly at tax time.
- ninetyninenine 1y agoWho pays the megacorps to use AI? The peasants. Business only succeeds if the peasantry want it. So if businesses are fucking over the peasantry what’s really going on is that the businesses are just middlemen for peasantry fucking themselves. In the end the megacorps are just a feedback loop to the peasantry. That’s the irony. You like to blame corporations but it’s just a mirror to yourself. Still doesn’t hurt to lower taxes for peasants and heighten taxes for corps. That’s how you prevent wealth inequality. And also middlemen are by nature just a bit corrupt as they siphon resources just by being in the middle.
- handfuloflight 1y agoSo this boils down to Cloudflare, supposedly having more regulatory capture than Google.
- nathanyz 1y agoThe thing that he doesn't mention is that as soon as they do something legislatively and announce routes there, etc.....well Google just won't crawl those sites. It turns into a game of whether you would like 0 traffic from Google, or allow them to use your content both for search results and AI summaries. Google is the bringer of traffic and if you want it, then you play by their rules. I don't like that the web is in that position, but here we are.
- lbhdc 1y agoI think it gets tricky with AI overviews. Some sites are saying they have lost the majority of their traffic (some losing more than 90%). Many of those make money from ads (often by google). Some sites don't get enough traffic from google to sustain their business where they previously did. Wholesale blocking google crawlers doesn't seem like a risky move for them. It makes me wonder if that is a trend? Will more sites go to 'google zero'?
- handfuloflight 1y agoDoesn't this also cannibalize Google's ad revenue?
- lbhdc 1y agoTo some extent, yes. I speculate that is why they are sometimes not putting the AI overview first. They do have ads in search though. I guess it depends on how much more money they make from people spending more time not leaving google vs showing you display ads on other sites.
- imilk 1y agoNot really because most of the links in current AI overviews for many queries keep you in the Google ecosystem, such as links to Google Business profiles on Google Maps. And Google has recently changed Google Business profiles to make outbound website links less prominent compared to the paid features on that platform.
- zelon88 1y agoWatching these two behemoths wrestle over the future of a space we all share, and wondering if they will need to loop in regulators on one side or another, convinces me that we shouldn't have gifted all of our digital infrastructure to just 2 companies. Inlcuding our economy, healthcare, government, and civil infrastructure. We've put all our eggs into only a couple of very greedy, impossible to audit baskets. We've really done this all to ourselves. We've raced ourselves all the way to the bottom.
- coldpie 1y agoAfter 50 years of effectively zero activity, we had some glimmers of anti-trust enforcement under the Biden admin. But then eggs were expensive in the summer of 2024, so we decided it was actually no problem for these half-dozen companies to control our speech and economy, and here we are.
- eddythompson80 1y agoIt used to be just Ma Bell
- philipallstar 1y ago> we shouldn't have gifted all of our digital infrastructure to just 2 companies We didn't. Just as we didn't gift all our chocolate-making infrastructure to Hershey's and Cadbury's.
- AlecSchueler 1y agoHey did you forget the market is consumer driven?
- imilk 1y agoThere is nothing stopping other CDN/DNS providers from implementing similar services and tools to what Cloudflare offers. Part of the reason CF has become so popular is because so many of their competitors don't offer nearly the same convenience for routine tasks & protection.
- msgodel 1y agoI've still never understood the complaint here. Robots are part of the web, the whole point of HTML is that robots can read it.
- marginalia_nu 1y agoIt's a cost thing. It costs more to render a website than it does to consume it. When you have some bot traffic mixed in with human traffic, that is fine. When you have egregious bot traffic, say 10k requests per minute sustained load, it becomes a real problem for webmasters.
- pjc50 1y agoThe whole point of publication is so that humans can read it. Robots not so much, especially if they're not paying customers. This is the distinction between how the web works technically and how it works socioeconomically. This is the next iteration of things like the news snippet case. Publishers are not happy that Google crawls their content (at their expense) and then republishes it on their own site, while serving ads around it and getting user data, without cutting in the publisher who originally made it. And, for what little it's worth, owns the copyright.
- msgodel 1y agoRobots don't exist for their own sake, at the end of the day they are user agents for some group of humans. Again it sounds like the people who are upset by this really want to publish images rather than web pages.
- imilk 1y ago> Again it sounds like the people who are upset by this really want to publish images rather than web pages. More like people don't want to lose money because a 3rd party stole all of their content, and then repurposed it to show people before they visit their website.
- CaptainFever 1y agoWeb 3.0 is all about machine-readability: https://en.m.wikipedia.org/wiki/Semantic_Web https://en.m.wikipedia.org/wiki/Semantic_Web (Not to be confused with Web3.)
- SMKKK11 1y ago..
- radicalbyte 1y agoCan we block them for customers? As they're extremely inaccurate. For example the Google "summary" for "stop killing games started by" was returning "Scott Ross". It was started by Ross Scott. The first search results provide the correct example.
- CaptainFever 1y agoYou can use the element picker in uBlock Origins.
- andy99 1y agoWe're seeing the free (libre) internet destroyed in front of us.
- imilk 1y agoI come to the exact opposite conclusion. A few large companies scraping and repurposing original content from publishers kills original content on the internet because it takes out the ability to earn a livelihood from creating original content or running your own ecommerce store that is not tied to a mega-company's platform.
- ChrisArchitect 1y agoRelated discussion previously: Cloudflare to introduce pay-per-crawl for AI bots https://news.ycombinator.com/item?id=44432385 https://news.ycombinator.com/item?id=44432385
- nunez 1y ago> Worst case we’ll pass a law somewhere that requires them to break out their crawlers and then announce all routes to their crawlers from there. And that wouldn’t be hard. But I’m hopeful it won’t need to come to that. It's interesting to see Matthew say the quiet part out loud here, if by "pass a law," Matthew means get federal legislation passed.
- imilk 1y agoProbably because that's one of the only things you can say to Google these days that will escalate it from an low level support agent on the other side of the world.