3 ms·
I asked Google AI Mode “does Google ai mode make tens of site requests for a single prompt” and it showed “Looking at 69 sites” before giving a response about q
by drozycki 1y ago
I asked Google AI Mode “does Google ai mode make tens of site requests for a single prompt” and it showed “Looking at 69 sites” before giving a response about query fan-out.
Cloudflare has some large part of the web cached, IA takes too long to respond and couldn’t handle the load. Google/OpenAI and co could cache these pages but apparently don’t do it aggressively enough or at all
- ccgreg 1y agoI don't think you're correct about Google. Caching webpages is bread-and-butter for search engines, that's how they show snippets.
- danudey 1y agoThey might cache it, but what if it changed in the last 30 seconds and now their information is out of date? Better make another request just in case.
- ccgreg 1y agoThat's not how search engines work. They have a good idea of which pages might be frequently updated. That's how "news search" works, and even small startup search engines like blekko had news search.
- TheServitor 1y agoIndeed. My understanding is that crawl is a real expense at scale so they optimize for "just enough" to catch most site update rhythms and then use other signals (like blog pings, or someone searching for a URL that's not yet crawled, etc) to selectively chase fresher content.
- ccgreg 1y agoMy experience is that a news crawl is not a big expense at scale, but so far I've only built one and inherited one. BTW No one uses blog pings, the latest hotness is IndexNow.