3 ms·
I don't think you're correct about Google. Caching webpages is bread-and-butter for search engines, that's how they show snippets.
by ccgreg 1y ago
I don't think you're correct about Google. Caching webpages is bread-and-butter for search engines, that's how they show snippets.
- danudey 1y agoThey might cache it, but what if it changed in the last 30 seconds and now their information is out of date? Better make another request just in case.
- ccgreg 1y agoThat's not how search engines work. They have a good idea of which pages might be frequently updated. That's how "news search" works, and even small startup search engines like blekko had news search.
- TheServitor 1y agoIndeed. My understanding is that crawl is a real expense at scale so they optimize for "just enough" to catch most site update rhythms and then use other signals (like blog pings, or someone searching for a URL that's not yet crawled, etc) to selectively chase fresher content.
- ccgreg 1y agoMy experience is that a news crawl is not a big expense at scale, but so far I've only built one and inherited one. BTW No one uses blog pings, the latest hotness is IndexNow.