4 ms·
The rules are that you can't send google different page content than regular browsers, but there's no reason they have to run all the background processes on go
by ultrasaurus 16y ago
The rules are that you can't send google different page content than regular browsers, but there's no reason they have to run all the background processes on googlebot requests -- can't they just send it the most recent cached version?
- eli 16y agoNot a bad idea, but seems like it would be tricky to get right. You do kinda want Google to have the most recent version of a page, all other things being equal.
- ultrasaurus 16y agoAgreed, there's a sweet spot somewhere though, and it might not be the same for googlebot as a regular viewer).
- tomkarlo 16y agoWhy not put the cached content in the page by default, then do an update via AJAX only in the case where the cached version is old? That way it's not triggered for crawlers. It's probably secondary content anyway.
- iuytghyjuyh 16y agoIIRC they were talking about having a separate web server just for google's spider so it could be redirected by the load balancer and be tuned for more static content