3 ms·
> Google doesn't scrape every 15 minutes. Google has a variable timing crawler. Google News crawls in almost realtime, but all Google crawlers back off if the
by thephyber 9y ago
> Google doesn't scrape every 15 minutes.
Google has a variable timing crawler. Google News crawls in almost realtime, but all Google crawlers back off if the site responds slow.
Additionally, Google respects `/robots.txt` (which allows the site owner to define a crawl delay) and uses sitemaps for hints, so it doesn't have to re-crawl every single HTTP object "every 15 minutes".