3 ms·

Looks like somebody didn't read the parent. > "scraping" site every 15 minutes with massive ddos Google doesn't scrape every 15 minutes. It was implied that t

• tokenizerrr 9y ago
  Looks like someone has never ran a website with even a slight amount of traffic. A few requests every 15 minutes is nothing and your site should be able to handle it. Additionally, it clearly wasn't a DDoS. Quoting from the parent: > Some swedish equal rights server was "scraping" site every 15 minutes with massive ddos. We put capacha for the server ip only (let them scratch their heads now) and let it be If there is only one server ip, it is by definition not distributed. Just a misuse of the term DDoS.
• thephyber 9y ago
  > Google doesn't scrape every 15 minutes. Google has a variable timing crawler. Google News crawls in almost realtime, but all Google crawlers back off if the site responds slow. Additionally, Google respects `/robots.txt` (which allows the site owner to define a crawl delay) and uses sitemaps for hints, so it doesn't have to re-crawl every single HTTP object "every 15 minutes".