2 ms·
> Google doesn't repeat requests every day. I don't repeat requests every day. I scrape one site whose content changes every second. How is it "nonsense" to ma
by matheusmoreira 5y ago
> Google doesn't repeat requests every day. I don't repeat requests every day.
I scrape one site whose content changes every second. How is it "nonsense" to make one request every 24 hours? I make hundreds, thousands when I browse their site normally using my browser.
We've got people in this very thread talking about bots making hundreds of requests per second. How is one request every 24 hours harming anyone? People told me to make one request per hour to avoid hammering their servers, I decided to wait a day instead. It boggles my mind that this generous interval could possibly be considered abuse. How long should the interval be then? A month? A year? Infinitely long so the scraper never makes requests?