4 ms·
"default = 100 [requests per second]" How many new TCP connections per second. Is this a "scraper" or a "crawler". It appears to accept a "starting URL" and
by 1vuio0pswjnm7 3y ago
"default = 100 [requests per second]"
How many new TCP connections per second.
Is this a "scraper" or a "crawler".
It appears to accept a "starting URL" and to follow links.
Opening many TCP connections is arguably still a reason why website operators try to prevent crawling (except from Googlebot IPs). As for scraping, it can be done with a single TCP connection. Perhaps "developers" instead opt to use many TCP connections and then complain when they get blocked.