3 ms·
Haven't some companies explicitly ignored robots.txt to scrape the sites more quickly (and pissing off a number of people)
by _joel 2y ago
Haven't some companies explicitly ignored robots.txt to scrape the sites more quickly (and pissing off a number of people)