2 ms·
It is in the name, rules for the robots. Any scraping ai or not, and even mass recrsive or single page, should abide by the rules.
by gunalx 1y ago
It is in the name, rules for the robots. Any scraping ai or not, and even mass recrsive or single page, should abide by the rules.