4 ms·
> Many people don't want their data used for free/any training. That is true. But robots.txt is not designed to give them the ability to prevent this.
by JimDabell 1y ago
> Many people don't want their data used for free/any training.
That is true. But robots.txt is not designed to give them the ability to prevent this.
- gunalx 1y agoIt is in the name, rules for the robots. Any scraping ai or not, and even mass recrsive or single page, should abide by the rules.