15 ms·

A robots.txt that asks AI scrapers not to scrape? There’s a couple services that keep updated lists of known scraper user agents. A quick search reveals a hand

by dartos 2y ago

A robots.txt that asks AI scrapers not to scrape? There’s a couple services that keep updated lists of known scraper user agents. A quick search reveals a handful.