3 ms·
Maybe scrapers should respect robots.txt and go away if there's a `scraperbot` entry, the same way Google goes away if you put `googlebot` in there.
by dafoex 6y ago
Maybe scrapers should respect robots.txt and go away if there's a `scraperbot` entry, the same way Google goes away if you put `googlebot` in there.