2 ms·
> Does any of these scrapers uniquely and unambiguously identify themselves as a bot? It seems like all of them do, yeah: https://github.com/eob/isai/blob/b906
by diggan 2y ago
> Does any of these scrapers uniquely and unambiguously identify themselves as a bot?
It seems like all of them do, yeah: https://github.com/eob/isai/blob/b9060db7dc1a7789b322b8c2838514fc4e596779/src/patterns.json https://github.com/eob/isai/blob/b9060db7dc1a7789b322b8c2838...
Not sure if they're really "scrapers" though, if they're initiated by a user for a single webpage/website, more like "user-agents" in that case, unless it automatically fans out from there to get more content.
- deleted 2y ago[deleted]