3 ms·
If we did, bot authors would comply by just changing their User-Agent to something different that’s not expressly forbidden. (Disallowing * isn’t usually an op
by WhyNotHugo 2y ago
If we did, bot authors would comply by just changing their User-Agent to something different that’s not expressly forbidden.
(Disallowing * isn’t usually an option since it makes you disappear from search engines).
- danaris 2y agoAny such law would absolutely have to include a requirement that all bots use at least some common element in their User-Agent strings to identify themselves as bots.