4 ms·
> Any legit person building a legit bot would know to fake the agent string. What, that's totally backwards. Anyone using a bot to do things that might get blo
by adwww 7y ago
> Any legit person building a legit bot would know to fake the agent string.
What, that's totally backwards. Anyone using a bot to do things that might get blocked by publishers fakes the string, legit purposes should really show who / what they are.
- jedberg 7y agoWe only blocked agent strings with "lib" in them. You could change the agent to "WebScraperSupreme.com" and it would have been fine (and in fact some people did do that).
- xiongchiamiov 7y agoIt actually is encouraging people to have useful user agents. By default most people end up with a user agent that's something like "libcurl version foo.bar.baz", which isn't actually a description of who or what they are; given the prevalence of curl, it really just tells you that it's a program that uses http.