4 ms·
Here is an example of the "good bot"/"bad bot" nonsense in action. This is an article about the LinkedIn v hiQ case at AdWeek. curl --user-agent INSERT_ANYT
by feelin_googley 9y ago
Here is an example of the "good bot"/"bad bot" nonsense in action.
This is an article about the LinkedIn v hiQ case at AdWeek.
curl --user-agent INSERT_ANYTHING_HERE http://www.adweek.com/digital/rami-essaid-distil-networks-guest-post-linkedin-hiq-labs/
It seems AdWeek can distinguish a "good bot" from a "bad bot" irrespective of the behavior of the user^W bot, i.e., whether it is one single HTTP request or 10,000 consecutive requests is irrelevant.
How do they do it?
Pattern match against the User-Agent string.
Effective shibboleth.^W engineering.
Clarification: If a user, not a "bot", makes the "wrong" choice of user-agent string (e.g. in the browser settings), then they will be labeled a "bad bot", even if their behavior is no different than other users who are not labeled "bad bots". For example, they make one HTTP GET request just like any other user. There are databases of "acceptable" user-agent strings available to anyone. If still unsure about the point I am making, see this post from several days ago: https://www.sigbus.info/software-compatibility-and-our-own-user-agent-problem.html https://www.sigbus.info/software-compatibility-and-our-own-u...
- alexdoma 9y agoWhat would be a better solution, IP address check to allow only known google crawlers perhaps?
- marcosdumay 9y agoThe real solution is disallowing behaviors, instead of shibboleths. It's surprising that malicious bots aren't exploiting those things already.
- andreareina 9y agoClassify IPs based on their recent behavior[2]. Most bots behave very differently from the median user, along many different dimensions -- volume of requests, time between requests, visit length, which links are followed, etc. And if this means that bots are altered to become indistinguishable from users, and therefore have a minimal impact on a site's loading? Well, mission accomplished[1]. [1] https://xkcd.com/810/ https://xkcd.com/810/ ETA: [2] Recent behavior (as opposed to all historical behavior) is used so that someone inheriting a "bad" IP isn't completely screwed over.
- mark-r 9y agoThat's a superb xkcd that I hadn't seen yet, thanks.
- ben_w 9y agoThat practically invites them to present a different page to google as to a normal user, the former pure SEO, the latter perhaps pure advertising.
- fjsolwmv 9y agoAnd Google will happily deindex the site as soon as they find out
- ben_w 9y agoRaising an interesting question: can a website owner (use the law to) ban google from accessing their website by any mechanism other than their crawler in order that google doesn’t find out? Sure, obviously limited utility just like “the right to be forgotten”s flaw of diffing USA internet from EU internet to find specifically what people want forgotten, but shenanigans interest me.
- someonewithpc 9y agoA related story is that windows 9 isn't a thing because software used to check for windows 95 and 98 by matching the name to "windows 9".