4 ms·

Blocking access based on arbitrary user agent strings is a really bad idea. Every single bad bot will avoid known user agent strings or pretend to be Google, so

Blocking access based on arbitrary user agent strings is a really bad idea. Every single bad bot will avoid known user agent strings or pretend to be Google, so you're only blocking well behaved ones. Plus there's thousands of browser versions out there, so there's a very good chance you're blocking some users for no reason. The proper way to do this is to block by IP, based on behavior. Block IPs slowing down the site or throw up a captcha like cloudflare does. Blocking bots sounds great but it just brings Google one step closer to a monopoly. Even good bots just pretend to be people nowadays because lots of people are implementing naive site protection strategies. Edited: to be less mean

John23832 9y ago
> single dumbest idea I agree with your sentiment, but you should try to be a little more constructive.
alexandrnikitin 9y ago
Yes, you're right. There are many ways to block robots: IP, UA, behaviour analysis. An advertising company has to have UA based filtering to be compliant with standards. However, the focus of the blog post is on performance rather on how to block bots.
- deleted 9y ago
  [deleted]
cm2187 9y ago
Plus it's baking planned obsolescence into your code. As soon as you stop updating it, it will start blocking newer browser editions and versions.
krzrak 9y ago
> Every single bad bot will avoid known user agent strings If I was writing a bot, I would set user agent to some well known and very popular value, i.e. newest Chrome on Windows, or something like that.
jlg23 9y ago
In theory you are right, but in reality 99% of the rogue bots are actually just some scraper tools were ignorant users changed sane defaults to "make it go faster". They usually don't have enough knowledge to even understand that they are routed into a black hole - not to speak of being able to do something about it. Disclaimer: Getting rid of those idiots^wmisguides poor souls is part of my job description.
- throwasehasdwi 9y ago
  Then why not block based on bots that ignore robots text or make requests too quickly like I suggested? Looking for sub-strings in the UA header is a hack at best.
  - jlg23 9y ago
    They all require one to keep state information on the server side and with overall little benefit. Yes, I do what you proposed, but in hindsight it was a waste of time to set that up: 99% simply don't know what they are doing and therefore easy to catch. I don't care about this one person who knows to change the user agent sent, those are the ones you can usually talk to and they'll happily throttle their crawlers. The other 99 are the problem - and they are a problem that can be solved pretty well by simple string matching.
Twirrim 9y ago
The article is not about filtering bad bots. It even says so right near the top: >We won’t cover black bots because it is a huge topic with sophisticated analysis and Machine learning algorithms. We will focus on the white and grey bots that identify themselves as such. This is about not wasting time & effort showing advertising banners to good bots.

John23832 9y ago

> single dumbest idea I agree with your sentiment, but you should try to be a little more constructive.

alexandrnikitin 9y ago

Yes, you're right. There are many ways to block robots: IP, UA, behaviour analysis. An advertising company has to have UA based filtering to be compliant with standards. However, the focus of the blog post is on performance rather on how to block bots.

deleted 9y ago
[deleted]

cm2187 9y ago

Plus it's baking planned obsolescence into your code. As soon as you stop updating it, it will start blocking newer browser editions and versions.

krzrak 9y ago

> Every single bad bot will avoid known user agent strings If I was writing a bot, I would set user agent to some well known and very popular value, i.e. newest Chrome on Windows, or something like that.

jlg23 9y ago

In theory you are right, but in reality 99% of the rogue bots are actually just some scraper tools were ignorant users changed sane defaults to "make it go faster". They usually don't have enough knowledge to even understand that they are routed into a black hole - not to speak of being able to do something about it. Disclaimer: Getting rid of those idiots^wmisguides poor souls is part of my job description.

throwasehasdwi 9y ago
Then why not block based on bots that ignore robots text or make requests too quickly like I suggested? Looking for sub-strings in the UA header is a hack at best.
- jlg23 9y ago
  They all require one to keep state information on the server side and with overall little benefit. Yes, I do what you proposed, but in hindsight it was a waste of time to set that up: 99% simply don't know what they are doing and therefore easy to catch. I don't care about this one person who knows to change the user agent sent, those are the ones you can usually talk to and they'll happily throttle their crawlers. The other 99 are the problem - and they are a problem that can be solved pretty well by simple string matching.

Twirrim 9y ago

The article is not about filtering bad bots. It even says so right near the top: >We won’t cover black bots because it is a huge topic with sophisticated analysis and Machine learning algorithms. We will focus on the white and grey bots that identify themselves as such. This is about not wasting time & effort showing advertising banners to good bots.