6 ms·

While interesting and well-intentioned, the advice in the article can cause issues. The agent isn't downloading CSS or images? Could be a blind user. Pages bein

While interesting and well-intentioned, the advice in the article can cause issues. The agent isn't downloading CSS or images? Could be a blind user. Pages being downloaded in quick succession? Could be browser prefetch. Lack of mouse movements during navigation? It could be a user on a mobile device or screen reader. What if the bot is making individual requests from unique IP addresses? What if it's scraping many pages over time rather than a single smash-and-grab approach? The article admits that this is a hard thing to solve. In most cases, it's probably not very worthwhile to try to detect bots in the ways that are suggested. Focusing on patterns in the server logs to find out what's being targeted might be more beneficial. Then slap a login or captcha around anything valuable that bots shouldn't have access to

hinkley 7y ago
I joke that I only have two real skills. Asking questions and identifying good ideas. In the latter category I have heterogenous load balancing, segregated by traffic type. For example, if you have a read-mostly web application, often the most expensive parts are in the administrative functions. If they run on separate hardware, then a regression doesn’t affect everyone, and a client side bug or flood of traffic doesn’t automatically lock people out of the admin workflow. Similarly, you have server side rendering for web crawlers and anyone with JavaScript disabled for Angular and other interactive frameworks, which generally involves running a small quantity of specialized services on a few machines. Shunting all users who smell like a bot to a small set of machines doesn’t lock them out, but it does preserve the experience for everybody else.
zzzcpan 7y ago
I somewhat agree, but those can be good signals though, you just have to gather statistical data on them from known good users and combine with other signals. Like what are the chances for an agent to be used by a real person if it doesn't download images, lacks mouse movements and, for example, is coming from a datacenter? Obviously pretty low. The same way access patterns alone don't give you much, but are good against scrapers when combined with other signals. And putting captchas or logins on anything you have to protect from bots also "protects" from real people, most of them simply leave and don't bother with those thing.
- heavyset_go 7y ago
  > Like what are the chances for an agent to be used by a real person if it doesn't download images, lacks mouse movements and, for example, is coming from a datacenter? Obviously pretty low. Congrats, you just identified a visually impaired user with a screen reader who uses a VPN through AWS/GCP/etc.
  - zzzcpan 7y ago
    And your point is?
    - bryanrasmussen 7y ago
      that depending on your sites purpose and where your servers are located / where your main customer base is you might be opening yourself to extremely negative public relations by blocking this person as a bot, or even legal repercussions.
      - zzzcpan 7y ago
        This is completely irrelevant to the conversation about technical solutions. My example is just about probabilities and how to use them for bot detection.
        gregoryl 7y ago
        If that logic is extended, the true technical solution is to turn the site off entirely.
        zzzcpan 7y ago
        What exactly in that logic in any way leads to turning off the site entirely? It's literally the opposite: narrowing down the bots by calculating probability of them not being real people using multiple signals and like Bayesian inference or something like that, it's pretty much a primitive machine learning. I'm surprised there are so many trolling comments here.
forgotmypw 7y ago
I use a keyboard-driven browser to reduce wrist strain, and I sometimes get tagged as a bot, probably for lack of mouse movement.
- bryanrasmussen 7y ago
  basically I've written bots for scraping governmental sites, and fashion sites. I've never yet had a problem (although the code on one fashion site looked like all classes were randomly generated with implied some attempt to keep scraping from working), so what I'm wondering is - what kinds of sites actually go through the trouble of tagging you as a bot, since you've got experience care to share?
  - edoceo 7y ago
    Ticketmaster
    - MonkeyIsNull 7y ago
      Good response, I definitely never though of that before. Serious case where you don't want Bots buying stuff up.
      - edoceo 7y ago
        Ages ago I was one of those bots. Most fun I've had playing cat-and-mouse. Their team was actually quite good. And I learned that Perl can solve any problem.
        GrayTextIsTruth 7y ago
        Have you written about this? Sounds like an entertaining story.
        edoceo 7y ago
        I have not. But I will and will post to HN.

hinkley 7y ago

I joke that I only have two real skills. Asking questions and identifying good ideas. In the latter category I have heterogenous load balancing, segregated by traffic type. For example, if you have a read-mostly web application, often the most expensive parts are in the administrative functions. If they run on separate hardware, then a regression doesn’t affect everyone, and a client side bug or flood of traffic doesn’t automatically lock people out of the admin workflow. Similarly, you have server side rendering for web crawlers and anyone with JavaScript disabled for Angular and other interactive frameworks, which generally involves running a small quantity of specialized services on a few machines. Shunting all users who smell like a bot to a small set of machines doesn’t lock them out, but it does preserve the experience for everybody else.

zzzcpan 7y ago

I somewhat agree, but those can be good signals though, you just have to gather statistical data on them from known good users and combine with other signals. Like what are the chances for an agent to be used by a real person if it doesn't download images, lacks mouse movements and, for example, is coming from a datacenter? Obviously pretty low. The same way access patterns alone don't give you much, but are good against scrapers when combined with other signals. And putting captchas or logins on anything you have to protect from bots also "protects" from real people, most of them simply leave and don't bother with those thing.

heavyset_go 7y ago
> Like what are the chances for an agent to be used by a real person if it doesn't download images, lacks mouse movements and, for example, is coming from a datacenter? Obviously pretty low. Congrats, you just identified a visually impaired user with a screen reader who uses a VPN through AWS/GCP/etc.
- zzzcpan 7y ago
  And your point is?
  - bryanrasmussen 7y ago
    that depending on your sites purpose and where your servers are located / where your main customer base is you might be opening yourself to extremely negative public relations by blocking this person as a bot, or even legal repercussions.
    - zzzcpan 7y ago
      This is completely irrelevant to the conversation about technical solutions. My example is just about probabilities and how to use them for bot detection.
      - gregoryl 7y ago
        If that logic is extended, the true technical solution is to turn the site off entirely.
        zzzcpan 7y ago
        What exactly in that logic in any way leads to turning off the site entirely? It's literally the opposite: narrowing down the bots by calculating probability of them not being real people using multiple signals and like Bayesian inference or something like that, it's pretty much a primitive machine learning. I'm surprised there are so many trolling comments here.

forgotmypw 7y ago

I use a keyboard-driven browser to reduce wrist strain, and I sometimes get tagged as a bot, probably for lack of mouse movement.

bryanrasmussen 7y ago
basically I've written bots for scraping governmental sites, and fashion sites. I've never yet had a problem (although the code on one fashion site looked like all classes were randomly generated with implied some attempt to keep scraping from working), so what I'm wondering is - what kinds of sites actually go through the trouble of tagging you as a bot, since you've got experience care to share?
- edoceo 7y ago
  Ticketmaster
  - MonkeyIsNull 7y ago
    Good response, I definitely never though of that before. Serious case where you don't want Bots buying stuff up.
    - edoceo 7y ago
      Ages ago I was one of those bots. Most fun I've had playing cat-and-mouse. Their team was actually quite good. And I learned that Perl can solve any problem.
      - GrayTextIsTruth 7y ago
        Have you written about this? Sounds like an entertaining story.
        edoceo 7y ago
        I have not. But I will and will post to HN.