5 ms·

What's your take on ML for bot classification? How successful has that family of strategies been in your opinion? One could speculate on what particular feature

What's your take on ML for bot classification? How successful has that family of strategies been in your opinion? One could speculate on what particular features of client behavior a model would hone in on to detect a bot, but it would actually likely be unexpected behavioral oddities not shared by legit clients that a human developer's intuition wouldn't think of.

folkhack 7y ago
This is an excellent question! When it comes to a web "interaction" there's a ton of differences depending on the job at-hand. If it's jumping on a Nike drop you're not going to run into those sort of methods where-as if it's scraping hundreds of thousands of SKUs there's a huge chance you will. When a human opens a "conversation" with a website usually it's for a specific action which is a small burst of data, ie: looking at a few products on Amazon. So as long as you can design your bot's "conversation" with said website in the same way you can totally bypass ML detection (and everything else for that matter). Basically if you think of ALL of the unique variables that are happening back and forth in that conversation it's a finite list. If you can check off each and every one of those you can get past anti-bot measures no-problemo, it's just a matter of budget/time. ML can HELP to identify bot traffic, but if you're sending perfect headers/traffic/browser metrics/CC numbers/shipping addresses (this list gets long) you're still going to squeak by with an acceptable risk score without a problem. The other thing is going raw-HTTP request often bypasses almost all of that crap, way more than one would think! I will often get "fingerprinted" with a very valid browser, then continue my work after treating their site as an API; ie: talking directly to the backend (or rendered HTML page) and no one else. The #1 thing I'd say where bot classification works is network reputation... so I use services that allow me to proxy-out to VERY reputable networks that are actually business/residential ISP connections. This lets me get past the majority of countermeasures because if they start blocking those sort of IP addresses/ranges they're going to block real users at some point. Unfortunately, highly reputable proxy services do cost money - but for someone who does this professionally it's all about budgeting etc. To implement these systems well takes a really keen implementation between your front-end and back-end and most companies (even the big ones) don't have the development sophistication to pull it off. Those that do are just more expensive because you'll be bouncing around between 40 different "origins" with unique browser profiles. Sorry for the essay =)
- yetanotherjosh 7y ago
  > I will often get "fingerprinted" with a very valid browser, then continue my work after treating their site as an API; ie: talking directly to the backend (or rendered HTML page) and no one else. This seems like just the kind of behavior that an ML approach could easily identify. Even just feeding a trained model your basic request log data would show you as quite a different kind of user: you're not fetching images, javascript, etc and would have a substantially different traffic profile. Obviously you could get around that by scripting a browser, but that just kicks the can down the road: a scripted browser will still likely behave in some measurable way different than a human-driven browser, and the specifics of those differences are unlikely to be found by intuition but rather by ML which can hone in on things we wouldn't think of. For example, the time spent in certain pages/activities, or the position of the cursor on links being clicked (when you tell a headless browser to click a link, do the mouse coordinates in the click event look normal or are they at the upper left coordinate of the link's position?) etc. The more data you feed such an approach the more details it can use to find anomalies that differentiate humans from bots.
  - folkhack 7y ago
    Totally agree but the thing is it's a TON of sophistication to pull that off and block proactively to the point that I'd say the only people who employ methods like that are Amazon (that I've encountered). Typically when you do that you will spoof a crawler and it gets you by just fine.

folkhack 7y ago

This is an excellent question! When it comes to a web "interaction" there's a ton of differences depending on the job at-hand. If it's jumping on a Nike drop you're not going to run into those sort of methods where-as if it's scraping hundreds of thousands of SKUs there's a huge chance you will. When a human opens a "conversation" with a website usually it's for a specific action which is a small burst of data, ie: looking at a few products on Amazon. So as long as you can design your bot's "conversation" with said website in the same way you can totally bypass ML detection (and everything else for that matter). Basically if you think of ALL of the unique variables that are happening back and forth in that conversation it's a finite list. If you can check off each and every one of those you can get past anti-bot measures no-problemo, it's just a matter of budget/time. ML can HELP to identify bot traffic, but if you're sending perfect headers/traffic/browser metrics/CC numbers/shipping addresses (this list gets long) you're still going to squeak by with an acceptable risk score without a problem. The other thing is going raw-HTTP request often bypasses almost all of that crap, way more than one would think! I will often get "fingerprinted" with a very valid browser, then continue my work after treating their site as an API; ie: talking directly to the backend (or rendered HTML page) and no one else. The #1 thing I'd say where bot classification works is network reputation... so I use services that allow me to proxy-out to VERY reputable networks that are actually business/residential ISP connections. This lets me get past the majority of countermeasures because if they start blocking those sort of IP addresses/ranges they're going to block real users at some point. Unfortunately, highly reputable proxy services do cost money - but for someone who does this professionally it's all about budgeting etc. To implement these systems well takes a really keen implementation between your front-end and back-end and most companies (even the big ones) don't have the development sophistication to pull it off. Those that do are just more expensive because you'll be bouncing around between 40 different "origins" with unique browser profiles. Sorry for the essay =)

yetanotherjosh 7y ago
> I will often get "fingerprinted" with a very valid browser, then continue my work after treating their site as an API; ie: talking directly to the backend (or rendered HTML page) and no one else. This seems like just the kind of behavior that an ML approach could easily identify. Even just feeding a trained model your basic request log data would show you as quite a different kind of user: you're not fetching images, javascript, etc and would have a substantially different traffic profile. Obviously you could get around that by scripting a browser, but that just kicks the can down the road: a scripted browser will still likely behave in some measurable way different than a human-driven browser, and the specifics of those differences are unlikely to be found by intuition but rather by ML which can hone in on things we wouldn't think of. For example, the time spent in certain pages/activities, or the position of the cursor on links being clicked (when you tell a headless browser to click a link, do the mouse coordinates in the click event look normal or are they at the upper left coordinate of the link's position?) etc. The more data you feed such an approach the more details it can use to find anomalies that differentiate humans from bots.
- folkhack 7y ago
  Totally agree but the thing is it's a TON of sophistication to pull that off and block proactively to the point that I'd say the only people who employ methods like that are Amazon (that I've encountered). Typically when you do that you will spoof a crawler and it gets you by just fine.