4 ms·

"We built a page logger. Any time a page was loaded, we'd keep track of it. You know what we found? The 80% of clicks we were paying for were from bots." How d

by veyron 14y ago

"We built a page logger. Any time a page was loaded, we'd keep track of it. You know what we found? The 80% of clicks we were paying for were from bots." How do you tell if it's a bot? Just by user-agent string or repeated slamming by IP address, or is there some other technique?

vladd 14y ago
The main technique is to notice a pattern that is only explainable by bot activity. Generally speaking, in addition to the patterns you mentioned (invalid user-agent strings, rapid succession of same-IP access) there are also other patterns, such as failure to load referenced CSS or image files, or 100% failure on Captcha challenges.
- veyron 14y ago
  "failure to load referenced CSS or image files" -- is there a way for bots (in particular, mechanize in perl/python) to also download those files? "100% failure on Captcha challenges" -- are captchas generally deployed on click-through ads?
  - mootothemax 14y ago
    "failure to load referenced CSS or image files" -- is there a way for bots (in particular, mechanize in perl/python) to also download those files? Very much so, after all your browser does without you having to tell it to do so. It all depends on whether the hassle is worth it for the bot writer.
  - loumf 14y ago
    Of course they could -- all they need to do is parse the HTML, look for the img and link and other tags, and then request the URL. If they were trying to act more like a browser, they would. If they are just trying to click ads, they wouldn't.
  - vladd 14y ago
    > "100% failure on Captcha challenges" -- are captchas generally deployed on click-through ads? Cloudflare has a mode (used in general for DDOS protection) which you can turn on and get stats on how many users pass a Captcha test.
  - slig 14y ago
    > "failure to load referenced CSS or image files" -- is there a way for bots (in particular, mechanize in perl/python) to also download those files? Use http://phantomjs.org/ http://phantomjs.org/ and the page will loaded by webkit.
  - netfire 14y ago
    Using Captcha on an ad landing page to determine whether someone is bot seems like a bad idea. It would almost certainly increase the bounce rate on that page and you couldn't just say that people who didn't choose fill out the captcha were bots.
  - oelmekki 14y ago
    > is there a way for bots (in particular, mechanize in perl/python) to also download those files? The point was mostly that if it is not downloaded, then it's a bot (or any non-browser getter, at least), not that it's not a bot if it does. Or maybe you were asking for a way to avoid bot detection ? :)
patrickmclaren 14y ago
Most likely a combination. In addition to analyzing the user's navigational path, exit points, IP ranges, etc...
jiggy2011 14y ago
They said that for 80% of their clicks , Javascript was off. Now there will be plenty of users who have disabled Javascript but nowhere near 80%. Although they stated they built their own analytics software, so there's also a chance that it wasn't working as they thought it was.
- oellegaard 14y ago
  Furthermore, facebook users in specific I would say have javascript enabled, even though its possible to control per site, I would assume most facebook users would not be aware how it works - and facebook without javascript seems rather impossible or at least, very discomforting.
mootothemax 14y ago
How do you tell if it's a bot? Just by user-agent string or repeated slamming by IP address, or is there some other technique? A combination of things. Off the top of my head: * IP address ranges. Are they all from, e.g., EC2 instances? * Are images and other content (e.g. CSS) linked from the page loaded? How long does the client take to do so? Are other pages being viewed in line with what you've seen from other users? * What combinations of user agents are you seeing? * What's in the headers being sent by the browser? Any foreign language support, or oddities when it comes to not accepting compressed content? Is the referer header always the same? Does it always vary? * Do you see "normal" user behaviour that indicates activity such as clicking the back button, then clicking the ad again a few seconds later once their brain has kicked in? (nb: not sure if this is possible for Facebook with the way their site works). * Are there any patterns in the page access times? Is there anything to indicate a cron job is kicking off once an hour at the same time?
- veyron 14y ago
  Is there a single tool that can take access/error logs and compile these stats? This sounds like something relatively useful ...
  - AUmrysh 14y ago
    If not, it sounds like a great place to use ML/classification algorithms.
pandog 14y ago
From the article it would appear that their definition of a bot is a client with JavaScript disabled
- veyron 14y ago
  "So we did what any good developers would do. We built our own analytic software. Here's what we found: on about 80% of the clicks Facebook was charging us for, JavaScript wasn't on." Later: "So we did what any good developers would do. We built a page logger. Any time a page was loaded, we'd keep track of it. You know what we found? The 80% of clicks we were paying for were from bots. That's correct. Bots were loading pages and driving up our advertising costs" I think they use a different definition of bot. Otherwise they wouldn't have to use a second logger ...

by veyron 14y ago

vladd 14y ago

The main technique is to notice a pattern that is only explainable by bot activity. Generally speaking, in addition to the patterns you mentioned (invalid user-agent strings, rapid succession of same-IP access) there are also other patterns, such as failure to load referenced CSS or image files, or 100% failure on Captcha challenges.

veyron 14y ago
"failure to load referenced CSS or image files" -- is there a way for bots (in particular, mechanize in perl/python) to also download those files? "100% failure on Captcha challenges" -- are captchas generally deployed on click-through ads?
- mootothemax 14y ago
  "failure to load referenced CSS or image files" -- is there a way for bots (in particular, mechanize in perl/python) to also download those files? Very much so, after all your browser does without you having to tell it to do so. It all depends on whether the hassle is worth it for the bot writer.
- loumf 14y ago
  Of course they could -- all they need to do is parse the HTML, look for the img and link and other tags, and then request the URL. If they were trying to act more like a browser, they would. If they are just trying to click ads, they wouldn't.
- vladd 14y ago
  > "100% failure on Captcha challenges" -- are captchas generally deployed on click-through ads? Cloudflare has a mode (used in general for DDOS protection) which you can turn on and get stats on how many users pass a Captcha test.
- slig 14y ago
  > "failure to load referenced CSS or image files" -- is there a way for bots (in particular, mechanize in perl/python) to also download those files? Use http://phantomjs.org/ http://phantomjs.org/ and the page will loaded by webkit.
- netfire 14y ago
  Using Captcha on an ad landing page to determine whether someone is bot seems like a bad idea. It would almost certainly increase the bounce rate on that page and you couldn't just say that people who didn't choose fill out the captcha were bots.
- oelmekki 14y ago
  > is there a way for bots (in particular, mechanize in perl/python) to also download those files? The point was mostly that if it is not downloaded, then it's a bot (or any non-browser getter, at least), not that it's not a bot if it does. Or maybe you were asking for a way to avoid bot detection ? :)

patrickmclaren 14y ago

Most likely a combination. In addition to analyzing the user's navigational path, exit points, IP ranges, etc...

jiggy2011 14y ago

They said that for 80% of their clicks , Javascript was off. Now there will be plenty of users who have disabled Javascript but nowhere near 80%. Although they stated they built their own analytics software, so there's also a chance that it wasn't working as they thought it was.

oellegaard 14y ago
Furthermore, facebook users in specific I would say have javascript enabled, even though its possible to control per site, I would assume most facebook users would not be aware how it works - and facebook without javascript seems rather impossible or at least, very discomforting.

mootothemax 14y ago

How do you tell if it's a bot? Just by user-agent string or repeated slamming by IP address, or is there some other technique? A combination of things. Off the top of my head: * IP address ranges. Are they all from, e.g., EC2 instances? * Are images and other content (e.g. CSS) linked from the page loaded? How long does the client take to do so? Are other pages being viewed in line with what you've seen from other users? * What combinations of user agents are you seeing? * What's in the headers being sent by the browser? Any foreign language support, or oddities when it comes to not accepting compressed content? Is the referer header always the same? Does it always vary? * Do you see "normal" user behaviour that indicates activity such as clicking the back button, then clicking the ad again a few seconds later once their brain has kicked in? (nb: not sure if this is possible for Facebook with the way their site works). * Are there any patterns in the page access times? Is there anything to indicate a cron job is kicking off once an hour at the same time?

veyron 14y ago
Is there a single tool that can take access/error logs and compile these stats? This sounds like something relatively useful ...
- AUmrysh 14y ago
  If not, it sounds like a great place to use ML/classification algorithms.

pandog 14y ago

From the article it would appear that their definition of a bot is a client with JavaScript disabled

veyron 14y ago
"So we did what any good developers would do. We built our own analytic software. Here's what we found: on about 80% of the clicks Facebook was charging us for, JavaScript wasn't on." Later: "So we did what any good developers would do. We built a page logger. Any time a page was loaded, we'd keep track of it. You know what we found? The 80% of clicks we were paying for were from bots. That's correct. Bots were loading pages and driving up our advertising costs" I think they use a different definition of bot. Otherwise they wouldn't have to use a second logger ...