8 ms·

"failure to load referenced CSS or image files" -- is there a way for bots (in particular, mechanize in perl/python) to also download those files? "100% failur

by veyron 14y ago

"failure to load referenced CSS or image files" -- is there a way for bots (in particular, mechanize in perl/python) to also download those files? "100% failure on Captcha challenges" -- are captchas generally deployed on click-through ads?

mootothemax 14y ago
"failure to load referenced CSS or image files" -- is there a way for bots (in particular, mechanize in perl/python) to also download those files? Very much so, after all your browser does without you having to tell it to do so. It all depends on whether the hassle is worth it for the bot writer.
loumf 14y ago
Of course they could -- all they need to do is parse the HTML, look for the img and link and other tags, and then request the URL. If they were trying to act more like a browser, they would. If they are just trying to click ads, they wouldn't.
vladd 14y ago
> "100% failure on Captcha challenges" -- are captchas generally deployed on click-through ads? Cloudflare has a mode (used in general for DDOS protection) which you can turn on and get stats on how many users pass a Captcha test.
slig 14y ago
> "failure to load referenced CSS or image files" -- is there a way for bots (in particular, mechanize in perl/python) to also download those files? Use http://phantomjs.org/ http://phantomjs.org/ and the page will loaded by webkit.
netfire 14y ago
Using Captcha on an ad landing page to determine whether someone is bot seems like a bad idea. It would almost certainly increase the bounce rate on that page and you couldn't just say that people who didn't choose fill out the captcha were bots.
oelmekki 14y ago
> is there a way for bots (in particular, mechanize in perl/python) to also download those files? The point was mostly that if it is not downloaded, then it's a bot (or any non-browser getter, at least), not that it's not a bot if it does. Or maybe you were asking for a way to avoid bot detection ? :)

by veyron 14y ago

mootothemax 14y ago

"failure to load referenced CSS or image files" -- is there a way for bots (in particular, mechanize in perl/python) to also download those files? Very much so, after all your browser does without you having to tell it to do so. It all depends on whether the hassle is worth it for the bot writer.

loumf 14y ago

Of course they could -- all they need to do is parse the HTML, look for the img and link and other tags, and then request the URL. If they were trying to act more like a browser, they would. If they are just trying to click ads, they wouldn't.

vladd 14y ago

> "100% failure on Captcha challenges" -- are captchas generally deployed on click-through ads? Cloudflare has a mode (used in general for DDOS protection) which you can turn on and get stats on how many users pass a Captcha test.

slig 14y ago

> "failure to load referenced CSS or image files" -- is there a way for bots (in particular, mechanize in perl/python) to also download those files? Use http://phantomjs.org/ http://phantomjs.org/ and the page will loaded by webkit.

netfire 14y ago

Using Captcha on an ad landing page to determine whether someone is bot seems like a bad idea. It would almost certainly increase the bounce rate on that page and you couldn't just say that people who didn't choose fill out the captcha were bots.

oelmekki 14y ago

> is there a way for bots (in particular, mechanize in perl/python) to also download those files? The point was mostly that if it is not downloaded, then it's a bot (or any non-browser getter, at least), not that it's not a bot if it does. Or maybe you were asking for a way to avoid bot detection ? :)