5 ms·

"I have JS and cookies disabled" So imagine what bots often don't have. Adding JS interaction and cookies takes more effort on the part of the programmer writ

by nathas 12y ago

"I have JS and cookies disabled" So imagine what bots often don't have. Adding JS interaction and cookies takes more effort on the part of the programmer writing a bot. So yeah, you'd look a lot more like a robot. How else would you quickly differentiate between human vs non-human based on a single request, or even a collection of requests over time? It's a game of stats at scale.

fragmede 12y ago
Here's a snippet of Python using the splinter library, to visit Google, type in a search query, and click 'search' (which is very Javascript heavy these days with their annoying 'instant' search). from splinter import Browser b = Browser() b.visit('http://google.com' http://google.com') b.fill('q', 'browser automation') btn = b.find_by_name('btnG') btn.click() Not exactly 'more effort'...
- xanderstrike 12y ago
  With Selenium you can open a full web browser such as Chrome or Firefox and have it run through. A Google search is six lines: require "selenium-webdriver" driver = Selenium::WebDriver.for :firefox driver.navigate.to "http://google.com" http://google.com" element = driver.find_element(:name, 'q') element.send_keys "Hello WebDriver!" element.submit https://code.google.com/p/selenium/wiki/RubyBindings https://code.google.com/p/selenium/wiki/RubyBindings Writing a bot with js and cookies is trivial, but it definitely won't defeat these tools. They probably look for time between actions or track mouse movements, stuff that makes the bots super inefficient.
- heinrich5991 12y ago
  It's more effort as less libraries support it, you need to execute unknown code on your computer, etc.
- tracker1 12y ago
  Yeah, but if you are trying to automate thousands of simultaneous requests, you'll have to use a lot of servers, which is costly even in the cloud. Right now google and bing will run sites with JS enabled to see the DOM after any JS changes take hold. Usually these crawls aren't nearly as often as the general crawling, because there is quite a lot more CPU/Memory overhead to such utilities. I can't speak for splinter, but similar tools in node or phantomjs have a lot over overhead to them.
  - takeda 12y ago
    Still less effort than typical captcha.
- Hoffmannnn 12y ago
  Seriously? https://www.google.com/search?q=PROGRAMATICALLY_REPLACE_ME https://www.google.com/search?q=PROGRAMATICALLY_REPLACE_ME
  - andrewguenther 12y ago
    That wasn't the point

by nathas 12y ago

fragmede 12y ago

Here's a snippet of Python using the splinter library, to visit Google, type in a search query, and click 'search' (which is very Javascript heavy these days with their annoying 'instant' search). from splinter import Browser b = Browser() b.visit('http://google.com' http://google.com') b.fill('q', 'browser automation') btn = b.find_by_name('btnG') btn.click() Not exactly 'more effort'...

xanderstrike 12y ago
With Selenium you can open a full web browser such as Chrome or Firefox and have it run through. A Google search is six lines: require "selenium-webdriver" driver = Selenium::WebDriver.for :firefox driver.navigate.to "http://google.com" http://google.com" element = driver.find_element(:name, 'q') element.send_keys "Hello WebDriver!" element.submit https://code.google.com/p/selenium/wiki/RubyBindings https://code.google.com/p/selenium/wiki/RubyBindings Writing a bot with js and cookies is trivial, but it definitely won't defeat these tools. They probably look for time between actions or track mouse movements, stuff that makes the bots super inefficient.
heinrich5991 12y ago
It's more effort as less libraries support it, you need to execute unknown code on your computer, etc.
tracker1 12y ago
Yeah, but if you are trying to automate thousands of simultaneous requests, you'll have to use a lot of servers, which is costly even in the cloud. Right now google and bing will run sites with JS enabled to see the DOM after any JS changes take hold. Usually these crawls aren't nearly as often as the general crawling, because there is quite a lot more CPU/Memory overhead to such utilities. I can't speak for splinter, but similar tools in node or phantomjs have a lot over overhead to them.
- takeda 12y ago
  Still less effort than typical captcha.
Hoffmannnn 12y ago
Seriously? https://www.google.com/search?q=PROGRAMATICALLY_REPLACE_ME https://www.google.com/search?q=PROGRAMATICALLY_REPLACE_ME
- andrewguenther 12y ago
  That wasn't the point