4 ms·

Unfortunately, Perl-WWW::Mechanize does not yet handle Javascript, has anybody (Perl, Python, Ruby) a solution for that? DOM+Javascript simulation comes to my m

by olaf 16y ago

Unfortunately, Perl-WWW::Mechanize does not yet handle Javascript, has anybody (Perl, Python, Ruby) a solution for that? DOM+Javascript simulation comes to my mind. Or maybe a fully controllable browser component (where DOM and JS are built-in) with an API. Anyone with first-hand experience?

paraschopra 16y ago
You can use Java based HTMLUnit. Emulates a headless browser complete with Javascript and DOM.
- kunley 16y ago
  There's awesome JRuby wrapper around it: Celerity I went the route: Mechanize -> HtmlUnit (from inside Clojure to be more scriptable) -> ended with Celerity and most happy.
iamelgringo 16y ago
Have you looked at Selenium? It might have more of what you're looking for.
- hugs 16y ago
  Things like Mechanize and HTMLUnit are what I call "HTTP-only protocol drivers" and are not a true-fidelity simulation of the HTML+JavaScript environment. This is exactly the reason I created Selenium in the first place. I had regression bugs in IE and Mozilla that I needed to verify were fixed in the actual browsers. An emulated browser environment like Mechanize or HTMLUnit isn't a real browser that users actually use, so it doesn't solve the regression testing problem I had. Best of both worlds these days is in driving a headless WebKit (a true browser driver that includes the JS interaction), which will be coming in Selenium 2.
  - olaf 16y ago
    Thanks, Selenium can replay user interaction sessions, but I need to set e.g. form fields programmatically, is that possible? Maybe CPAN/Gtk2::WebKit is also an option.
    - ericb 16y ago
      Yes, it is possible, and fairly routine in Selenium. Also, you can execute arbitrary Javascript for whatever else you may need.
    - aaronblohowiak 16y ago
      you can even use your recorded sessions to spit out the code that you can then use as a template for your code
- olaf 16y ago
  Not yet, thanks for your tip.
- aantix 16y ago
  Yes, along the same lines, you could use Waitr. http://www.layeredthoughts.com/automation/how-to-write-your-first-ruby-web-bot-in-watir-scraping-weather-com http://www.layeredthoughts.com/automation/how-to-write-your-...
jbarnette 16y ago
Aaron Patterson (the maintainer of Ruby's mechanize) and I have been working on this on and off for a few years. Take a look at http://github.com/jbarnette/johnson http://github.com/jbarnette/johnson, http://github.com/mynyml/harmony http://github.com/mynyml/harmony, and http://github.com/mynyml/holygrail http://github.com/mynyml/holygrail to see where things are heading.
- olaf 16y ago
  Thanks, harmony looks interesting, can actions like "set form field", "click button" etc. be programmed?
tomjen3 16y ago
Internet Explorer. No seriously, IE is fully controllable via COM and you can use pretty much any scripting language you want (the python COM api is pretty nice, so is the ruby one). Steve Yeggey had a post on the subject some time ago: http://sites.google.com/site/steveyegge2/scripting-windows-apps http://sites.google.com/site/steveyegge2/scripting-windows-a... BTW, MS Office has a similar feature.

by olaf 16y ago

paraschopra 16y ago

You can use Java based HTMLUnit. Emulates a headless browser complete with Javascript and DOM.

kunley 16y ago
There's awesome JRuby wrapper around it: Celerity I went the route: Mechanize -> HtmlUnit (from inside Clojure to be more scriptable) -> ended with Celerity and most happy.

iamelgringo 16y ago

Have you looked at Selenium? It might have more of what you're looking for.

hugs 16y ago
Things like Mechanize and HTMLUnit are what I call "HTTP-only protocol drivers" and are not a true-fidelity simulation of the HTML+JavaScript environment. This is exactly the reason I created Selenium in the first place. I had regression bugs in IE and Mozilla that I needed to verify were fixed in the actual browsers. An emulated browser environment like Mechanize or HTMLUnit isn't a real browser that users actually use, so it doesn't solve the regression testing problem I had. Best of both worlds these days is in driving a headless WebKit (a true browser driver that includes the JS interaction), which will be coming in Selenium 2.
- olaf 16y ago
  Thanks, Selenium can replay user interaction sessions, but I need to set e.g. form fields programmatically, is that possible? Maybe CPAN/Gtk2::WebKit is also an option.
  - ericb 16y ago
    Yes, it is possible, and fairly routine in Selenium. Also, you can execute arbitrary Javascript for whatever else you may need.
  - aaronblohowiak 16y ago
    you can even use your recorded sessions to spit out the code that you can then use as a template for your code
olaf 16y ago
Not yet, thanks for your tip.
aantix 16y ago
Yes, along the same lines, you could use Waitr. http://www.layeredthoughts.com/automation/how-to-write-your-first-ruby-web-bot-in-watir-scraping-weather-com http://www.layeredthoughts.com/automation/how-to-write-your-...

jbarnette 16y ago

Aaron Patterson (the maintainer of Ruby's mechanize) and I have been working on this on and off for a few years. Take a look at http://github.com/jbarnette/johnson http://github.com/jbarnette/johnson, http://github.com/mynyml/harmony http://github.com/mynyml/harmony, and http://github.com/mynyml/holygrail http://github.com/mynyml/holygrail to see where things are heading.

olaf 16y ago
Thanks, harmony looks interesting, can actions like "set form field", "click button" etc. be programmed?

tomjen3 16y ago

Internet Explorer. No seriously, IE is fully controllable via COM and you can use pretty much any scripting language you want (the python COM api is pretty nice, so is the ruby one). Steve Yeggey had a post on the subject some time ago: http://sites.google.com/site/steveyegge2/scripting-windows-apps http://sites.google.com/site/steveyegge2/scripting-windows-a... BTW, MS Office has a similar feature.