5 ms·

Sell me MechanicalSoup over Selenium.

wc- 12y ago
Selenium seems very heavy-weight to me (granted I have only used selenium server). If you don't need to interpret javascript after a page loads then you might be able to use mechanize. In my experience I've gotten better performance and a higher ease of development with mechanize over selenium or other "full" headless browsers. Different tools for different jobs I suppose, I just tend to go for the smallest tool first. edit: replace mechanize with mechanicalsoup in the above paragraph, they are aiming to solve the same problems in the same way.
- dkhar 12y ago
  > Selenium seems very heavy-weight to me Fair point. Perhaps try Sulfur?
  - mhluongo 12y ago
    I started googling, excited about discovering a new library, only to realize what you'd done there -_-
    - mherrmann 12y ago
      Try "Helium web automation" ;-)
  - MrMeker 12y ago
    Selenium and Sulfur both have a certain smell to their code, just rotten. I prefer Oxygen.
- hugs 12y ago
  With Selenium 2, you aren't required to use the selenium-server anymore. You can drive browsers directly from the client lib.
smellf 12y ago
If you don't care what browser is making the requests, use MechanicalSoup. If you do care, use Selenium.
auvrw 12y ago
they're for different use cases. mechanicalsoup is more like zombie.js ( http://zombie.labnotes.org/ http://zombie.labnotes.org/ ) than selenium, chromedriver, or some other webdriver ( https://dvcs.w3.org/hg/webdriver/raw-file/default/webdriver-spec.html https://dvcs.w3.org/hg/webdriver/raw-file/default/webdriver-... ) implementation in that it emulates browser functionality from within a runtime, making http requests and parsing the response directly from the python (or node or ruby or whatever) runtime rather than communicating with a browser that, in turn, makes requests to the website. the advantages of these "emulated" browsers is that tests run faster and are easier to set up. the disadvantage is that they don't fully duplicate browser functionality, particularly for client-side javascript. i think zombie might be able to run some javascript since it's in no, but mechanical soup appears ( https://github.com/hickford/MechanicalSoup/blob/master/mechanicalsoup/browser.py https://github.com/hickford/MechanicalSoup/blob/master/mecha... ) not to execute javascript at all. this is a nice little library that, as the README explains, fills a spot in the python ecosystem that had apparently become somewhat stagnant, but there's really not much to it other than combining Requests with beautifulsoup in order to provide a drop-in replacement for some existing api. i think this would mainly be useful for scraping rather than testing. the emulated browser and custom unittest module that ship with Django are probably better for the latter.
seanp2k2 12y ago
Personally I really like selenium + PhantomJS (headless WebKit browser) since it allows you to do things like automate real user interactions in your tests, then run the test suite on your CI: http://www.realpython.com/blog/python/headless-selenium-testing-with-python-and-phantomjs/ http://www.realpython.com/blog/python/headless-selenium-test... Also interesting if you're going down this road is CasperJS which is basically the same thing for JavaScript, and Velocity ( https://github.com/xolvio/velocity https://github.com/xolvio/velocity ) which is a test runner for Meteor that seems to run all your tests constantly and give you ~real-time feedback for TDD. Lastly, there are many things to tie in Cucumber-style testing specs with Capybara and these other tools if you're into that.

wc- 12y ago

Selenium seems very heavy-weight to me (granted I have only used selenium server). If you don't need to interpret javascript after a page loads then you might be able to use mechanize. In my experience I've gotten better performance and a higher ease of development with mechanize over selenium or other "full" headless browsers. Different tools for different jobs I suppose, I just tend to go for the smallest tool first. edit: replace mechanize with mechanicalsoup in the above paragraph, they are aiming to solve the same problems in the same way.

dkhar 12y ago
> Selenium seems very heavy-weight to me Fair point. Perhaps try Sulfur?
- mhluongo 12y ago
  I started googling, excited about discovering a new library, only to realize what you'd done there -_-
  - mherrmann 12y ago
    Try "Helium web automation" ;-)
- MrMeker 12y ago
  Selenium and Sulfur both have a certain smell to their code, just rotten. I prefer Oxygen.
hugs 12y ago
With Selenium 2, you aren't required to use the selenium-server anymore. You can drive browsers directly from the client lib.

smellf 12y ago

If you don't care what browser is making the requests, use MechanicalSoup. If you do care, use Selenium.

auvrw 12y ago

they're for different use cases. mechanicalsoup is more like zombie.js ( http://zombie.labnotes.org/ http://zombie.labnotes.org/ ) than selenium, chromedriver, or some other webdriver ( https://dvcs.w3.org/hg/webdriver/raw-file/default/webdriver-spec.html https://dvcs.w3.org/hg/webdriver/raw-file/default/webdriver-... ) implementation in that it emulates browser functionality from within a runtime, making http requests and parsing the response directly from the python (or node or ruby or whatever) runtime rather than communicating with a browser that, in turn, makes requests to the website. the advantages of these "emulated" browsers is that tests run faster and are easier to set up. the disadvantage is that they don't fully duplicate browser functionality, particularly for client-side javascript. i think zombie might be able to run some javascript since it's in no, but mechanical soup appears ( https://github.com/hickford/MechanicalSoup/blob/master/mechanicalsoup/browser.py https://github.com/hickford/MechanicalSoup/blob/master/mecha... ) not to execute javascript at all. this is a nice little library that, as the README explains, fills a spot in the python ecosystem that had apparently become somewhat stagnant, but there's really not much to it other than combining Requests with beautifulsoup in order to provide a drop-in replacement for some existing api. i think this would mainly be useful for scraping rather than testing. the emulated browser and custom unittest module that ship with Django are probably better for the latter.

seanp2k2 12y ago

Personally I really like selenium + PhantomJS (headless WebKit browser) since it allows you to do things like automate real user interactions in your tests, then run the test suite on your CI: http://www.realpython.com/blog/python/headless-selenium-testing-with-python-and-phantomjs/ http://www.realpython.com/blog/python/headless-selenium-test... Also interesting if you're going down this road is CasperJS which is basically the same thing for JavaScript, and Velocity ( https://github.com/xolvio/velocity https://github.com/xolvio/velocity ) which is a test runner for Meteor that seems to run all your tests constantly and give you ~real-time feedback for TDD. Lastly, there are many things to tie in Cucumber-style testing specs with Capybara and these other tools if you're into that.