5 ms·

Zombie.js: insanely fast, full-stack, headless testing

74 points by tswicegood 16y ago

EGreg 16y ago
does this thing actually compute the rendered coordinates properly, as a browser would? Can I load jquery and call offset() ?
- a-priori 16y ago
  I would guess no, since it includes only a mock of the DOM. But really, are element locations something you want to test?
  - dools 16y ago
    Definitely! But it would then also require a CSS renderer. Automated testing of something like Decal CMS (decalcms.com) where we provide javascript which manipulates the DOM but allows the designer to plug whatever markup/CSS they like into the system means that automated testing is incredibly difficult and does rely on actually checking for the position (and other computed styles) of an element after we've run certain pieces of code.
  - EGreg 16y ago
    Really, YES. Half of the stuff I want to test is that the right stuff is showing up in the right place. Why, what would you want to test?
- hammerdr 16y ago
  It uses JSDOM (http://jsdom.org/ http://jsdom.org/) under the hood. JSDOM is currently at DOM Level 1 (+additions). It also implements some of Level 2 and Level 3 (https://github.com/tmpvar/jsdom/blob/master/test/testlog.txt https://github.com/tmpvar/jsdom/blob/master/test/testlog.txt).
arnorhs 16y ago
I like the cleanliness of not needing a browser. But it seems like you'll have to duplicate a lot of functionality and as soon as there are a couple of things you can't test against or that don't produce the exact same result then this becomes unreliable. Am I wrong?
- reid 16y ago
  Simulated-browser tests are more like smoke tests. It's an approximation of the environment you're running tests in. Zombie and other simulated-browser systems can be useful because they're faster and convenient, but especially because they're faster. If you're not able to test quickly, like the article says, you lose the flow. Nevertheless, you'll be burned if you don't also test in the browsers you're supporting. davnola, posted above, has the right idea: use Selenium with your CI system in addition to Zombie. Another idea: make real-browser testing for development faster. I'm working on a tool that does just that, if you're curious: https://github.com/yui/yeti https://github.com/yui/yeti http://reid.github.com/decks/2010/yuiconf/yeti.html http://reid.github.com/decks/2010/yuiconf/yeti.html Like it's peers, my project won't work for everyone, yet. But it's awesome to see so many people caring about testing web apps in dev quickly. Hooray! :)
hammerdr 16y ago
Reliable headless DOM implementations are one of major obstacles for JS as a first class language. Kudos and thanks to everyone that is working to solve this!
gojomo 16y ago
Any chance this could run in Rhino or does the node.js dependency demand V8? Feature for feature, how does this compare to HTMLUnit?
croby 16y ago
First, great work on this implementation. I can't imagine all the work that must have gone into this. To me, this seems to only solve a corner case in the main problem of JS testing - testing the DOM without a browser. The first question that comes to mind is, is a DOM even useful in a headless browser when it comes to JS? For now, grant that we're only testing presence and manipulation of DOM elements and no CSS manipulations or accuracy -- which I'd argue is a decent case for testing. Now, testing presence and manipulation of DOM elements in the abstract isn't very useful, as different browsers might treat DOM manipulations differently. Using a headless browser that no one actually uses doesn't solve this problem. I'd see a case for this in running something like QUnit[1] test that just test JS completeness, but again, implementations differ so testing in one environment that no one actually is using isn't good enough. The only thing that I've come up with for a solution for actually testing JavaScript in a robust, real-world way is using QUnit and Selenium[2] to define the tests, with Testswarm[3] actually running the tests. This setup allows unit tests to be run against JS by itself (QUnir), mock interactions to take place to ensure bindings work properly (Selenium), and have these executed against real browsers in a distributed manner (Testswarm). I concede that this isn't an ideal solution, and I wish there were more people working on solutions like Zombie.js that would combine all of these in one so that JS could finally be apart of our continuous integration testing. The only (sic) thing preventing this solution is that it needs to have headless, identical implementations of all the browser's JS engines and DOM parsers in order to be valid. [1] http://docs.jquery.com/Qunit http://docs.jquery.com/Qunit [2] http://seleniumhq.org/ http://seleniumhq.org/ [3] https://github.com/jeresig/testswarm/wiki https://github.com/jeresig/testswarm/wiki
- davnola 16y ago
  There needn't be One Test Suite To Rule Them All. I like having something similar to zombie.js during dev, then Selenium during CI triggered on git push, perhaps with Testswarm scheduled to run daily.
  - eclark 16y ago
    Is test swarm even still up? I can't ever seem to get the website to load in a reasonable amount of time.
bokchoi 16y ago
How does this compare to env.js? http://www.envjs.com/ http://www.envjs.com/
wildmXranat 16y ago
Does this load external styles, or javascript files? What about the images and does it evaluate any script tags? I have done a few albeit scattered attempts at headless or semi-headless html/js unit tests. What I found out to be a good middle ground is running is this: Run selenium server and firefox on a remote server. Trigger tests via a task queue, which at the moment is Gearman, have firefox draw to a virtual frame buffer. What that does for me is cover pretty much every case that a real browser does, and completeness is key in my point of view. I would like to see how Zombie compares on the completeness side and then speed.
deleted 16y ago
[deleted]
pierrefar 16y ago
As I understand it, the JS DOM implementation does not have the quirks of the different browsers. So theoretically, you could write a test that passes and works in browser X but does not work in browser Y. My understanding is that jQuery spends most of its time dealing with these quirks.

74 points by tswicegood 16y ago

EGreg 16y ago

does this thing actually compute the rendered coordinates properly, as a browser would? Can I load jquery and call offset() ?

a-priori 16y ago
I would guess no, since it includes only a mock of the DOM. But really, are element locations something you want to test?
- dools 16y ago
  Definitely! But it would then also require a CSS renderer. Automated testing of something like Decal CMS (decalcms.com) where we provide javascript which manipulates the DOM but allows the designer to plug whatever markup/CSS they like into the system means that automated testing is incredibly difficult and does rely on actually checking for the position (and other computed styles) of an element after we've run certain pieces of code.
- EGreg 16y ago
  Really, YES. Half of the stuff I want to test is that the right stuff is showing up in the right place. Why, what would you want to test?
hammerdr 16y ago
It uses JSDOM (http://jsdom.org/ http://jsdom.org/) under the hood. JSDOM is currently at DOM Level 1 (+additions). It also implements some of Level 2 and Level 3 (https://github.com/tmpvar/jsdom/blob/master/test/testlog.txt https://github.com/tmpvar/jsdom/blob/master/test/testlog.txt).

arnorhs 16y ago

I like the cleanliness of not needing a browser. But it seems like you'll have to duplicate a lot of functionality and as soon as there are a couple of things you can't test against or that don't produce the exact same result then this becomes unreliable. Am I wrong?

reid 16y ago
Simulated-browser tests are more like smoke tests. It's an approximation of the environment you're running tests in. Zombie and other simulated-browser systems can be useful because they're faster and convenient, but especially because they're faster. If you're not able to test quickly, like the article says, you lose the flow. Nevertheless, you'll be burned if you don't also test in the browsers you're supporting. davnola, posted above, has the right idea: use Selenium with your CI system in addition to Zombie. Another idea: make real-browser testing for development faster. I'm working on a tool that does just that, if you're curious: https://github.com/yui/yeti https://github.com/yui/yeti http://reid.github.com/decks/2010/yuiconf/yeti.html http://reid.github.com/decks/2010/yuiconf/yeti.html Like it's peers, my project won't work for everyone, yet. But it's awesome to see so many people caring about testing web apps in dev quickly. Hooray! :)

hammerdr 16y ago

Reliable headless DOM implementations are one of major obstacles for JS as a first class language. Kudos and thanks to everyone that is working to solve this!

gojomo 16y ago

Any chance this could run in Rhino or does the node.js dependency demand V8? Feature for feature, how does this compare to HTMLUnit?

croby 16y ago

First, great work on this implementation. I can't imagine all the work that must have gone into this. To me, this seems to only solve a corner case in the main problem of JS testing - testing the DOM without a browser. The first question that comes to mind is, is a DOM even useful in a headless browser when it comes to JS? For now, grant that we're only testing presence and manipulation of DOM elements and no CSS manipulations or accuracy -- which I'd argue is a decent case for testing. Now, testing presence and manipulation of DOM elements in the abstract isn't very useful, as different browsers might treat DOM manipulations differently. Using a headless browser that no one actually uses doesn't solve this problem. I'd see a case for this in running something like QUnit[1] test that just test JS completeness, but again, implementations differ so testing in one environment that no one actually is using isn't good enough. The only thing that I've come up with for a solution for actually testing JavaScript in a robust, real-world way is using QUnit and Selenium[2] to define the tests, with Testswarm[3] actually running the tests. This setup allows unit tests to be run against JS by itself (QUnir), mock interactions to take place to ensure bindings work properly (Selenium), and have these executed against real browsers in a distributed manner (Testswarm). I concede that this isn't an ideal solution, and I wish there were more people working on solutions like Zombie.js that would combine all of these in one so that JS could finally be apart of our continuous integration testing. The only (sic) thing preventing this solution is that it needs to have headless, identical implementations of all the browser's JS engines and DOM parsers in order to be valid. [1] http://docs.jquery.com/Qunit http://docs.jquery.com/Qunit [2] http://seleniumhq.org/ http://seleniumhq.org/ [3] https://github.com/jeresig/testswarm/wiki https://github.com/jeresig/testswarm/wiki

davnola 16y ago
There needn't be One Test Suite To Rule Them All. I like having something similar to zombie.js during dev, then Selenium during CI triggered on git push, perhaps with Testswarm scheduled to run daily.
- eclark 16y ago
  Is test swarm even still up? I can't ever seem to get the website to load in a reasonable amount of time.

bokchoi 16y ago

How does this compare to env.js? http://www.envjs.com/ http://www.envjs.com/

wildmXranat 16y ago

Does this load external styles, or javascript files? What about the images and does it evaluate any script tags? I have done a few albeit scattered attempts at headless or semi-headless html/js unit tests. What I found out to be a good middle ground is running is this: Run selenium server and firefox on a remote server. Trigger tests via a task queue, which at the moment is Gearman, have firefox draw to a virtual frame buffer. What that does for me is cover pretty much every case that a real browser does, and completeness is key in my point of view. I would like to see how Zombie compares on the completeness side and then speed.

deleted 16y ago

[deleted]

pierrefar 16y ago

As I understand it, the JS DOM implementation does not have the quirks of the different browsers. So theoretically, you could write a test that passes and works in browser X but does not work in browser Y. My understanding is that jQuery spends most of its time dealing with these quirks.