8 ms·
Ask HN: What do you use to test Frontend code?
At our startup, we are trying to sort out the best way to handle testing against our supported Browser matrix. We want something that is repeatable and automated. We've looked into QUnit and Selenium but have this nagging feeling that we are doing it wrong.
Our application is primarily one page and uses a lot of jQuery to both animate things on screen and to work with the backend via AJAX.
Any hard won tips out there?
- simplegeek 15y agoI guess Selenium is fine, out of curiosity, why do you think using Selenium isn't the right choice for you?
- outside1234 15y agoI'm not sure that it isn't. I'm worried about the maintenance costs of keeping it up to date - but maybe that is the price you pay.
- otaku_coder 15y agoIf you're worried about keeping it up to date, why not outsource the test runner to something like saucelabs?
- hopeless 15y agoSauce is a useful service but it doesn't address the test maintenance problem... And it is a big concern. My advice: Do NOT use the Selenium IDE. write your own framework, using Selenium, which isolates changes when they occur. If someone changes the login page, you only need to change a login method in your test suite not all 200 rest cases! Look up the Page Objects pattern as one way of doing this
- notJim 15y agoI believe outside1234 is referring to the maintenance cost of keeping the tests up to date.
- jtheory 15y agoI'm using Selenium on a major project, and from the start refactored out common tasks. I record new tests using the Selenium IDE formerly, now the Sauce Builder plugin, then export to code and replace quite a lot with calls to our own much-simplified API (often cutting out huge sections of test code & replacing with one call). Updating the tests even when we make significant changes to the website is must easier because of this approach. One issue we ran into quickly... tests need to verify just one thing as much as possible, not long strings of actions that wander through the whole site. The problem is that there's considerable setup before testing that one thing -- creating users and getting them to the appropriate state, especially (worst of all when test cases require 3+ users to interact). To address this I've set up some helper requests Selenium can make -- actions that aren't included in our production build -- to create/destroy test users of different types & in different states directly. I'm also thinking of adding a page that'll simply return a full dump of info on a given test user, so we can test that the end result of actions in the UI have resulted in the expected data. It's all very much work in progress, but I'm convinced that with a little extra elbow grease added as pain points show up, we can keep a full suite of Selenium tests alive and useful.
- hiwaylon 15y agoTake a look at JsTestDriver, for hooking in to your CI system.
- mfolnovic 15y agoYou could try with Jasmine, although it's not for this specific purpose... And I agree with simplegeek, selenium should be fine in your case. Although can't imagine how you could test animations... :)
- rchiniquy 15y agoJasmine isn't what you're asking for, but you should use it in addition to what you're asking for.
- dbalatero 15y agoIf you're in the Ruby world, Capybara is a great addon to Cucumber and RSpec. It solves the animation problem by adding a timeout to wait for elements in the DOM. A step like this: When I click the link that does a 4000ms animation I should see "#mydiv" Capybara will wait for #mydiv to appear for some timeout period (30 seconds). If it appears before then, it keeps going. This way you can have complex animation and still assert final state of the page. Capybara allows Selenium, headless WebKit, and other JS drivers. It's one line to swap. If you aren't in the Ruby world, you might at least steal some ideas from that lib to make your Selenium tests better. Also, you might drop the folks at Pivotal Tracker an email. They have a fat one-page JS-heavy product, and I bet you anything they automatically test it. They might have cool stuff to share.
- schrieaj 15y agoCapybara works pretty well for any web based front end. I'm using it to do some basic automation of websites (scheduling tasks and the like). You write the tests in ruby but honestly it is really simple: http://pastie.org/2476405 http://pastie.org/2476405 is my script for logging in to google voice and sending a text message. If I recall it was simply gem install capybara, a gem install selenium-webdriver and that was it.
- dbalatero 15y agoGood point, I forgot that you could just use it standalone!
- pacmon 15y agoCould try watir (http://watir.com/ http://watir.com/)
- tombot 15y agoWatir is what Facebook use I think, it has plugin support for most of the major browsers and tests are fairly easy to write.
- timruffles 15y agoWe use jsTestDriver at Picklive to test our game, alongside Sinon.js for a mock implementation of our game server. It's very easy to test simultaneously on multiple browsers, run tests on the command line, and we run it headless for continuous integration. In comparison with Selenium, I've found unit and integration tests using this setup fast and resilient.
- juanbyrge 15y agozombiejs?
- mpd 15y agoHaving dealt with the Selenium and its frustrations in the past, I tend to outsource this sort of testing if at all possible. Some things can only be truly tested with eyeballs. One tip I do have is to disable jQ animations during your test runs so you aren't having to jump through hoops in your test suite just to wait through animations - jQuery.fx.off = true will do that.
- arbrandes 15y agoThis! Also, try to mock any setTimeouts and setIntervals so that they execute callbacks synchronously (i.e, timeout = 0).
- riffraff 15y agoon a side note, do you people test graphics? I mean, it's kind of easy to say "I get a div #foo". But how do you check if it has the right shade of $color and a rounded corner and the text content is not overflowing the containing box? I live in constant worry of changing css and totally screwing up a completely unrelated bit of the web app, and the usual thing I get told is "well, it's very fast to check if it still works", which reminds me of me not understanding why unit testing is useful.
- socratic 15y agoWhy is unit testing useful, to you? I have often heard two justifications for testing: (a) correctness and (b) developer speed. Phrasing (a) differently, is X functioning as intended? Phrasing (b) differently, how quickly can I make a change to Y, and if I change Y, how long will it take to manually check Y (with a browser, etc.), how likely am I to introduce a bug (leading to extra development time) and/or will I have to rewrite a test? If your primary justification for testing is developer speed rather than correctness, does that impact which features (models, controllers, views with div ids, actual view graphics) you should write tests for?
- jtheory 15y agoI'm putting extra work into unit and UI testing to build a safety net so I can convince my CEO & other developers to let me carry out serious refactorings on our code, like deleting large chunks that aren't used anymore, and changing objects that are used pervasively. We also have an increasingly broad application; as we add new features, I (as a developer) find that I don't visit older parts of the app, though of course the users still need them to work. Particularly when refactoring basic back end stuff that's called from lots of places, it's too tedious to manually track down every part of the UI that would need to be retested due to a given change (and then test it manually... ugh).
- MartinCron 15y agoI live in constant worry of changing css and totally screwing up a completely unrelated bit of the web app I understand this worry, but it makes me think that it may be a design problem with the CSS. Unrelated bits shouldn't be using the same CSS, related bits should probably change together. In a more practical note, I have created selenium tests which capture screenshots and then saved those screenshots as artifacts in the CI (TeamCity) system. Having a human look at a bunch of static images makes for a faster and less error-prone sanity test to see if things are totally broken.
- gudmundur 15y agoCheck out Ghostbuster (https://github.com/joshbuddy/ghostbuster https://github.com/joshbuddy/ghostbuster).
- qixxiq 15y agoI use a custom selenium test runner that runs them in parallel (php). I plan on neatening it up and releasing it at some point if theres interest. It also has a 'live mode' which is useful for writing up new tests. It basically follows a file and whenever there is a change it begins running the commands from that line onwards again. If it hits an exception it'll pause there and wait for you to fix the broken test/site before resuming.
- troels 15y agoI'd be interested in seeing that. Oh, and I'd rather see a mess soon, than wait for the neated-up version.
- jtheory 15y agoI'd be interested as well. I wrote a custom test runner as well, though not addressing the same issues; mine is specifically for non automated testing. It's a Java GUI that lets you choose from all our existing tests, tinker with parameters, and run the test live in your own browser against our test server with a few extras (like stepping through the test, go/pause, run all tests in sequence, etc.). It's partly to let our CEO really interact with our testing -- i.e, see what's being tested, and rapidly explore the site as a whole -- but also is pretty useful as a robot; i.e., when I want to register three new users to try something out, I don't have to do it manually.
- ByteMuse 15y agoSome things require manual testing. Animations and a lot of visual stuff needs to be analyzed holistically; there are sometimes visual quirks that are near impossible to write tests for but can easily be detected from someone quickly viewing the site. Have different environments set up. After every big push, run through a simple user flow to make sure what the user will actually see is what you intended.
- MartinCron 15y agoAfter every big push... One way that I minimize my front-end testing liability is to do as few "big" pushes as I can. The safest change to make is the smallest change possible.
- barnaby 15y agoI use qUnit all the time because I wrote lots of jQuery plugins for everything that we do at my small startup. Then we have a QA team in India that are always clicking through the site and finding new bugs (they also have extensive selenium scripts that go through the whole site every night, but oddly enough none that run the unit tests... grr).
- ltamake 15y agoSelenium works fine for me.
- sgrove 15y agoI always feel frustration at this question, I can't believe that sauce labs hasn't taken off. Use capybara as the driver layer, and a proper test framework that separates the different layers, and sauce labs to hit any browser matrix you want. Tie it into a CI system and never worry about this again - it's a solved problem, and it was solved by Sauce Labs. disclaimer: I used to work there, and put my blood, sweat, and tears into building it out just for situations like this.
- tdavis 15y agoFor Javascript testing, I really like JSTestDriver for automation and Sinon for mocking/stubbing. I found bulky solutions like Selenium to be somewhat hit-or-miss and overkill for most tasks. It's very rare that with proper unit testing I really need a whole additional layer just to click around the site and make sure stuff is working as expected. Maybe some basic sanity checking, but it's really not my preferred way to think about testing--which of course is just a personal preference.
- pmorel 15y agoSame, I feel that if there is a good coverage from unit testing, then you should only need sanity checking. Furthermore, if you are using jquery (which itself is well testing on all the browsers), then you should have more confidence that your javascript will work (providing you're not using non-cross-browser javascript on top of jquery)
- pmorel 15y agoFor reference, the list of tools and framework mentioned on this page can be found at http://www.romku.com/list/343858/web-testing-framework http://www.romku.com/list/343858/web-testing-framework. Feel free to rate them and add any missing information.
- BerislavLopac 15y agoAdmittedly, Selenium's standard interface leaves a lot to be desired; however, I've had great time using it in combination with a Python wrapper, where you write Selenium test cases as Python unit tests.
- zzleeper 15y agoThis. Just write the Selenium base code in the Firefox addon, and then click, export as python :D
- long 15y agoyou might want to check out SIKULI (http://sikuli.org/ http://sikuli.org/) - it's like python on top of a computer vision stack, so you can write code like click([part of a screenshot]).
- jph 15y agoWe use SauceLabs with Selenium, Jellyfish (very fast), Capybara (very flexible), and Jasmine. Yes, Selenium may feel hard to work with at times yet the payoff is enormous when you aim to support many browser for many years. Edit: I have a great outsourcing team that builds Selenium for me, and runs in SauceLabs. PM me if you want details.
- JesseAldridge 15y agoI'm not using full on unit tests for my current project. I just break the code into a few independently functioning layers, and make a single html file to test the code in each layer. I run the corresponding html file manually as I develop code in a given layer. It may be heresy to say this, but I've found that automated unit tests are more trouble than they're worth in a lot of cases. The second thing I do is use my tool every day. I use the unstable branch on the unstable server whenever possible. Problems usually become obvious very quickly. I don't push to production until I've used the unstable server for 24 hours without incident. I find this combined with the testing practices described above to be the best compromise between reliability and getting-stuff-done-ability.
- wlievens 15y agoFor algorithms or application logic, even rather simple cases, unit tests are an absolute must. For user interfaces, it's a lot more effort for a lot less return. YMMV
- douglasmiranda 15y agoHmmm, I'm using Splinter (http://splinter.cobrateam.info/ http://splinter.cobrateam.info/) to do tests. It lets you automate browser actions and much more, like execute javascript, find elements returned via AJAX, interact with forms, links and more... it has drivers for chrome, firefox... You shold to test it :)