11 ms·

How we took our tests from 100 seconds to 100ms

203 points by kevinburke 11y ago

msoad 11y ago
In my experience unit tests don't catch many bugs. On other hands end-to-end tests always surprise me with the bugs they catch. Based on that, I'm not sure advising for more stubbing is a good advise. Maybe I'm wrong but that was my experience building UI apps.
- DannoHung 11y ago
  In my experience building backend apps, unit tests are way more effective at catching logic bugs than end to end tests. End to end tests are great for catching connectivity bugs though.
  - reverius42 11y ago
    UIs tend to have a ton of connectivity and little logic/computation.
  - michaelmcmillan 11y ago
    An added benefit of using unit tests over integration tests is that unit tests shows you what went wrong very clearly. If an integration test fails I have literally no idea where to start hunting for the bug. It could be located at any level in the stack. I highly recommend this brilliant talk by J.B Rainsberger on how integration tests are a scam: https://vimeo.com/80533536 https://vimeo.com/80533536
    - specialk 11y ago
      If you don't have that much time to write tests I'd recommend integration tests. Then at least you have some tests. Treat integration tests as smoke tests. If there is smoke then you can start looking for the fire. Integration tests are always better then no tests at all. At least they tell you there is a fire even if it is vague in telling you where it is.
      - michaelmcmillan 11y ago
        As someone who always writes unit tests before any production code, I don't know how I would have the time to not write tests. It drives the design and reduces time debugging by a ridiculous amount. I work with information systems though, might be different for other fields.
        lmm 11y ago
        Since I've switched to a language with a richer type system (Scala) I've found a lot of what I used to do with unit tests can go into the type level instead (see e.g. http://spin.atomicobject.com/2014/12/09/typed-language-tdd-part1/ http://spin.atomicobject.com/2014/12/09/typed-language-tdd-p... - but in Scala the techniques are a lot less verbose to apply).
        michaelmcmillan 11y ago
        I am most familiar with Javascript & Python, but I think you're onto something having fiddled with Java for 6 months or so. Some logic obviously can't be replaced with only a rich type system, but a lot can.
- noselasd 11y ago
  Though you also have to ask how many additional bugs you'd have if there were no unit tests.
- juliangregorian 11y ago
  Doesn't sound like you're unit testing well then. I'd take good unit tests over integration tests any day. Integration tests are inherently flaky.
  - kansface 11y ago
    Your sentiment is strange to me - I've only ever seen unit tests as absolutely worthless. The reason integration tests are flaky is exactly the same reason they are valuable.
    - hinkley 11y ago
      Someone who checks in broken code will also take flaky tests as an excuse not to look at the build results, or to convince themselves that the failure is caused by flakiness and not their changes. If I had a dollar for every time I asked someone why they weren't fixing their shit, and they answered "Oh, I thought the build was failing randomly again." I'd have a lot of dollars.
      - phpnode 11y ago
        The fact that your tests are flaky at all means that the integration tests have revealed a bug that you need to fix. The tests have done their job. The fact that people aren't looking at the results speaks to a larger cultural issue.
        hinkley 11y ago
        Not if it's a timing error in the test harness. That's usually what people mean when they say flaky tests. Selenium, especially three or four years ago, would simply fail occasionally no matter how good your tests are. Yes, there are bugs that look like bad luck, but are really flaws in your logic. I have fixed a lot of bugs that look Random, but eventually you're left with, for instance, selenium just refusing to click a button.
  - sheepmullet 11y ago
- tunesmith 11y ago
  For me, the value of writing unit tests is in the writing of them more than the running of them.
  - amyjess 11y ago
    Alternately, the reading of them as well. Reading the unit tests can help you understand an unfamiliar project by acting as documentation, telling you what is expected behavior.
    - curun1r 11y ago
      Yeah...for me, unit tests are the gift that you give to the developer who will be maintaining and improving the code 6 months from the time you write it. Beyond documentation, it gives him/her the ability to refactor the code without fear of it breaking horribly and tells that developer which behavior of the application is intentional and which was accidental. And before you say, "Screw 'em...I've got a code freeze to meet," consider that the person maintaining the code 6 months from now is probably you.
      - madeofpalk 11y ago
        IMO, this all relies on having good unit tests. If you have bad application code that's hard to read, what's to say that the tests will be any better?
        tunesmith 11y ago
        How would you have good unit tests if you have bad application code? It's writing the tests that forces you to improve the application code, because otherwise you won't find it straightforward to write the tests.
  - deleted 11y ago
    [deleted]
- EugeneOZ 11y ago
  Try to add fuzz-testing into your unit tests.
- bliti 11y ago
  That's why you don't stop with unit tests. You make sure to automate testing of ever layer of the application. It does require a bit more effort, but it's it's a savior when the UI breaks, and it's not something obvious yet breaks the user experience.
  - jononor 11y ago
    I prefer to write end-to-end tests first, then fill in integration and unit-tests on a as-needed basis. If I am finding it complicated to write, or if experience from production says it is a problematic area. After all, the important thing is whether your application/service/library does what it promises to its users/consumers. Whether some function/class deep inside works or not is mostly irrelevant (can help in fixing the issue faster though). Unit-testing-first has a tendency to make the tests very fine-grained, often need changes when you refactor internals (how do you know test did not break?), and needlessly keeping rigid internal details that one should have the freedom to change. Have seen plenty of code being kept around because it had lots of tests - when it was not actually used for anything at all! Note: Above approach is coupled with a pretty strict adherence to keeping projects small and independent, preferably <10kLOC.
- Confusion 11y ago
  The distribution of when bugs are caught, and what bugs are caught, by different kinds of tests is very different. Units tests catch bugs while they are being written and when the units they test are being refactored. The bugs they catch are trivial, silly, 'would have been caught anyway'. Outside of those activities, they rarely catch bugs. On the other hand, integration tests more consistently catch bugs and catch bugs you didn't foresee, have to investigate before understanding. You don't remember the last time a unit test caught a bug, because on average it is much longer ago and if it was a test for a unit you wrote this morning, you usually blame it on 'work in progress' and think you would have caught that trivial bug anyway. I think the nature of units tests make it easy to underappreciate them.
  - Glide 11y ago
    This is too true. A team I was on started seeing way more value in integration tests and ended up writing way more integration tests than unit tests. What ended up happening is that the code quality ended up worse because they stopped running tests very often because there were too many integration tests and they took too long to run. Sad part is, a lot of those tests would have made a lot more sense as unit tests.
- LoSboccacc 11y ago
  my experience is like yours that unit test developed in development are rarely useful. however, having a framework in place helps a lot when working at user submitted issues: those are the issue one should strive to replicate trough unit test. In a six month of production, you get a fairly robust suites of test on all corner cases undiscovered in development.
- ed_blackburn 11y ago
  Admittedly not in node, but I've found that writing end-to-end tests but substituting different 'units' for in-memory or not depending on the test marries the best of both worlds. In-memory tests tend to focus on behaviour. In the rush to shoe-horn tests into a taxonomy, or 'integration' vs 'acceptances' vs 'unit' we often loose sight of the actual value proposition. Assisting design and building confidence to make changes.
- taco_emoji 11y ago
  You can use both things...
- danieltillett 11y ago
  It depends a bit on the problem you are trying to solve and the structure of your code. I have projects where only end-to-end tests find bugs and others where unit tests are helpful. What is never helpful is imposing some sort of semi-religious "thou shalt do x" in every case.
- josteink 11y ago
  > In my experience unit tests don't catch many bugs. On other hands end-to-end tests always surprise me with the bugs they catch. Obviously if you've walked the walk with unit-tests and done proper TDD and all that, you've caught the bugs before submitting your code. Ofcourse the remaining bugs (which there will be plenty of!) will be found in your end-to-end testing where all the bits are tied together. This is only natural and does is no way suggest unit-testing was a wasted effort.
  - sheepmullet 11y ago
    "Obviously if you've walked the walk with unit-tests and done proper TDD and all that, you've caught the bugs before submitting your code." Unit testing and TDD are quite separate things. Perhaps you should try writing unit tests after you write your code and then you can come back and add value to the conversation.
    - josteink 11y ago
      > Perhaps you should try writing unit tests after you write your code and then you can come back and add value to the conversation. Nice snark there. Feel free to keep it to yourself. To add actual value to the conversation (as opposed to your contribution), I can very much recommend the book "Working Effectively with Legacy Code"[1] for how to handle unit-testing in the scenario of existing "legacy" code-bases. It's full of useful tips and methods to get testing in place "anywhere" and has a pragmatical (as opposed religious) approach to getting it done. To spark some interest: The book defines "legacy code" as any code not covered by unit-tests. It may be seem dated (from 2004 and all), but it's been the most useful book I've read on unit-testing by far. [1] http://www.amazon.com/gp/product/0131177052/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0131177052&linkCode=as2&tag=jostekjoni-20&linkId=XZ6RUGMKOUPE4HTQ http://www.amazon.com/gp/product/0131177052/ref=as_li_tl?ie=...
      - sheepmullet 11y ago
        You came along and dismissed the parents views without any real thought. If you write the tests afterwards you can better gauge how well they catch bugs. It sounds like the parent has tried this and has found them pretty useless. I've personally found that unit tests are not worthwhile for many components, and yet are critical for others.
zobzu 11y ago
i think it usually boils down to "its tests, its not performance sensitive" then tests take 10x the build time and eventually impact productivity like crazy :(
- tajen 11y ago
  At my startup (35 devs on the main product), we'd spin off a dozen AWS instances so the tests would run in 20 minutes in parallel. And only tested commits could be merged into master. We didn't feel it as a bad situation.
  - nulltype 11y ago
    It seems like renting computers by the minute is pretty cheap these days. I'm surprised more people don't just have one machine per test, run them all in parallel. Testing seems to be embarrassingly parallel in that usually your tests don't depend on other tests at all.
    - zobzu 11y ago
      most of the problem is when tests depend on each other, which they oftentimes do or when they shouldnt but do, theres a large amount of work to change that
    - tajen 11y ago
      Unit tests are short, but integration tests require the app to start and setup, which takes around 3 minutes. Then each test is quite independent.
Asparagirl 11y ago
Wait, Shyp uses Sails.js for their framework? Huh, they might be the most well-known startup to use it. Sails development has really slowed down lately, especially since one of their core creators/commiters got into YC with a different project; I wonder how Shyp feels about that... "Avoiding Sails isn’t possible in every situation, but we’ve found it’s a useful design principle - it helps us separate our business logic and our data storage layer, and will make it easier for us to move off Sails in the future." Ooops.
- angersock 11y ago
  Well, to be fair, one of the patterns I've seen with Rails projects is to move as much out of Rails proper as possible, and treat it just as your web/(maybe)database interface.
- kevinburke 11y ago
  Yep - we've run into a lot of really unexpected behavior with it and with the ORM (Waterline), so we try to restrict ourselves to "the minimum possible framework subset that gets the code to run". For example, we disable `dynamicFinders`, don't use policies, don't use blueprints, don't use Waterline's associate/populate/destroy, don't use the "turn any controller function into a named route" setting, don't use globals, disabled all of the Grunt/websockets junk, and overwrote all of the error response handlers. The result is that day to day we're writing something that resembles an Express app with its own `routes` file. We're looking to move off of Waterline/Sails entirely, it's slow going though since we're a small team and it's tough to prioritize that, as it "works" now. Every company has parts of their tech stack that don't do a great job - we're somewhat fortunate this one is easy to mitigate.
  - iLoch 11y ago
    Sounds like you guys pulled out everything except for Express. I too found the more I used Sails the less useful it became, as though it was missing a solution for anything lying outside of the very basics. Compare Sails to say, Laravel, and you begin to see how much work needs to be put in to bring it up to date.
shanemhansen 11y ago
So basically their biggest problem was that node.js machine-gun hammers the filesystem with stat calls when looking for modules. This was made worse by running on a shared folder. Vagrant has a mode called rsync-auto which will synchronize directories into a local vm. Or you could just always have the code local to the node process. It makes more sense to have your editor work with remote files. Next biggest problem was relational database. I think that tests requiring a SQL database are usually a bad sign, and the statement "we'd like to use transactions but our orm doesn't support them" makes me sad. To paraphrase Ben Franklin, "those who would give up transactions for a framework deserve neither". Here's some ideas for speeding up your tests, in order from best to worst. - Make most business code rely on a service or interface. Swap that our during tests for a trivial in-memory implementation. - Use sqlite :memory: for tests - TRUNCATE TABLE - Set your database to running with scissors mode (disable fsync) [edit] I just took a look at the ORM used, waterline https://github.com/balderdashy/waterline https://github.com/balderdashy/waterline . Why wasn't step one moving to sails-memory for running tests?
- pyre 11y ago
  > Use sqlite :memory: for tests Then you run into issues where the functionality works on SQLite, but not on your real database.
  - bliti 11y ago
    May not work. Given the use of ORMs these days, this is a smaller issue. Though it still requires attention. Having the tests run quicker allows for better use of the programmers time
    - pyre 11y ago
      > Given the use of ORMs these days, this is a smaller issue The ORM can't abstract away all differences between databases. For example, SQLite doesn't have Numeric type, so all of the Numeric(10, 2) fields on (e.g.) my SQLAlchemy (a Python ORM) models will trigger warnings when saving to a SQLite database.
steveklabnik 11y ago
This post is full of great general advice that's not specific to Node. In Ruby-land, we've been talking about the same things for the past few years, complete with 'omg how many stat calls are we making again?' I'm sure others were doing the same before Ruby too. The wheel keeps turning...
- shanemhansen 11y ago
  I worked on a perl application server for a big ecommerce company. We'd stat so many files. Pulling in the meta-object library ('use Moose') was guaranteed to make your program take a long time to start.
- why-el 11y ago
  Any link to a discussion on these Ruby stat calls? We might look into them as well.
  - steveklabnik 11y ago
    The core issue is that Rubygems adds the `lib` directory of every gem to your path, so you end up calling stat a zillion times. I am admittedly a bit out of the Ruby loop these days, so it's possible that this has been improved. The fastest way of course is to not load a zillion gems...
jkot 11y ago
Nice problem to have. I work on database engine, and full test suite takes about 2 weeks. After adding concurrency test, it might take a few months. I started migrating to AWS spot instances to make this whole monster manageable.
- shanemhansen 11y ago
  This is somewhat more reasonable. If your core business logic encompasses flushing data to lying disk controllers and being robust in the face of kill -9's, the root filesystem filling up, not corrupting the database on segfault, and not falling over on multi-terabyte datasets then your tests might take a long time.
- Bluestrike2 11y ago
  I just about choked on my coffee. If you're serious, is the issue just the size of the test suite, or are the tests themselves just woefully inefficient? By the time you're done, you've forgotten whatever it was that you modified.
  - masklinn 11y ago
    I expect they have subsets of the suites working as "fast tests". The full sqlite test suite is 3 orders of magnitude bigger than sqlite itself and takes hours on beefy machines, but they're not run "online" by developers. Developers generally run the "veryquick" suite which takes a few minutes (and is sufficient to catch the vast majority of issues)
  - Sanddancer 11y ago
    Database test suites tend to be on the ridiculously thorough side, for good reason. For example, the SQLite test suite ( https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html ) all but beats the code with a sledgehammer trying to break out bugs. Some software can't be upgraded once deployed, so it only makes sense to ensure that it is well-tested.
throwaway999666 11y ago
If it so happens that people who are having problems with the running time of tests have something like on the order of as much test code as they have "actual" code, I'm not surprised that they have to optimize the test suite. Apparently that was a trend some time ago (I don't know if it still is).
rdsubhas 11y ago
While bringing down test runtime 1000x is certainly a nice feat, I don't think this is setting a right precedent. I, for one, would be very afraid. Tests are not supposed to be written with speed in mind. Metrics seem to dominate the world of tests, where the spirit is supposed to be catching real world bugs. For example, the following can be done without destroying the spirit of testing: * Stub the sign in (big savings) * Keep a tab on fixtures (big savings) * Parallellize (big savings) * Buy a better machine (big savings) * Truncate with transactions/container-snapshots/etc (medium savings) * Tune database for speed than safety (medium savings) * Tune GC (medium savings), with enough RAM (16GB) disable it entirely (big savings) * Disable logging and instrumentation, yes including timing measurements, unless intentionally needed (small to medium savings) * Mount entire codebase, database and resources in tmpfs (small savings) The below, while they might improve performance, go against the spirit: * Tautological tests (http://fabiopereira.me/blog/2010/05/27/ttdd-tautological-test-driven-development-anti-pattern/ http://fabiopereira.me/blog/2010/05/27/ttdd-tautological-tes...) * Too many mocks/metaprogramming * Using a different database driver I'm planning to write a tool, which given a Class, will pump out lines and lines of tautological tests, line for line (a.a should == a.a), method for method (a.should receive save, a.save), branch for branch (with some metaprogramming). Coverage will report a cool 100%, tests will be blazing fast, and that should calm the lead devs down.
- maccard 11y ago
  > Tests are not supposed to be written with speed in mind This is why the tests took 100 seconds in the first place. While I agree that speed isn't the number one priority for everything, Reducing the length of time your entire test suite takes to run to be doable on every single run of your framework is absolutely worthwhile.
- josteink 11y ago
  > Tests are not supposed to be written with speed in mind. There's always extremes. If people write tests with no regard for speed (as I've seen) you can have "unit-tests" run for 45 minutes before completing. That might be OK to guard against regressions on a nightly-build, but it will in no way be a test-suite I'm willing to running prior to every commit I ship. If the test-suite runs in less than a minute, I'm much more willing to do so before submitting code, and in that way avoiding regressions which needs to be fixed later on. If the tests runs in less than 10 seconds, I'm probably going to run them for every major change I do, even if I'm not planning on submitting anything yet, just for peace of mind. There's IDEs where tests can be run automatically in the background as you make changes and let you know the second you broke something. For that to work efficiently, your tests has to be sufficiently fast. So maybe the aim shouldn't be that tests must be fast above everything else. But if the test-suite can be run quickly and casually, it's value will increase tremendously.
wilgertvelinga 11y ago
Wallaby.js is another way of speeding up tests during development.
wilgertvelinga 11y ago
For faster tests during development use wallaby.js!
dbbolton 11y ago
What vim colorscheme is that in the screencaps?
- kevinburke 11y ago
  hybrid_material - here: https://bitbucket.org/kevinburke/small-dotfiles/src/e4e1baf907ae103bedf445e022a795b57d9da9be/.vim/gvimrc?at=master#gvimrc-6 https://bitbucket.org/kevinburke/small-dotfiles/src/e4e1baf9... and here https://bitbucket.org/kevinburke/small-dotfiles/src/e4e1baf907ae103bedf445e022a795b57d9da9be/.gitmodules?at=master#.gitmodules-130 https://bitbucket.org/kevinburke/small-dotfiles/src/e4e1baf9...
  - dbbolton 11y ago
    Thanks.
andresmanz 11y ago
I don't know what it is with this site, but it made Chrome (Android) crash four times already. Which means that, thanks to that wonderful browser, my two billion research tabs are gone since the first crash. Anyway, I can't imagine how running unit tests could possibly reach 100 seconds. Usually, that means that the tests are coupled to the environment (database, filesystem, ...). So without being able to read the article, that would be my guess. There are tons of good (and old) reasons why you should decouple unit tests from the environment as much as possible. Execution time is only one of them.
- kevinburke 11y ago
  Sorry to hear that Andres - what version of Chrome are you using on Android? Will try and see if we can figure out what is going wrong
  - andresmanz 11y ago
    It's version 43.0.2357.93. I don't think it's your fault, though. Chrome loves crashing and throwing away all of my research tabs. That wasn't the first time. It's only the first time I could reproduce it with a specific website. Still, as I said, I don't think it's your fault. I'll just finish reading the article when I'm back at my PC. Test optimization is a great thing.
jgalt212 11y ago
> We run tests 400-500 times a day; a ten second improvement in every test run translates to an hour of saved developer time, every day. Is it just me, or does that seem like running a lot of tests per day. I mean, how many lines of code do they write per day? 1000? Do they run the test suite ever time they write two lines of code?
- Fargren 11y ago
  If your suite run in 100ms, why wouldn't you?
  - Gigablah 11y ago
    The thing is, after reading all that, I think they're referring to their test suite startup time.
  - jgalt212 11y ago
    because then I think you become to reliant upon tests, and not reliant enough upon correctly understanding the problem at hand. furthermore, your code is then only as good as your tests. And thus your code become vulnerable to failing on anything outside the test cases.
- gtramont 11y ago
  That is probably on a CI server. During development, I'd probably be running way more than that, at every production or test line written, having a task that watches for file changes and re-runs all tests. Then, having a fast test suite is key. While speeding up the process is always good, tools like lucifer or spork are only masking the problem: loading the whole environment for unit-level tests. Sails, like Rails, promote tight coupling between all parts of the system, which harms test speed, thus harming another development tool: TDD. The more decoupled and isolated you write your business logic, the easier and faster it is to test drive it. Then you simply plug an HTTP interface (with routes and whatnot) that calls into your core.
henvic 11y ago
Some npm requires are indeed painfully slow. For this reason I have wrote a small, but useful tool called require-time. https://www.npmjs.com/package/require-time https://www.npmjs.com/package/require-time https://asciinema.org/a/21275 https://asciinema.org/a/21275
- srn_ 11y ago
  There's also this: https://www.npmjs.com/package/require-times https://www.npmjs.com/package/require-times
nraynaud 11y ago
A decade ago I brought the test time from 86min down to a few seconds. An intern had thought that synchronizing things with "sleep one tenth of a second then check that flag again" at the heart of the product and reading and splitting by line a file where some lines might be 86Mo long were good ideas.
supertruth 11y ago
lucifer, scary

203 points by kevinburke 11y ago

msoad 11y ago

In my experience unit tests don't catch many bugs. On other hands end-to-end tests always surprise me with the bugs they catch. Based on that, I'm not sure advising for more stubbing is a good advise. Maybe I'm wrong but that was my experience building UI apps.

DannoHung 11y ago
In my experience building backend apps, unit tests are way more effective at catching logic bugs than end to end tests. End to end tests are great for catching connectivity bugs though.
- reverius42 11y ago
  UIs tend to have a ton of connectivity and little logic/computation.
- michaelmcmillan 11y ago
  An added benefit of using unit tests over integration tests is that unit tests shows you what went wrong very clearly. If an integration test fails I have literally no idea where to start hunting for the bug. It could be located at any level in the stack. I highly recommend this brilliant talk by J.B Rainsberger on how integration tests are a scam: https://vimeo.com/80533536 https://vimeo.com/80533536
  - specialk 11y ago
    If you don't have that much time to write tests I'd recommend integration tests. Then at least you have some tests. Treat integration tests as smoke tests. If there is smoke then you can start looking for the fire. Integration tests are always better then no tests at all. At least they tell you there is a fire even if it is vague in telling you where it is.
    - michaelmcmillan 11y ago
      As someone who always writes unit tests before any production code, I don't know how I would have the time to not write tests. It drives the design and reduces time debugging by a ridiculous amount. I work with information systems though, might be different for other fields.
      - lmm 11y ago
        Since I've switched to a language with a richer type system (Scala) I've found a lot of what I used to do with unit tests can go into the type level instead (see e.g. http://spin.atomicobject.com/2014/12/09/typed-language-tdd-part1/ http://spin.atomicobject.com/2014/12/09/typed-language-tdd-p... - but in Scala the techniques are a lot less verbose to apply).
        michaelmcmillan 11y ago
        I am most familiar with Javascript & Python, but I think you're onto something having fiddled with Java for 6 months or so. Some logic obviously can't be replaced with only a rich type system, but a lot can.
noselasd 11y ago
Though you also have to ask how many additional bugs you'd have if there were no unit tests.
juliangregorian 11y ago
Doesn't sound like you're unit testing well then. I'd take good unit tests over integration tests any day. Integration tests are inherently flaky.
- kansface 11y ago
  Your sentiment is strange to me - I've only ever seen unit tests as absolutely worthless. The reason integration tests are flaky is exactly the same reason they are valuable.
  - hinkley 11y ago
    Someone who checks in broken code will also take flaky tests as an excuse not to look at the build results, or to convince themselves that the failure is caused by flakiness and not their changes. If I had a dollar for every time I asked someone why they weren't fixing their shit, and they answered "Oh, I thought the build was failing randomly again." I'd have a lot of dollars.
    - phpnode 11y ago
      The fact that your tests are flaky at all means that the integration tests have revealed a bug that you need to fix. The tests have done their job. The fact that people aren't looking at the results speaks to a larger cultural issue.
      - hinkley 11y ago
        Not if it's a timing error in the test harness. That's usually what people mean when they say flaky tests. Selenium, especially three or four years ago, would simply fail occasionally no matter how good your tests are. Yes, there are bugs that look like bad luck, but are really flaws in your logic. I have fixed a lot of bugs that look Random, but eventually you're left with, for instance, selenium just refusing to click a button.
- sheepmullet 11y ago
tunesmith 11y ago
For me, the value of writing unit tests is in the writing of them more than the running of them.
- amyjess 11y ago
  Alternately, the reading of them as well. Reading the unit tests can help you understand an unfamiliar project by acting as documentation, telling you what is expected behavior.
  - curun1r 11y ago
    Yeah...for me, unit tests are the gift that you give to the developer who will be maintaining and improving the code 6 months from the time you write it. Beyond documentation, it gives him/her the ability to refactor the code without fear of it breaking horribly and tells that developer which behavior of the application is intentional and which was accidental. And before you say, "Screw 'em...I've got a code freeze to meet," consider that the person maintaining the code 6 months from now is probably you.
    - madeofpalk 11y ago
      IMO, this all relies on having good unit tests. If you have bad application code that's hard to read, what's to say that the tests will be any better?
      - tunesmith 11y ago
        How would you have good unit tests if you have bad application code? It's writing the tests that forces you to improve the application code, because otherwise you won't find it straightforward to write the tests.
- deleted 11y ago
  [deleted]
EugeneOZ 11y ago
Try to add fuzz-testing into your unit tests.
bliti 11y ago
That's why you don't stop with unit tests. You make sure to automate testing of ever layer of the application. It does require a bit more effort, but it's it's a savior when the UI breaks, and it's not something obvious yet breaks the user experience.
- jononor 11y ago
  I prefer to write end-to-end tests first, then fill in integration and unit-tests on a as-needed basis. If I am finding it complicated to write, or if experience from production says it is a problematic area. After all, the important thing is whether your application/service/library does what it promises to its users/consumers. Whether some function/class deep inside works or not is mostly irrelevant (can help in fixing the issue faster though). Unit-testing-first has a tendency to make the tests very fine-grained, often need changes when you refactor internals (how do you know test did not break?), and needlessly keeping rigid internal details that one should have the freedom to change. Have seen plenty of code being kept around because it had lots of tests - when it was not actually used for anything at all! Note: Above approach is coupled with a pretty strict adherence to keeping projects small and independent, preferably <10kLOC.
Confusion 11y ago
The distribution of when bugs are caught, and what bugs are caught, by different kinds of tests is very different. Units tests catch bugs while they are being written and when the units they test are being refactored. The bugs they catch are trivial, silly, 'would have been caught anyway'. Outside of those activities, they rarely catch bugs. On the other hand, integration tests more consistently catch bugs and catch bugs you didn't foresee, have to investigate before understanding. You don't remember the last time a unit test caught a bug, because on average it is much longer ago and if it was a test for a unit you wrote this morning, you usually blame it on 'work in progress' and think you would have caught that trivial bug anyway. I think the nature of units tests make it easy to underappreciate them.
- Glide 11y ago
  This is too true. A team I was on started seeing way more value in integration tests and ended up writing way more integration tests than unit tests. What ended up happening is that the code quality ended up worse because they stopped running tests very often because there were too many integration tests and they took too long to run. Sad part is, a lot of those tests would have made a lot more sense as unit tests.
LoSboccacc 11y ago
my experience is like yours that unit test developed in development are rarely useful. however, having a framework in place helps a lot when working at user submitted issues: those are the issue one should strive to replicate trough unit test. In a six month of production, you get a fairly robust suites of test on all corner cases undiscovered in development.
ed_blackburn 11y ago
Admittedly not in node, but I've found that writing end-to-end tests but substituting different 'units' for in-memory or not depending on the test marries the best of both worlds. In-memory tests tend to focus on behaviour. In the rush to shoe-horn tests into a taxonomy, or 'integration' vs 'acceptances' vs 'unit' we often loose sight of the actual value proposition. Assisting design and building confidence to make changes.
taco_emoji 11y ago
You can use both things...
danieltillett 11y ago
It depends a bit on the problem you are trying to solve and the structure of your code. I have projects where only end-to-end tests find bugs and others where unit tests are helpful. What is never helpful is imposing some sort of semi-religious "thou shalt do x" in every case.
josteink 11y ago
> In my experience unit tests don't catch many bugs. On other hands end-to-end tests always surprise me with the bugs they catch. Obviously if you've walked the walk with unit-tests and done proper TDD and all that, you've caught the bugs before submitting your code. Ofcourse the remaining bugs (which there will be plenty of!) will be found in your end-to-end testing where all the bits are tied together. This is only natural and does is no way suggest unit-testing was a wasted effort.
- sheepmullet 11y ago
  "Obviously if you've walked the walk with unit-tests and done proper TDD and all that, you've caught the bugs before submitting your code." Unit testing and TDD are quite separate things. Perhaps you should try writing unit tests after you write your code and then you can come back and add value to the conversation.
  - josteink 11y ago
    > Perhaps you should try writing unit tests after you write your code and then you can come back and add value to the conversation. Nice snark there. Feel free to keep it to yourself. To add actual value to the conversation (as opposed to your contribution), I can very much recommend the book "Working Effectively with Legacy Code"[1] for how to handle unit-testing in the scenario of existing "legacy" code-bases. It's full of useful tips and methods to get testing in place "anywhere" and has a pragmatical (as opposed religious) approach to getting it done. To spark some interest: The book defines "legacy code" as any code not covered by unit-tests. It may be seem dated (from 2004 and all), but it's been the most useful book I've read on unit-testing by far. [1] http://www.amazon.com/gp/product/0131177052/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0131177052&linkCode=as2&tag=jostekjoni-20&linkId=XZ6RUGMKOUPE4HTQ http://www.amazon.com/gp/product/0131177052/ref=as_li_tl?ie=...
    - sheepmullet 11y ago
      You came along and dismissed the parents views without any real thought. If you write the tests afterwards you can better gauge how well they catch bugs. It sounds like the parent has tried this and has found them pretty useless. I've personally found that unit tests are not worthwhile for many components, and yet are critical for others.

zobzu 11y ago

i think it usually boils down to "its tests, its not performance sensitive" then tests take 10x the build time and eventually impact productivity like crazy :(

tajen 11y ago
At my startup (35 devs on the main product), we'd spin off a dozen AWS instances so the tests would run in 20 minutes in parallel. And only tested commits could be merged into master. We didn't feel it as a bad situation.
- nulltype 11y ago
  It seems like renting computers by the minute is pretty cheap these days. I'm surprised more people don't just have one machine per test, run them all in parallel. Testing seems to be embarrassingly parallel in that usually your tests don't depend on other tests at all.
  - zobzu 11y ago
    most of the problem is when tests depend on each other, which they oftentimes do or when they shouldnt but do, theres a large amount of work to change that
  - tajen 11y ago
    Unit tests are short, but integration tests require the app to start and setup, which takes around 3 minutes. Then each test is quite independent.

Asparagirl 11y ago

Wait, Shyp uses Sails.js for their framework? Huh, they might be the most well-known startup to use it. Sails development has really slowed down lately, especially since one of their core creators/commiters got into YC with a different project; I wonder how Shyp feels about that... "Avoiding Sails isn’t possible in every situation, but we’ve found it’s a useful design principle - it helps us separate our business logic and our data storage layer, and will make it easier for us to move off Sails in the future." Ooops.

angersock 11y ago
Well, to be fair, one of the patterns I've seen with Rails projects is to move as much out of Rails proper as possible, and treat it just as your web/(maybe)database interface.
kevinburke 11y ago
Yep - we've run into a lot of really unexpected behavior with it and with the ORM (Waterline), so we try to restrict ourselves to "the minimum possible framework subset that gets the code to run". For example, we disable `dynamicFinders`, don't use policies, don't use blueprints, don't use Waterline's associate/populate/destroy, don't use the "turn any controller function into a named route" setting, don't use globals, disabled all of the Grunt/websockets junk, and overwrote all of the error response handlers. The result is that day to day we're writing something that resembles an Express app with its own `routes` file. We're looking to move off of Waterline/Sails entirely, it's slow going though since we're a small team and it's tough to prioritize that, as it "works" now. Every company has parts of their tech stack that don't do a great job - we're somewhat fortunate this one is easy to mitigate.
- iLoch 11y ago
  Sounds like you guys pulled out everything except for Express. I too found the more I used Sails the less useful it became, as though it was missing a solution for anything lying outside of the very basics. Compare Sails to say, Laravel, and you begin to see how much work needs to be put in to bring it up to date.

shanemhansen 11y ago

So basically their biggest problem was that node.js machine-gun hammers the filesystem with stat calls when looking for modules. This was made worse by running on a shared folder. Vagrant has a mode called rsync-auto which will synchronize directories into a local vm. Or you could just always have the code local to the node process. It makes more sense to have your editor work with remote files. Next biggest problem was relational database. I think that tests requiring a SQL database are usually a bad sign, and the statement "we'd like to use transactions but our orm doesn't support them" makes me sad. To paraphrase Ben Franklin, "those who would give up transactions for a framework deserve neither". Here's some ideas for speeding up your tests, in order from best to worst. - Make most business code rely on a service or interface. Swap that our during tests for a trivial in-memory implementation. - Use sqlite :memory: for tests - TRUNCATE TABLE - Set your database to running with scissors mode (disable fsync) [edit] I just took a look at the ORM used, waterline https://github.com/balderdashy/waterline https://github.com/balderdashy/waterline . Why wasn't step one moving to sails-memory for running tests?

pyre 11y ago
> Use sqlite :memory: for tests Then you run into issues where the functionality works on SQLite, but not on your real database.
- bliti 11y ago
  May not work. Given the use of ORMs these days, this is a smaller issue. Though it still requires attention. Having the tests run quicker allows for better use of the programmers time
  - pyre 11y ago
    > Given the use of ORMs these days, this is a smaller issue The ORM can't abstract away all differences between databases. For example, SQLite doesn't have Numeric type, so all of the Numeric(10, 2) fields on (e.g.) my SQLAlchemy (a Python ORM) models will trigger warnings when saving to a SQLite database.

steveklabnik 11y ago

This post is full of great general advice that's not specific to Node. In Ruby-land, we've been talking about the same things for the past few years, complete with 'omg how many stat calls are we making again?' I'm sure others were doing the same before Ruby too. The wheel keeps turning...

shanemhansen 11y ago
I worked on a perl application server for a big ecommerce company. We'd stat so many files. Pulling in the meta-object library ('use Moose') was guaranteed to make your program take a long time to start.
why-el 11y ago
Any link to a discussion on these Ruby stat calls? We might look into them as well.
- steveklabnik 11y ago
  The core issue is that Rubygems adds the `lib` directory of every gem to your path, so you end up calling stat a zillion times. I am admittedly a bit out of the Ruby loop these days, so it's possible that this has been improved. The fastest way of course is to not load a zillion gems...

jkot 11y ago

Nice problem to have. I work on database engine, and full test suite takes about 2 weeks. After adding concurrency test, it might take a few months. I started migrating to AWS spot instances to make this whole monster manageable.

shanemhansen 11y ago
This is somewhat more reasonable. If your core business logic encompasses flushing data to lying disk controllers and being robust in the face of kill -9's, the root filesystem filling up, not corrupting the database on segfault, and not falling over on multi-terabyte datasets then your tests might take a long time.
Bluestrike2 11y ago
I just about choked on my coffee. If you're serious, is the issue just the size of the test suite, or are the tests themselves just woefully inefficient? By the time you're done, you've forgotten whatever it was that you modified.
- masklinn 11y ago
  I expect they have subsets of the suites working as "fast tests". The full sqlite test suite is 3 orders of magnitude bigger than sqlite itself and takes hours on beefy machines, but they're not run "online" by developers. Developers generally run the "veryquick" suite which takes a few minutes (and is sufficient to catch the vast majority of issues)
- Sanddancer 11y ago
  Database test suites tend to be on the ridiculously thorough side, for good reason. For example, the SQLite test suite ( https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html ) all but beats the code with a sledgehammer trying to break out bugs. Some software can't be upgraded once deployed, so it only makes sense to ensure that it is well-tested.

throwaway999666 11y ago

If it so happens that people who are having problems with the running time of tests have something like on the order of as much test code as they have "actual" code, I'm not surprised that they have to optimize the test suite. Apparently that was a trend some time ago (I don't know if it still is).

rdsubhas 11y ago

While bringing down test runtime 1000x is certainly a nice feat, I don't think this is setting a right precedent. I, for one, would be very afraid. Tests are not supposed to be written with speed in mind. Metrics seem to dominate the world of tests, where the spirit is supposed to be catching real world bugs. For example, the following can be done without destroying the spirit of testing: * Stub the sign in (big savings) * Keep a tab on fixtures (big savings) * Parallellize (big savings) * Buy a better machine (big savings) * Truncate with transactions/container-snapshots/etc (medium savings) * Tune database for speed than safety (medium savings) * Tune GC (medium savings), with enough RAM (16GB) disable it entirely (big savings) * Disable logging and instrumentation, yes including timing measurements, unless intentionally needed (small to medium savings) * Mount entire codebase, database and resources in tmpfs (small savings) The below, while they might improve performance, go against the spirit: * Tautological tests (http://fabiopereira.me/blog/2010/05/27/ttdd-tautological-test-driven-development-anti-pattern/ http://fabiopereira.me/blog/2010/05/27/ttdd-tautological-tes...) * Too many mocks/metaprogramming * Using a different database driver I'm planning to write a tool, which given a Class, will pump out lines and lines of tautological tests, line for line (a.a should == a.a), method for method (a.should receive save, a.save), branch for branch (with some metaprogramming). Coverage will report a cool 100%, tests will be blazing fast, and that should calm the lead devs down.

maccard 11y ago
> Tests are not supposed to be written with speed in mind This is why the tests took 100 seconds in the first place. While I agree that speed isn't the number one priority for everything, Reducing the length of time your entire test suite takes to run to be doable on every single run of your framework is absolutely worthwhile.
josteink 11y ago
> Tests are not supposed to be written with speed in mind. There's always extremes. If people write tests with no regard for speed (as I've seen) you can have "unit-tests" run for 45 minutes before completing. That might be OK to guard against regressions on a nightly-build, but it will in no way be a test-suite I'm willing to running prior to every commit I ship. If the test-suite runs in less than a minute, I'm much more willing to do so before submitting code, and in that way avoiding regressions which needs to be fixed later on. If the tests runs in less than 10 seconds, I'm probably going to run them for every major change I do, even if I'm not planning on submitting anything yet, just for peace of mind. There's IDEs where tests can be run automatically in the background as you make changes and let you know the second you broke something. For that to work efficiently, your tests has to be sufficiently fast. So maybe the aim shouldn't be that tests must be fast above everything else. But if the test-suite can be run quickly and casually, it's value will increase tremendously.

wilgertvelinga 11y ago

Wallaby.js is another way of speeding up tests during development.

wilgertvelinga 11y ago

For faster tests during development use wallaby.js!

dbbolton 11y ago

What vim colorscheme is that in the screencaps?

kevinburke 11y ago
hybrid_material - here: https://bitbucket.org/kevinburke/small-dotfiles/src/e4e1baf907ae103bedf445e022a795b57d9da9be/.vim/gvimrc?at=master#gvimrc-6 https://bitbucket.org/kevinburke/small-dotfiles/src/e4e1baf9... and here https://bitbucket.org/kevinburke/small-dotfiles/src/e4e1baf907ae103bedf445e022a795b57d9da9be/.gitmodules?at=master#.gitmodules-130 https://bitbucket.org/kevinburke/small-dotfiles/src/e4e1baf9...
- dbbolton 11y ago
  Thanks.

andresmanz 11y ago

I don't know what it is with this site, but it made Chrome (Android) crash four times already. Which means that, thanks to that wonderful browser, my two billion research tabs are gone since the first crash. Anyway, I can't imagine how running unit tests could possibly reach 100 seconds. Usually, that means that the tests are coupled to the environment (database, filesystem, ...). So without being able to read the article, that would be my guess. There are tons of good (and old) reasons why you should decouple unit tests from the environment as much as possible. Execution time is only one of them.

kevinburke 11y ago
Sorry to hear that Andres - what version of Chrome are you using on Android? Will try and see if we can figure out what is going wrong
- andresmanz 11y ago
  It's version 43.0.2357.93. I don't think it's your fault, though. Chrome loves crashing and throwing away all of my research tabs. That wasn't the first time. It's only the first time I could reproduce it with a specific website. Still, as I said, I don't think it's your fault. I'll just finish reading the article when I'm back at my PC. Test optimization is a great thing.

jgalt212 11y ago

> We run tests 400-500 times a day; a ten second improvement in every test run translates to an hour of saved developer time, every day. Is it just me, or does that seem like running a lot of tests per day. I mean, how many lines of code do they write per day? 1000? Do they run the test suite ever time they write two lines of code?

Fargren 11y ago
If your suite run in 100ms, why wouldn't you?
- Gigablah 11y ago
  The thing is, after reading all that, I think they're referring to their test suite startup time.
- jgalt212 11y ago
  because then I think you become to reliant upon tests, and not reliant enough upon correctly understanding the problem at hand. furthermore, your code is then only as good as your tests. And thus your code become vulnerable to failing on anything outside the test cases.
gtramont 11y ago
That is probably on a CI server. During development, I'd probably be running way more than that, at every production or test line written, having a task that watches for file changes and re-runs all tests. Then, having a fast test suite is key. While speeding up the process is always good, tools like lucifer or spork are only masking the problem: loading the whole environment for unit-level tests. Sails, like Rails, promote tight coupling between all parts of the system, which harms test speed, thus harming another development tool: TDD. The more decoupled and isolated you write your business logic, the easier and faster it is to test drive it. Then you simply plug an HTTP interface (with routes and whatnot) that calls into your core.

henvic 11y ago

Some npm requires are indeed painfully slow. For this reason I have wrote a small, but useful tool called require-time. https://www.npmjs.com/package/require-time https://www.npmjs.com/package/require-time https://asciinema.org/a/21275 https://asciinema.org/a/21275

srn_ 11y ago
There's also this: https://www.npmjs.com/package/require-times https://www.npmjs.com/package/require-times

nraynaud 11y ago

A decade ago I brought the test time from 86min down to a few seconds. An intern had thought that synchronizing things with "sleep one tenth of a second then check that flag again" at the heart of the product and reading and splitting by line a file where some lines might be 86Mo long were good ideas.

supertruth 11y ago

lucifer, scary