7 ms·

The thing I really dislike are unit testing styles that use mock objects a lot. I see some people say that you should test each object in isolation, mocking ou

The thing I really dislike are unit testing styles that use mock objects a lot. I see some people say that you should test each object in isolation, mocking out any dependencies. That seems wrong headed to me. I prefer my unit tests to test everything all the way down. The only thing I would mock is the file system, which is useful for testing file loading code.

dale-cooper 14y ago
I think that's more towards some kind of integration testing. You're supposed to test one unit independently, and then write integration tests that actually tests that the whole system works. Not saying that's the best way but i think that's how people think.
azylman 14y ago
I prefer my unit tests to test everything all the way down. They already have that, that's called integration testing. Unit testing is trying to test as small of a unit of code as possible - this helps you identify exactly where the error is. If all of your tests are integration tests and something fails, you have no idea which part of your stack the problem is in. Obviously different projects require different kinds and amounts of test cases, but I prefer a mixture of both unit tests and integration tests (unit tests for small, complex blocks of logic, integration tests for most of the other stuff).
- Silhouette 14y ago
  If all of your tests are integration tests and something fails, you have no idea which part of your stack the problem is in. Not necessarily, because if you have a lot of integration tests then probably multiple ones will fail given a bug in a certain module, and the pattern of the failures might be all you need to locate the source of the problem. One of the most successful projects I ever worked on, in robustness/quality terms, didn't really have any unit tests at all, but it had a comprehensive suite of end-to-end test cases that could be run automatically. Many of those didn't (and couldn't) have an absolute true/false outcome, either, but looking at the results they generated and applying various heuristics developed from experience, they were remarkably useful for similar reasons to unit tests.
d0ugal 14y ago
So, you prefer your unit tests to be integration tests?
kibibyte 14y ago
That's what I thought until my first summer internship at a Rails shop. Our unit test suite could take a very long time to run (on my Macbook Air dev machine, it would take 15-20 minutes to run). The slowdown came from testing ActiveRecord models with many dependencies, and that we were creating sample models. As a result, our tests hit the test database multiple times, which made them slow down. Our solution was to start mocking as many models as we could, which really did improve test runtime. (Slight edit: after reading the other replies, now I really understand what integration tests are for.)
- Firehed 14y ago
  How many issues do you think may be hidden by mocking out the database layer? It's easy to write bad data due to a subtle bug in a query builder for example and then write garbage into the database. Mocking would make that significantly harder to catch.
  - urbanautomaton 14y ago
    You're quite right, query builder statements are error-prone and difficult to stub; that's why you shouldn't sprinkle them throughout your app. Rather than write things like User.active.in_group(x).not_replied.each { ... } all over the place, write scopes on your User model (e.g. User.awaiting_response_for_group(x)) that represent each use case, test them thoroughly against the database, and then stub them out when testing client code. That way you only have to test the queries in one place, and you don't couple the rest of your code to your query builder.
jakejake 14y ago
Technically it is called integration testing but I agree with you on the idea. Unless there's a reason to mock something up (like simulating a fail condition from a DB or network call, or simulating other code that isn't finished yet) I prefer to just test everything as close as possible to what is actually running. The level of granularity, like the OP said, changes depending on the complexity and the critical nature of each part of the code. Some people argue that if I make an error in function A and that causes a unit test fail for function B - that can mislead a developer into thinking function B has an error. I argue that a) it rarely misleads anybody for very long and b) if I had function B tested in total isolation with mocks then that error in function A might have slipped through unnoticed. It's tricky to mock up some combination that you weren't expecting to happen. Obviously in ideal circumstances you would be doing both unit and integration testing. But, you have to draw a line somewhere otherwise you just write tests all day and get no production code running.
discreteevent 14y ago
Here are qualities of good unit tests: 1) They run fast. 2) They help us localize problems. In the industry, people often go back and forth about whether particular tests are unit tests. Is a test really a unit test if it uses another production class? I go back to the two qualities: Does the test run fast? Can it help us localize errors quickly? Naturally, there is a continuum. Some tests are larger, and they use several classes together. In fact, they may seem to be little integration tests. By themselves, they might seem to run fast, but what happens when you run them all together? When you have a test that exercises a class along with several of its collaborators, it tends to grow. If you haven't taken the time to make a class separately instantiable in a test harness, how easy will it be when you add more code? It never gets easier. - From "Working Effectively With Legacy Code" by Michael Feathers. (A really good book that actually does a better job of teaching design than many a book because like all good teaching it starts with the examples and abstracts from there, not the other way around.)
smackay 14y ago
To avoid getting hung up on taxonomy I much prefer defining the scope of tests with the control theory terms of Observability and Controllability, http://en.wikipedia.org/wiki/Observability http://en.wikipedia.org/wiki/Observability. That makes it way easier to categorize tests in terms of how effective they are and gives useful starting point to determine whether the granularity of the test really is able to adequately test the desired behaviour.
- TeMPOraL 14y ago
  Control theory also tells us how feedback loops are related to those two terms; a lot of discussions about Agile, etc. would be much more productive if people understood that it is the feedback loop that matters (as opposed to particular methodology flavour) and why it does.
meaty 14y ago
If you need lots of mocks, it's a sign of coupling which means a bad fundamental design. You need to test both the components individually (unit tests) and their interaction (integration tests). We use Commons VFS for filesystem abstraction ( http://commons.apache.org/vfs/ http://commons.apache.org/vfs/ ) and our own port of the API for .Net. In fact we have our own systems API (more Google AppEngine style thank say the .Net Framework or Java libraries) on which we build our software.
- jiggy2011 14y ago
  That may be true, but sometimes it can also be a pain when you are talking to everything through a load of abstraction layers. When something blows up it can be nice just to get a simple error rather than a 20 page stack trace.
  - Silhouette 14y ago
    That may be true, but sometimes it can also be a pain when you are talking to everything through a load of abstraction layers. Exactly. Sometimes the benefit of a certain kind of test does not outweigh the cost of distorting a useful design so it will support the test code. That cost might be directly in terms of performance overhead, or it can be a more subtle overhead through (for example) weakening language-enforced modularity in your architecture so you can insert mock/stub placeholders where you want them for testing purposes, which can hurt everything from maintenance overheads to the effectiveness of code reviews.
    - jiggy2011 14y ago
      I think dependency injection is designed to mostly solve that problem but with that you want to program in a very specific way (e.g calling the DI framework's factory methods rather than just using "new" and a constructor method). This makes it more difficult to retrofit to code and is sort of awkward when you are integrating with different bits of code and libraries that don't use DI. Thus the usefulness of mocking frameworks to "force" a particular class to be a certain implementation for the purposes of testing.
jrogers65 14y ago
A unit test tests a unit of code. An integration or functional test tests the interaction between multiple units of code, usually from the point of view of "user needs functionality X, functional test Y ensures that X works". Unit testing is not supposed to catch bugs, it is primarily to ensure a solid design. Integration/functional tests are primarily there to ensure that the code works. It only makes sense to unit test code which is in the domain of architecture. This ensures that your domain model is solid. It only makes sense to functional test code which is a concrete implementation running on top of said architecture. This ensures that the end user sees what they need to see. One of the often quoted benefits of unit testing is that they run fast. That fact is only incidental. Not to mention the insanity of unit testing as much as you can just because they're faster to run - they are also an order of magnitude slower to write than functional tests. tl;dr - use the right tool for the right job. Unit testing is for design/architecture. Functional testing is for concrete functionality.

dale-cooper 14y ago

I think that's more towards some kind of integration testing. You're supposed to test one unit independently, and then write integration tests that actually tests that the whole system works. Not saying that's the best way but i think that's how people think.

azylman 14y ago

I prefer my unit tests to test everything all the way down. They already have that, that's called integration testing. Unit testing is trying to test as small of a unit of code as possible - this helps you identify exactly where the error is. If all of your tests are integration tests and something fails, you have no idea which part of your stack the problem is in. Obviously different projects require different kinds and amounts of test cases, but I prefer a mixture of both unit tests and integration tests (unit tests for small, complex blocks of logic, integration tests for most of the other stuff).

Silhouette 14y ago
If all of your tests are integration tests and something fails, you have no idea which part of your stack the problem is in. Not necessarily, because if you have a lot of integration tests then probably multiple ones will fail given a bug in a certain module, and the pattern of the failures might be all you need to locate the source of the problem. One of the most successful projects I ever worked on, in robustness/quality terms, didn't really have any unit tests at all, but it had a comprehensive suite of end-to-end test cases that could be run automatically. Many of those didn't (and couldn't) have an absolute true/false outcome, either, but looking at the results they generated and applying various heuristics developed from experience, they were remarkably useful for similar reasons to unit tests.

d0ugal 14y ago

So, you prefer your unit tests to be integration tests?

kibibyte 14y ago

That's what I thought until my first summer internship at a Rails shop. Our unit test suite could take a very long time to run (on my Macbook Air dev machine, it would take 15-20 minutes to run). The slowdown came from testing ActiveRecord models with many dependencies, and that we were creating sample models. As a result, our tests hit the test database multiple times, which made them slow down. Our solution was to start mocking as many models as we could, which really did improve test runtime. (Slight edit: after reading the other replies, now I really understand what integration tests are for.)

Firehed 14y ago
How many issues do you think may be hidden by mocking out the database layer? It's easy to write bad data due to a subtle bug in a query builder for example and then write garbage into the database. Mocking would make that significantly harder to catch.
- urbanautomaton 14y ago
  You're quite right, query builder statements are error-prone and difficult to stub; that's why you shouldn't sprinkle them throughout your app. Rather than write things like User.active.in_group(x).not_replied.each { ... } all over the place, write scopes on your User model (e.g. User.awaiting_response_for_group(x)) that represent each use case, test them thoroughly against the database, and then stub them out when testing client code. That way you only have to test the queries in one place, and you don't couple the rest of your code to your query builder.

jakejake 14y ago

Technically it is called integration testing but I agree with you on the idea. Unless there's a reason to mock something up (like simulating a fail condition from a DB or network call, or simulating other code that isn't finished yet) I prefer to just test everything as close as possible to what is actually running. The level of granularity, like the OP said, changes depending on the complexity and the critical nature of each part of the code. Some people argue that if I make an error in function A and that causes a unit test fail for function B - that can mislead a developer into thinking function B has an error. I argue that a) it rarely misleads anybody for very long and b) if I had function B tested in total isolation with mocks then that error in function A might have slipped through unnoticed. It's tricky to mock up some combination that you weren't expecting to happen. Obviously in ideal circumstances you would be doing both unit and integration testing. But, you have to draw a line somewhere otherwise you just write tests all day and get no production code running.

discreteevent 14y ago

Here are qualities of good unit tests: 1) They run fast. 2) They help us localize problems. In the industry, people often go back and forth about whether particular tests are unit tests. Is a test really a unit test if it uses another production class? I go back to the two qualities: Does the test run fast? Can it help us localize errors quickly? Naturally, there is a continuum. Some tests are larger, and they use several classes together. In fact, they may seem to be little integration tests. By themselves, they might seem to run fast, but what happens when you run them all together? When you have a test that exercises a class along with several of its collaborators, it tends to grow. If you haven't taken the time to make a class separately instantiable in a test harness, how easy will it be when you add more code? It never gets easier. - From "Working Effectively With Legacy Code" by Michael Feathers. (A really good book that actually does a better job of teaching design than many a book because like all good teaching it starts with the examples and abstracts from there, not the other way around.)

smackay 14y ago

To avoid getting hung up on taxonomy I much prefer defining the scope of tests with the control theory terms of Observability and Controllability, http://en.wikipedia.org/wiki/Observability http://en.wikipedia.org/wiki/Observability. That makes it way easier to categorize tests in terms of how effective they are and gives useful starting point to determine whether the granularity of the test really is able to adequately test the desired behaviour.

TeMPOraL 14y ago
Control theory also tells us how feedback loops are related to those two terms; a lot of discussions about Agile, etc. would be much more productive if people understood that it is the feedback loop that matters (as opposed to particular methodology flavour) and why it does.

meaty 14y ago

If you need lots of mocks, it's a sign of coupling which means a bad fundamental design. You need to test both the components individually (unit tests) and their interaction (integration tests). We use Commons VFS for filesystem abstraction ( http://commons.apache.org/vfs/ http://commons.apache.org/vfs/ ) and our own port of the API for .Net. In fact we have our own systems API (more Google AppEngine style thank say the .Net Framework or Java libraries) on which we build our software.

jiggy2011 14y ago
That may be true, but sometimes it can also be a pain when you are talking to everything through a load of abstraction layers. When something blows up it can be nice just to get a simple error rather than a 20 page stack trace.
- Silhouette 14y ago
  That may be true, but sometimes it can also be a pain when you are talking to everything through a load of abstraction layers. Exactly. Sometimes the benefit of a certain kind of test does not outweigh the cost of distorting a useful design so it will support the test code. That cost might be directly in terms of performance overhead, or it can be a more subtle overhead through (for example) weakening language-enforced modularity in your architecture so you can insert mock/stub placeholders where you want them for testing purposes, which can hurt everything from maintenance overheads to the effectiveness of code reviews.
  - jiggy2011 14y ago
    I think dependency injection is designed to mostly solve that problem but with that you want to program in a very specific way (e.g calling the DI framework's factory methods rather than just using "new" and a constructor method). This makes it more difficult to retrofit to code and is sort of awkward when you are integrating with different bits of code and libraries that don't use DI. Thus the usefulness of mocking frameworks to "force" a particular class to be a certain implementation for the purposes of testing.

jrogers65 14y ago

A unit test tests a unit of code. An integration or functional test tests the interaction between multiple units of code, usually from the point of view of "user needs functionality X, functional test Y ensures that X works". Unit testing is not supposed to catch bugs, it is primarily to ensure a solid design. Integration/functional tests are primarily there to ensure that the code works. It only makes sense to unit test code which is in the domain of architecture. This ensures that your domain model is solid. It only makes sense to functional test code which is a concrete implementation running on top of said architecture. This ensures that the end user sees what they need to see. One of the often quoted benefits of unit testing is that they run fast. That fact is only incidental. Not to mention the insanity of unit testing as much as you can just because they're faster to run - they are also an order of magnitude slower to write than functional tests. tl;dr - use the right tool for the right job. Unit testing is for design/architecture. Functional testing is for concrete functionality.