4 ms·

> Writing testable code is not "damage" Correct, but you can go a long way in writing testable code without requiring mock implementations. While I don’t deny

by klmr 7y ago

> Writing testable code is not "damage" Correct, but you can go a long way in writing testable code without requiring mock implementations. While I don’t deny the usefulness of mock testing, it comes at a steep cost (= vastly more complex code base), and I increasingly question whether that’s worth it. In practice I find it more useful to unit test what can be tested in isolation, and to perform integration testing for the test, and to limit mock testing (and thus unit tests which would require mocking) to a few well-defined interfaces. This cuts down drastically on boilerplate, with little (if any) loss of testability. And these few remaining mock object dependencies can be modelled via simple parameters and higher-order functions, further cutting down on extraneous interface classes.

BeetleB 7y ago
> Correct, but you can go a long way in writing testable code without requiring mock implementations. For some languages (e.g. C++), if you are using classes, it can actually be fairly challenging to do unit tests without mocks. And almost every solution to this problem out there is to use interfaces/dependency injection/inversion.
- klmr 7y ago
  > [in C++] it can actually be fairly challenging to do unit tests without mocks Yes, but you can write integration tests instead. There’s no rule that says that unit testing must cover every aspect of the software. Unit test those units that can be used in isolation, integration test the rest. Case in point, our main software at work, written in C++, has close to 100% test coverage, and uses almost no mock tests. Instead, we have extensive integration tests where unit testing without mocking doesn’t work. And, again: I’m not saying that there are no tradeoffs involved in that. But the benefit of having a drastically simpler code architecture outweigh the cost of having to move those tests into integration. Incidentally this is vastly easier when the architecture cleanly separates business logic from general logic and moves as much of it as possible into side-effect free general-purpose functions.
  - BeetleB 7y ago
    > Yes, but you can write integration tests instead. There’s no rule that says that unit testing must cover every aspect of the software. Unit test those units that can be used in isolation, integration test the rest. Case in point, our main software at work, written in C++, has close to 100% test coverage, and uses almost no mock tests. Instead, we have extensive integration tests where unit testing without mocking doesn’t work. In my last C++ job, perhaps less than 5% of our code was unit testable. We had very good test coverage using integrated tests, but those were very expensive tests. Running a full test suite took 10 hours. Our SW heavily relied on certain equipment. Fortunately, the vendor had provided a very good simulator for it - so almost all our integration tests needed that simulator. And the simulator was such that there was a fair amount of overhead in loading and setting it up and unloading it. We didn't do this for every test - that would have taken days/weeks to run - but for logically grouped tests. And unfortunately many of the integration tests we wrote were much more suited to unit tests, but because we didn't have interfaces, we were forced to test them via an integration test, which was very expensive. Oh, and the simulator would not allow parallelism - you could have only one simulator running on a given machine at a time. So all our tests had to be run in sequence. And of course, the simulator had bugs, so whenever we had a failure, we had to determine if the simulator was at fault. Especially for intermittent failures. So yes, we had very thorough tests and were proud of it, but unit testing for perhaps 30% of our tests would have been better. But to unit test those we really needed interfaces. I know because I convinced management to give me some weeks to convert one of our simplest portions of code into something we could unit test. I thought it would be trivial, but I ended up needing interfaces. There is a whole other approach which our sister team (very similar SW, but different HW) used. They did not have a reliable simulator. So they wrote a "fake" simulator. They simply reproduced the APIs and wrote code to return certain values under certain conditions - a poor man's mock. On the plus side their test suite ran very quickly. The down side was they needed to write many mocks for the same function (for when they needed to return different values). Writing tests was very expensive for them. Why did they write fakes instead of using a mocking framework? Because every mocking framework they found in C++ required interfaces, and they didn't want to change their SW architecture. In many cases integration tests are OK. But in our case, they were extremely expensive. > Incidentally this is vastly easier when the architecture cleanly separates business logic from general logic and moves as much of it as possible into side-effect free general-purpose functions. This is precisely where interfaces come from! It's not trivial to separate business logic from general logic without them. When I took the time to show how we could write unit tests for our code base, I ended up with interfaces. And no, I didn't read any books/sites telling me I should do it. I didn't even know they were called interfaces. I just took some of the simplest code in our codebase, and asked "How can I write unit tests for this?" and essentially reinvented interfaces. I separated the business logic from the logic requiring the simulator, and then wrote unit tests for the business logic, and had an interface class to interface with the simulator. To give you an idea, the code I converted to unit tests was trivial: Given this input string with these flags set, return this converted string, etc. It should be one of the easiest things to unit test! However, the input string and flags were entities understood by the HW and so we had simulator dependencies. I wrote a class that dealt only with strings and booleans to implement the business logic, and then had the interface class translate back and forth between strings/bools to entities the simulator expected/understood. I appreciate your comment, and trust me, I'm not trying to be difficult. But while I've heard many claims, every time I've gone into the details I end up with interface classes. I work at a large company and went around asking several teams in different C++ projects, and they all independently had come to the same conclusion: Either use interfaces or live without unit tests. I presented my work to convert portions of our code base to unit tests at internal SW conferences, and would say "This is a highly nontrivial and crappy way to do unit tests in C++. I'd really like to hear from people in the audience if they found a better way." And whoever approached me afterwards said they had gone down the same path as me and ended up with interfaces. Testing in C++ sucks, to be frank. As an aside, I'm not pro-mocks, BTW. I recently did a small personal Python project where I insisted doing TDD. I would not write code till I had a test. And this involved several mocks. And while all my tests involving mocks passed, in the majority of cases involving the mocked code, my program had bugs and would fail when I tested with real data. (Maybe that's more a criticism of TDD than of mocks, though).

by klmr 7y ago

BeetleB 7y ago

> Correct, but you can go a long way in writing testable code without requiring mock implementations. For some languages (e.g. C++), if you are using classes, it can actually be fairly challenging to do unit tests without mocks. And almost every solution to this problem out there is to use interfaces/dependency injection/inversion.

klmr 7y ago
> [in C++] it can actually be fairly challenging to do unit tests without mocks Yes, but you can write integration tests instead. There’s no rule that says that unit testing must cover every aspect of the software. Unit test those units that can be used in isolation, integration test the rest. Case in point, our main software at work, written in C++, has close to 100% test coverage, and uses almost no mock tests. Instead, we have extensive integration tests where unit testing without mocking doesn’t work. And, again: I’m not saying that there are no tradeoffs involved in that. But the benefit of having a drastically simpler code architecture outweigh the cost of having to move those tests into integration. Incidentally this is vastly easier when the architecture cleanly separates business logic from general logic and moves as much of it as possible into side-effect free general-purpose functions.
- BeetleB 7y ago
  > Yes, but you can write integration tests instead. There’s no rule that says that unit testing must cover every aspect of the software. Unit test those units that can be used in isolation, integration test the rest. Case in point, our main software at work, written in C++, has close to 100% test coverage, and uses almost no mock tests. Instead, we have extensive integration tests where unit testing without mocking doesn’t work. In my last C++ job, perhaps less than 5% of our code was unit testable. We had very good test coverage using integrated tests, but those were very expensive tests. Running a full test suite took 10 hours. Our SW heavily relied on certain equipment. Fortunately, the vendor had provided a very good simulator for it - so almost all our integration tests needed that simulator. And the simulator was such that there was a fair amount of overhead in loading and setting it up and unloading it. We didn't do this for every test - that would have taken days/weeks to run - but for logically grouped tests. And unfortunately many of the integration tests we wrote were much more suited to unit tests, but because we didn't have interfaces, we were forced to test them via an integration test, which was very expensive. Oh, and the simulator would not allow parallelism - you could have only one simulator running on a given machine at a time. So all our tests had to be run in sequence. And of course, the simulator had bugs, so whenever we had a failure, we had to determine if the simulator was at fault. Especially for intermittent failures. So yes, we had very thorough tests and were proud of it, but unit testing for perhaps 30% of our tests would have been better. But to unit test those we really needed interfaces. I know because I convinced management to give me some weeks to convert one of our simplest portions of code into something we could unit test. I thought it would be trivial, but I ended up needing interfaces. There is a whole other approach which our sister team (very similar SW, but different HW) used. They did not have a reliable simulator. So they wrote a "fake" simulator. They simply reproduced the APIs and wrote code to return certain values under certain conditions - a poor man's mock. On the plus side their test suite ran very quickly. The down side was they needed to write many mocks for the same function (for when they needed to return different values). Writing tests was very expensive for them. Why did they write fakes instead of using a mocking framework? Because every mocking framework they found in C++ required interfaces, and they didn't want to change their SW architecture. In many cases integration tests are OK. But in our case, they were extremely expensive. > Incidentally this is vastly easier when the architecture cleanly separates business logic from general logic and moves as much of it as possible into side-effect free general-purpose functions. This is precisely where interfaces come from! It's not trivial to separate business logic from general logic without them. When I took the time to show how we could write unit tests for our code base, I ended up with interfaces. And no, I didn't read any books/sites telling me I should do it. I didn't even know they were called interfaces. I just took some of the simplest code in our codebase, and asked "How can I write unit tests for this?" and essentially reinvented interfaces. I separated the business logic from the logic requiring the simulator, and then wrote unit tests for the business logic, and had an interface class to interface with the simulator. To give you an idea, the code I converted to unit tests was trivial: Given this input string with these flags set, return this converted string, etc. It should be one of the easiest things to unit test! However, the input string and flags were entities understood by the HW and so we had simulator dependencies. I wrote a class that dealt only with strings and booleans to implement the business logic, and then had the interface class translate back and forth between strings/bools to entities the simulator expected/understood. I appreciate your comment, and trust me, I'm not trying to be difficult. But while I've heard many claims, every time I've gone into the details I end up with interface classes. I work at a large company and went around asking several teams in different C++ projects, and they all independently had come to the same conclusion: Either use interfaces or live without unit tests. I presented my work to convert portions of our code base to unit tests at internal SW conferences, and would say "This is a highly nontrivial and crappy way to do unit tests in C++. I'd really like to hear from people in the audience if they found a better way." And whoever approached me afterwards said they had gone down the same path as me and ended up with interfaces. Testing in C++ sucks, to be frank. As an aside, I'm not pro-mocks, BTW. I recently did a small personal Python project where I insisted doing TDD. I would not write code till I had a test. And this involved several mocks. And while all my tests involving mocks passed, in the majority of cases involving the mocked code, my program had bugs and would fail when I tested with real data. (Maybe that's more a criticism of TDD than of mocks, though).