3 ms·

> I hard coded my A->B->C dependency chain instead of injecting it Read my above point about the cartesian product of tests...if you're doing this, you're almo

> I hard coded my A->B->C dependency chain instead of injecting it Read my above point about the cartesian product of tests...if you're doing this, you're almost certainly not getting particularly good coverage of edge cases even if your test coverage percentage is high. If I have test A1, A2, A3 that use a mocked B and B1, B2, B3 that use a mocked C and C1, C2, C3, I've essentially tested 27 different cases that you'd have to write in your hard-coded dependency style while writing only 9 tests. When an application reaches even a moderate amount of complexity and that dependency graph grows beyond 3, that ability for unit tests to cover a geometrically-increasing number of ways that the software behaves is going to be hard to overcome with integration tests like you seem to have been writing. But, again, maybe your dependency graph wasn't big enough to need this. Maybe there was some other mitigating factor that made your situation different. I don't want to speculate on any individual situation since I haven't seen the code and there's no universally correct answer. > I'm more the YAGNI type: those changes you're trying to anticipate? They likely won't happen. The one virtually certain thing about the future is that there will be change. Of course you mostly can't predict what will change, but you can be damn near certain that something will. YAGNI is not about predicting that nothing will ever change, it's only about avoiding guessing at what will. Tests on current functionality are not guessing about what will change. The whole point of a test suite is that it avoids ossification. What can be less rigid than something that can easily be pulled apart and put back together in some other form while being sure that it still works? Simple and flexible without that property of safe reorganization is still a landmine waiting to go off in the face of future change.

loup-vaillant 9y ago
OK, let's get practical. A happened to be a reactor base class inspired by http://ithare.com http://ithare.com ; B was a MWSR concurrent queue (scary mutexes and all) ; C was a ring buffer on which the queue was built. The goal of the whole thing was to pass plain data objects between reactors. The one "requirement" that ended up changing was that plain data objects weren't enough (we want to pass anything that has a move constructor). This means scrapping the ring buffer, and modify the MWSR queue to use something else (possibly an `std::queue`). I used property based tests. With relatively little effort, I ran thousands of tests. Allow me to laugh at your measly 27 from atop my high horse. More seriously, though, I don't know how I could have mocked my ring buffer without writing another full featured ring buffer. Mocking looks a bit ridiculous in this case. Same for the queue. And the reason why this is so is not tight coupling. It's because my queue relied on the functionality provided by the ring buffer, and my reactor relied on the functionality provided by the queue. If this wasn't the case, I would have written a simpler ring buffer and queue to begin with. Also, why mock stuff when I can test my code under real conditions? > The whole point of a test suite is that it avoids ossification. This feels backwards. If you just mean that test suites prevent the unintended consequences of change, sure. They're like a home made type system in that respect. But if you suggest we should bend a code base just to make it "let's mock everything" testable™, then no. Don't mock stuff for the sake of it, it's a wast of time. I'd rather concentrate on making the code usable in the first place. With small, well defined interfaces. In my experience, such designs are naturally testable.
- curun1r 9y ago
  I'm not going to comment on how you should have done it without seeing the code. You can throw out terms like reactor base class, MWSR (which I've never heard of but am assuming is just an MPSC using non-standard terminology) and ring buffer, but that's not enough to go on to talk about your specific situation. Moreover, it sounds like you're using C++ which isn't a language I've spent much time coding in, so I'm ill equipped to talk about mocking frameworks available to you. > Allow me to laugh at your measly 27 from atop my high horse If you're going make snide comments, I'll stop this discussion. Suffice it to say you missed the point. 27 isn't just a number, it's a number arrived at by applying an exponent. Exponents get really big in a hurry. As the author of a crypto suite, I'd expect you to pay exponents a bit more respect. Testing 3 different cases per dependency gives you 27. Testing 10 gives you 1000. Property based testing is not a substitute for this kind of coverage. It is, however, complementary. It allows you to increase the base number which is raised to the exponent. > I could have mocked my ring buffer without writing another full featured ring buffer. Mocking looks a bit ridiculous in this case. It's safe to assume from this comment that you don't really understand mocking. Mocks aren't an implementation of an interface to be used in tests. Mocks verify that they were called in an expected way/order and give back the appropriate value. They don't actually do anything. They're highly coupled with each test case. This is why most mocking frameworks dynamically generate the mock per-test and let you simply verify the call(s) and tell it what to return. > This feels backwards It's absolutely not backwards. The main purpose of a test suite is to find a very specific type of bug...regressions. It's not to avoid other sorts of bugs other than the fact that forcing developers to write tests will force them to think about their code in ways that make them realize they missed something during implementation. But the most important thing is to be able to rely on them to tell you when you've broken something that previously worked. Every single time I've come to a code base that was either missing tests or relied too heavily on integration tests, we've had a significant problem with regressions and, over time, developers shied away from making significant changes to the code base because they were afraid to break it. And the result is that ossification that you try not to achieve. But when you see a code base with a commitment to unit tests, you also see one where developers are free to make the necessary radical changes without fear of breaking existing functionality. They make the changes, run the suite and let it tell them everywhere they forgot to change, whether that be in application code or test code.

loup-vaillant 9y ago

OK, let's get practical. A happened to be a reactor base class inspired by http://ithare.com http://ithare.com ; B was a MWSR concurrent queue (scary mutexes and all) ; C was a ring buffer on which the queue was built. The goal of the whole thing was to pass plain data objects between reactors. The one "requirement" that ended up changing was that plain data objects weren't enough (we want to pass anything that has a move constructor). This means scrapping the ring buffer, and modify the MWSR queue to use something else (possibly an `std::queue`). I used property based tests. With relatively little effort, I ran thousands of tests. Allow me to laugh at your measly 27 from atop my high horse. More seriously, though, I don't know how I could have mocked my ring buffer without writing another full featured ring buffer. Mocking looks a bit ridiculous in this case. Same for the queue. And the reason why this is so is not tight coupling. It's because my queue relied on the functionality provided by the ring buffer, and my reactor relied on the functionality provided by the queue. If this wasn't the case, I would have written a simpler ring buffer and queue to begin with. Also, why mock stuff when I can test my code under real conditions? > The whole point of a test suite is that it avoids ossification. This feels backwards. If you just mean that test suites prevent the unintended consequences of change, sure. They're like a home made type system in that respect. But if you suggest we should bend a code base just to make it "let's mock everything" testable™, then no. Don't mock stuff for the sake of it, it's a wast of time. I'd rather concentrate on making the code usable in the first place. With small, well defined interfaces. In my experience, such designs are naturally testable.

curun1r 9y ago
I'm not going to comment on how you should have done it without seeing the code. You can throw out terms like reactor base class, MWSR (which I've never heard of but am assuming is just an MPSC using non-standard terminology) and ring buffer, but that's not enough to go on to talk about your specific situation. Moreover, it sounds like you're using C++ which isn't a language I've spent much time coding in, so I'm ill equipped to talk about mocking frameworks available to you. > Allow me to laugh at your measly 27 from atop my high horse If you're going make snide comments, I'll stop this discussion. Suffice it to say you missed the point. 27 isn't just a number, it's a number arrived at by applying an exponent. Exponents get really big in a hurry. As the author of a crypto suite, I'd expect you to pay exponents a bit more respect. Testing 3 different cases per dependency gives you 27. Testing 10 gives you 1000. Property based testing is not a substitute for this kind of coverage. It is, however, complementary. It allows you to increase the base number which is raised to the exponent. > I could have mocked my ring buffer without writing another full featured ring buffer. Mocking looks a bit ridiculous in this case. It's safe to assume from this comment that you don't really understand mocking. Mocks aren't an implementation of an interface to be used in tests. Mocks verify that they were called in an expected way/order and give back the appropriate value. They don't actually do anything. They're highly coupled with each test case. This is why most mocking frameworks dynamically generate the mock per-test and let you simply verify the call(s) and tell it what to return. > This feels backwards It's absolutely not backwards. The main purpose of a test suite is to find a very specific type of bug...regressions. It's not to avoid other sorts of bugs other than the fact that forcing developers to write tests will force them to think about their code in ways that make them realize they missed something during implementation. But the most important thing is to be able to rely on them to tell you when you've broken something that previously worked. Every single time I've come to a code base that was either missing tests or relied too heavily on integration tests, we've had a significant problem with regressions and, over time, developers shied away from making significant changes to the code base because they were afraid to break it. And the result is that ossification that you try not to achieve. But when you see a code base with a commitment to unit tests, you also see one where developers are free to make the necessary radical changes without fear of breaking existing functionality. They make the changes, run the suite and let it tell them everywhere they forgot to change, whether that be in application code or test code.