3 ms·

I know what property based testing is. And it's not a substitute for unit testing. It's basically just a way of generating many tests from a single test templat

I know what property based testing is. And it's not a substitute for unit testing. It's basically just a way of generating many tests from a single test template and is an alternative to example based testing. But it doesn't relate to what code you're testing, so what you're describing is still an integration test which serves a much different purpose from unit testing. Again, I think you just haven't had enough experience with unit testing and had that "aha" moment yet. It's not that I don't understand your point, I just doubt you have enough experience in testing strategies because you're trying to compare two things that aren't comparable. If I suggested your code use a function instead of a queue, you'd think I was ignorant and inexperienced because you can't replace a queue with a function...they're not concepts that can be swapped for one another. You suggesting property based testing instead of unit testing is likewise nonsensical. Unit testing and integration testing are different concepts of the same type. They cover what code gets tested. Property based testing is one of many strategies (example based testing, mutation testing, etc) that cover how code gets tested. Choosing to write unit tests instead of integration tests does not preclude the use of property based testing or any other technique that covers how you test code. As I stated before, if you were to use property based testing at the unit test level, it would be all the more powerful because you'd get that same exponential coverage just with significantly larger number of effective tests per unit of code. From the A, B, C dependency graph example, if you used property based testing to test A thousands of times, B thousands of times and C thousands of times, you'd effectively be testing thousands times thousands times thousands (aka billions) of different scenarios. And you'd still be able to use mocks because mocks are created in test code and can be as dynamic as you need them to be. I've done testing in many languages, both static and dynamic. Dynamic languages, thanks to the lack of static type checking, often don't need a mocking framework because the language is flexible enough that you can swap in a mock-like object at runtime. But the concepts are the same. And there really is a ton of value in testing code you've written in isolation. It's the testing equivalent of separation of concerns. And just like some developers never grok the single responsibility principle, some developers never grok the need to test code in isolation. You've obviously understood how to encapsulate your concerns based on the design you implemented. And yet the testing strategy you described is the opposite...trying to test everything at once rather than focusing on each test covering one specific bit of code.

loup-vaillant 9y ago
This exponential madness really irks me. I confess I have no idea how you can even imagine something like that, especially considering you test "in isolation". When you test A, B, and C three times each, you've got 9 tests. And since you were careful to test them in isolation, you can be sure testing A doesn't miraculously test C as well. Your 3^3=17 exponent comes out of nowhere. Me on the other hand could pretend something to this effect. By testing A, B, and C three times each, I effectively test B 6 times, and C 9 times, for a total of 3+6+9=18 "tests". But even then I would hesitate to pull such a number out of my ass, because the extra "free" tests are likely redundant with the real ones down the line. (And if my coverage is any good, those are guaranteed to be redundant). Surely you don't pretend mocking lets you test more things than my approach? Surely the mocs don't behave any differently than the real thing on the inputs they are given? How come then that my "integration" tests don't supersede your unit tests? > Choosing to write unit tests instead of integration tests does not preclude the use of property based testing I assume at this point that your definition of unit test means mocking everything, except perhaps the standard library. You said earlier: > Mocks verify that they were called in an expected way/order and give back the appropriate value. They don't actually do anything. They're highly coupled with each test case. So, if tests are generated randomly, mocs must be adapted to each random test, right? Now you need some moc generator, that will generate a moc for each test case. This veers eerily close to actually implementing the interface we were trying to moc. Now I realise I did miss something: > Mocks verify that they were called in an expected way/order and give back the appropriate value. Why would they do that? Oh, I see: we are mocking a mutable object that is used somewhere else. In other words, shared state. The one case for which I understand isolation could be good. This may all be a big misunderstanding. The A->B->C dependency I was thinking of is pretty simple: C is an implementation detail of B. Each B contains and uses a C, but that C isn't used anywhere else. Thus, mocking B or C would mean testing the implementation details of A, and that is a big no-no —the kind of thing Uncle Bob would call TDD gone wrong. > And there really is a ton of value in testing code you've written in isolation. What value exactly? Does isolating objects in the tests make those tests catch more bugs? What kind of bugs? Does isolating objects make detected bugs faster to track down? Does isolating objects in the tests influence how the code is written in the first place? How? Would the test suite run faster? > And yet the testing strategy you described is the opposite...trying to test everything at once rather than focusing on each test covering one specific bit of code. Now that's a strawman. I don't test everything at once. If I did, I would merely test A. No no no, I test and debug C first. When I'm confident it is bug-free, I test and debug B. Bugs are easily tracked down, because C is bug free, so if something went wrong it must be B. I only test A last, when I'm confident B and C are both bug-free.

loup-vaillant 9y ago

This exponential madness really irks me. I confess I have no idea how you can even imagine something like that, especially considering you test "in isolation". When you test A, B, and C three times each, you've got 9 tests. And since you were careful to test them in isolation, you can be sure testing A doesn't miraculously test C as well. Your 3^3=17 exponent comes out of nowhere. Me on the other hand could pretend something to this effect. By testing A, B, and C three times each, I effectively test B 6 times, and C 9 times, for a total of 3+6+9=18 "tests". But even then I would hesitate to pull such a number out of my ass, because the extra "free" tests are likely redundant with the real ones down the line. (And if my coverage is any good, those are guaranteed to be redundant). Surely you don't pretend mocking lets you test more things than my approach? Surely the mocs don't behave any differently than the real thing on the inputs they are given? How come then that my "integration" tests don't supersede your unit tests? > Choosing to write unit tests instead of integration tests does not preclude the use of property based testing I assume at this point that your definition of unit test means mocking everything, except perhaps the standard library. You said earlier: > Mocks verify that they were called in an expected way/order and give back the appropriate value. They don't actually do anything. They're highly coupled with each test case. So, if tests are generated randomly, mocs must be adapted to each random test, right? Now you need some moc generator, that will generate a moc for each test case. This veers eerily close to actually implementing the interface we were trying to moc. Now I realise I did miss something: > Mocks verify that they were called in an expected way/order and give back the appropriate value. Why would they do that? Oh, I see: we are mocking a mutable object that is used somewhere else. In other words, shared state. The one case for which I understand isolation could be good. This may all be a big misunderstanding. The A->B->C dependency I was thinking of is pretty simple: C is an implementation detail of B. Each B contains and uses a C, but that C isn't used anywhere else. Thus, mocking B or C would mean testing the implementation details of A, and that is a big no-no —the kind of thing Uncle Bob would call TDD gone wrong. > And there really is a ton of value in testing code you've written in isolation. What value exactly? Does isolating objects in the tests make those tests catch more bugs? What kind of bugs? Does isolating objects make detected bugs faster to track down? Does isolating objects in the tests influence how the code is written in the first place? How? Would the test suite run faster? > And yet the testing strategy you described is the opposite...trying to test everything at once rather than focusing on each test covering one specific bit of code. Now that's a strawman. I don't test everything at once. If I did, I would merely test A. No no no, I test and debug C first. When I'm confident it is bug-free, I test and debug B. Bugs are easily tracked down, because C is bug free, so if something went wrong it must be B. I only test A last, when I'm confident B and C are both bug-free.