3 ms·

That article is garbage. For example, the claim that an object has trillions of states ignores our ability to classify ranges of values. Sure, an int may contai

That article is garbage. For example, the claim that an object has trillions of states ignores our ability to classify ranges of values. Sure, an int may contain "four billion states", but for the requirements (which he makes such a big deal about) its highly likely that we can classify the integer into three states: less than zero, zero, greater than zero. As a bank, I might not care how much money you have, only that you have more than zero. In a transaction, I don't care how much money changes hands, as long as no money is lost. The article is a tantrum.

falcolas 9y ago
> we can classify the integer into three states: less than zero, zero, greater than zero Err, not quite. We're running on computers, remember. A comprehensive unit test that doesn't go through every possible integer input should still include: maximum and minimum integer values for that size maximum and minimum expected inputs 0, 1, -1 So, there's 7 test cases for a single integral input. And since you bring up banking, let's imagine you're working with a function with two inputs. Since most bugs come about because of the interaction between two variables [0], you'd want to check out each combination. So, 7 possible inputs for the first integer, 7 for the second: 49 test cases. Why bother? You're working for a bank... imagine facing a client whose balance shot negative because of an overflow error and started accruing "overdraft coverage" fees. Back to the original point: when was the last time anybody but the AFL tool wrote the "proper" 49+ unit tests for an "add_to_balance(int, int)" style function, when one test would give you 100% coverage for that function? [0] https://csrc.nist.gov/Projects/Automated-Combinatorial-Testing-for-Software https://csrc.nist.gov/Projects/Automated-Combinatorial-Testi...
- lowbloodsugar 9y ago
  If you mean "nobody would write 49 tests for a function when they could get away with writing one test and have it show up as 100% code coverage because people are lazy bastards" then I agree, and that's the problem: cargo culting the metrics. Look we have 100% code coverage! We are awesome! If you mean, "no look seriously, you'd need 49 tests for every method because you're a bank", well then, no, you actually need less than 49 (since some are subsets of the others), and in reality even this would get very tiring so you'd write a currency class that can't overflow, and yes, being a bank we would test every case and certainly use automated test generation while we're at it. But if you mean that in general, for non-banks, the classes of values is too large to manually write unit tests for, then we disagree. That means that your method is too complex.
  - falcolas 9y ago
    > some are subsets of the others In a purely mathematical sense, yes, this is true. But with computers those subsets matter. For example, I wouldn't drop the "max expected" for the "maxint" test; an overflow exception or implicit type change in the case of the latter shouldn't be tolerated for the former. > That means that your method is too complex. Would you consider the Python `requests` library's `get` method to be too complex? It has 10 parameters to it: 4 strings, 4 dictionaries, 1 file-like object, and 1 tuple/AuthHandler option. If you conservatively say there are 5 test cases which should be tested for each parameter (a very low count, especially with strings and dicts), there's approximately 1,125 test cases that should be explored. Not quite a trillion, but it's still a lot.
    - lowbloodsugar 9y ago
      Without getting into the pros and cons of python (which will likely get me downvoted to oblivion), 'get' is an example of what I would call a coordination method. I coach my teams to think of two kinds of methods: methods that do data processing, and methods that coordinate. To test a coordination method, you don't need to care about the value of the things it has to deal with. Maybe you care the they are not null, if your language has nulls. But other than that, what you are testing is that if your method A is supposed to pass arguments 1 thru 5 to method B, and 4 thru 10 to method C, then that's what you test. You don't need to test that argument 1 is a positive integer. The test for method B will do that, if it needs to. And finally you might test that if we expect the result of method B to be passed to method C, the test for our method A would do that using a mock B. Again, what mock B returned could be any object: we only care that it got passed to C. So it looks like its 1,125, but its actually three. If we have classes that are SOLID, then we tend to see the processing methods and the coordination methods end up in different classes. sessions.py isn't a great example of SOLID, and I would have a hard time writing tests for it. It is certainly not the kind of code you get if it was written with TDD.

falcolas 9y ago

> we can classify the integer into three states: less than zero, zero, greater than zero Err, not quite. We're running on computers, remember. A comprehensive unit test that doesn't go through every possible integer input should still include: maximum and minimum integer values for that size maximum and minimum expected inputs 0, 1, -1 So, there's 7 test cases for a single integral input. And since you bring up banking, let's imagine you're working with a function with two inputs. Since most bugs come about because of the interaction between two variables [0], you'd want to check out each combination. So, 7 possible inputs for the first integer, 7 for the second: 49 test cases. Why bother? You're working for a bank... imagine facing a client whose balance shot negative because of an overflow error and started accruing "overdraft coverage" fees. Back to the original point: when was the last time anybody but the AFL tool wrote the "proper" 49+ unit tests for an "add_to_balance(int, int)" style function, when one test would give you 100% coverage for that function? [0] https://csrc.nist.gov/Projects/Automated-Combinatorial-Testing-for-Software https://csrc.nist.gov/Projects/Automated-Combinatorial-Testi...

lowbloodsugar 9y ago
If you mean "nobody would write 49 tests for a function when they could get away with writing one test and have it show up as 100% code coverage because people are lazy bastards" then I agree, and that's the problem: cargo culting the metrics. Look we have 100% code coverage! We are awesome! If you mean, "no look seriously, you'd need 49 tests for every method because you're a bank", well then, no, you actually need less than 49 (since some are subsets of the others), and in reality even this would get very tiring so you'd write a currency class that can't overflow, and yes, being a bank we would test every case and certainly use automated test generation while we're at it. But if you mean that in general, for non-banks, the classes of values is too large to manually write unit tests for, then we disagree. That means that your method is too complex.
- falcolas 9y ago
  > some are subsets of the others In a purely mathematical sense, yes, this is true. But with computers those subsets matter. For example, I wouldn't drop the "max expected" for the "maxint" test; an overflow exception or implicit type change in the case of the latter shouldn't be tolerated for the former. > That means that your method is too complex. Would you consider the Python `requests` library's `get` method to be too complex? It has 10 parameters to it: 4 strings, 4 dictionaries, 1 file-like object, and 1 tuple/AuthHandler option. If you conservatively say there are 5 test cases which should be tested for each parameter (a very low count, especially with strings and dicts), there's approximately 1,125 test cases that should be explored. Not quite a trillion, but it's still a lot.
  - lowbloodsugar 9y ago
    Without getting into the pros and cons of python (which will likely get me downvoted to oblivion), 'get' is an example of what I would call a coordination method. I coach my teams to think of two kinds of methods: methods that do data processing, and methods that coordinate. To test a coordination method, you don't need to care about the value of the things it has to deal with. Maybe you care the they are not null, if your language has nulls. But other than that, what you are testing is that if your method A is supposed to pass arguments 1 thru 5 to method B, and 4 thru 10 to method C, then that's what you test. You don't need to test that argument 1 is a positive integer. The test for method B will do that, if it needs to. And finally you might test that if we expect the result of method B to be passed to method C, the test for our method A would do that using a mock B. Again, what mock B returned could be any object: we only care that it got passed to C. So it looks like its 1,125, but its actually three. If we have classes that are SOLID, then we tend to see the processing methods and the coordination methods end up in different classes. sessions.py isn't a great example of SOLID, and I would have a hard time writing tests for it. It is certainly not the kind of code you get if it was written with TDD.