8 ms·

Tests Should be Specific [video]

55 points by KentBeck 7y ago

2rsf 7y ago
This is also important for system, integration, e2e or any other complex test. The problem is that it is very hard to filter out failures due to environment or other unrelated issues, and even harder to pinpoint the problem
- mihaigalos 7y ago
  High-level testing suffers from this symptom, but catches overarching system problems. It usually means that you are missing a test at unit (spec), and possibly at contract/collaboration level.
cammil 7y ago
Not sure I get the point of this
bump64 7y ago
Specific tests doesn't exactly mean "Simple" tests. It is hard to balance between the two but from my experience when people try want to write specific tests they just start writing extremely simple tests.
cammil 7y ago
They seem to start talking about stacks and leaves of stacks, as if one should only test at leaves. Surely this is woefully inaccurate? Isn't the point of tests to test behaviour? Sure be specific about behaviour. And sure, unit tests should not overlap too much, but this is a separate matter to multiple tests failing, and now I don't know where my bug is. ?
- ptx 7y ago
  I think they were contrasting the first test (a more specific test of a leaf function, i.e the innermost parts of the app) with second test (a less specific test because it fails when either the outermost or innermost part is broken) and saying that to diagnose the second kind of test failure, you can make note of the fact that the more specific test is failing as well and start your debugging in that innermost part of the app (the leaf of the callstack).
- bluGill 7y ago
  The purpose of a test is to state that "This should NEVER change". Thus you need to figure out what will NEVER change before you can write any tests. You can use architecture to guide you - if you have a horizontal layer or vertical slice - either you know that whatever your layer/slice's interface to the other layer/slices will be hard to change and so you can always safely test there. The problem is your layer/slice is complex (if it isn't then your architecture is too inflexible!) and so you need to pick internal points to test. These internal points are then asserted they won't change - but you can legally refactor them latter if only you didn't have those pesky tests that would fail on conditions that might still be important. Where to inject tests is an extremely hard problem.
azangru 7y ago
I would like to hear some words of wisdom from either your guys personal experience or maybe opined by by the gurus about whether or not to use random data in unit tests. On the one hand, I understand the argument that random data makes your test irreproducible; so if something breaks the test, it make take a while to figure out exactly what and why went wrong. On the other hand, I feel that hard-coding test data is too restrictive. For example, in the linked video, they have a 40-hour week with a 8 dollars per hour rate, and they expect the result of the calculation to equal 320. An immediate question arises, what's so special about these numbers? Would the test pass if the input were different? What about a 38-hour week and 20 dollars per hour? And so on... What's your take on this?
- 2zcon 7y ago
  I had this exact discussion at work. Say you have test(X). X is a random number in range A to B. Say someone else has test_range(A, B). Instead of running for one input, it runs test for the full range of inputs. Actually, both tests run on the same set of inputs. The difference is that, for the first test, you're not running all of the inputs at once. You're running some of the inputs with your commit, some with a colleague's later commit, some when a customer downloads the program and tries to run it... by making the input random, you're accepting the possibility of merging code that sometimes fails the tests. And, actually, it will take you far more time running code to run the full set of inputs because there's nothing preventing you from running the same test twice. So then I'd ask why you're not just running the full set before you commit. If it's too expensive to run, my opinion is one or more of: A) You should be running fuzzing 24/7 B) You're using randomness as a means of avoiding deciding the inputs yourself because you don't the problem space C) There's not actually a need to test the full set, X being 222348 is isomorphic to X being 222349. >On the one hand, I understand the argument that random data makes your test irreproducible; so if something breaks the test, it make take a while to figure out exactly what and why fails the test. Usually I see random tests saving the seed if they fail.
  - RHSeeger 7y ago
    > B) You're using randomness as a means of avoiding deciding the inputs yourself because you don't the problem space This is one you can definitely improve on. Learning how to partition your inputs into different groups/types that are likely bring out different behavior/issues/bugs with the code is important. If you have an [add] method over integers, testing for [+,+], [+,-], [-,+], [-,-], [0,+], [0,-], [+,0], [-,0], and [0,0] are reasonable. The more complex the method, the more partitions you can wind up with; which also teaches building smaller methods.
rglover 7y ago
If anybody is interested, this appears to be part of a series: https://www.youtube.com/watch?v=5LOdKDqdWYU&list=PLlmVY7qtgT_lkbrk9iZNizp978mVzpBKl https://www.youtube.com/watch?v=5LOdKDqdWYU&list=PLlmVY7qtgT...
jve 7y ago
I rather prefer higher level tests (Integration tests?) that says, hey, you have an error. Or user will experience error. Or the expected high-level outcome is wrong. Because with unit testing, I miss and can miss so much conditions. If I have high level error, sure, I will figure it out - but it is part of development process rather than something breaks live and then you have to figure that out anyway. Ofcourse Unit tests have their place where I want to test that this input produces particular output (for example some parser, sanitizer class etc.) and I want to receive a signal when my future development breaks some stuff. Tldr: I don't think that having a very specific "what went wrong" is that important. I'm grateful when tests fail, because that saves me from mistakes going into production. It's like a safety net.
- dgrant 7y ago
  You're right - high level integration tests (end to end) are important, and most would argue that they are more important. If you had to choose one type of testing only, that would be it. Unit tests are also important, as you said.
- bluGill 7y ago
  I too prefer higher level tests, but for a different reason: the deep unit tests tend to make it harder to change the code in non-functional ways. If I decide my program structure is wrong most of my unit tests will not apply to the new structure, but the high level tests need to pass.

55 points by KentBeck 7y ago

2rsf 7y ago

This is also important for system, integration, e2e or any other complex test. The problem is that it is very hard to filter out failures due to environment or other unrelated issues, and even harder to pinpoint the problem

mihaigalos 7y ago
High-level testing suffers from this symptom, but catches overarching system problems. It usually means that you are missing a test at unit (spec), and possibly at contract/collaboration level.

cammil 7y ago

Not sure I get the point of this

bump64 7y ago

Specific tests doesn't exactly mean "Simple" tests. It is hard to balance between the two but from my experience when people try want to write specific tests they just start writing extremely simple tests.

cammil 7y ago

They seem to start talking about stacks and leaves of stacks, as if one should only test at leaves. Surely this is woefully inaccurate? Isn't the point of tests to test behaviour? Sure be specific about behaviour. And sure, unit tests should not overlap too much, but this is a separate matter to multiple tests failing, and now I don't know where my bug is. ?

ptx 7y ago
I think they were contrasting the first test (a more specific test of a leaf function, i.e the innermost parts of the app) with second test (a less specific test because it fails when either the outermost or innermost part is broken) and saying that to diagnose the second kind of test failure, you can make note of the fact that the more specific test is failing as well and start your debugging in that innermost part of the app (the leaf of the callstack).
bluGill 7y ago
The purpose of a test is to state that "This should NEVER change". Thus you need to figure out what will NEVER change before you can write any tests. You can use architecture to guide you - if you have a horizontal layer or vertical slice - either you know that whatever your layer/slice's interface to the other layer/slices will be hard to change and so you can always safely test there. The problem is your layer/slice is complex (if it isn't then your architecture is too inflexible!) and so you need to pick internal points to test. These internal points are then asserted they won't change - but you can legally refactor them latter if only you didn't have those pesky tests that would fail on conditions that might still be important. Where to inject tests is an extremely hard problem.

azangru 7y ago

I would like to hear some words of wisdom from either your guys personal experience or maybe opined by by the gurus about whether or not to use random data in unit tests. On the one hand, I understand the argument that random data makes your test irreproducible; so if something breaks the test, it make take a while to figure out exactly what and why went wrong. On the other hand, I feel that hard-coding test data is too restrictive. For example, in the linked video, they have a 40-hour week with a 8 dollars per hour rate, and they expect the result of the calculation to equal 320. An immediate question arises, what's so special about these numbers? Would the test pass if the input were different? What about a 38-hour week and 20 dollars per hour? And so on... What's your take on this?

2zcon 7y ago
I had this exact discussion at work. Say you have test(X). X is a random number in range A to B. Say someone else has test_range(A, B). Instead of running for one input, it runs test for the full range of inputs. Actually, both tests run on the same set of inputs. The difference is that, for the first test, you're not running all of the inputs at once. You're running some of the inputs with your commit, some with a colleague's later commit, some when a customer downloads the program and tries to run it... by making the input random, you're accepting the possibility of merging code that sometimes fails the tests. And, actually, it will take you far more time running code to run the full set of inputs because there's nothing preventing you from running the same test twice. So then I'd ask why you're not just running the full set before you commit. If it's too expensive to run, my opinion is one or more of: A) You should be running fuzzing 24/7 B) You're using randomness as a means of avoiding deciding the inputs yourself because you don't the problem space C) There's not actually a need to test the full set, X being 222348 is isomorphic to X being 222349. >On the one hand, I understand the argument that random data makes your test irreproducible; so if something breaks the test, it make take a while to figure out exactly what and why fails the test. Usually I see random tests saving the seed if they fail.
- RHSeeger 7y ago
  > B) You're using randomness as a means of avoiding deciding the inputs yourself because you don't the problem space This is one you can definitely improve on. Learning how to partition your inputs into different groups/types that are likely bring out different behavior/issues/bugs with the code is important. If you have an [add] method over integers, testing for [+,+], [+,-], [-,+], [-,-], [0,+], [0,-], [+,0], [-,0], and [0,0] are reasonable. The more complex the method, the more partitions you can wind up with; which also teaches building smaller methods.

rglover 7y ago

If anybody is interested, this appears to be part of a series: https://www.youtube.com/watch?v=5LOdKDqdWYU&list=PLlmVY7qtgT_lkbrk9iZNizp978mVzpBKl https://www.youtube.com/watch?v=5LOdKDqdWYU&list=PLlmVY7qtgT...

jve 7y ago

I rather prefer higher level tests (Integration tests?) that says, hey, you have an error. Or user will experience error. Or the expected high-level outcome is wrong. Because with unit testing, I miss and can miss so much conditions. If I have high level error, sure, I will figure it out - but it is part of development process rather than something breaks live and then you have to figure that out anyway. Ofcourse Unit tests have their place where I want to test that this input produces particular output (for example some parser, sanitizer class etc.) and I want to receive a signal when my future development breaks some stuff. Tldr: I don't think that having a very specific "what went wrong" is that important. I'm grateful when tests fail, because that saves me from mistakes going into production. It's like a safety net.

dgrant 7y ago
You're right - high level integration tests (end to end) are important, and most would argue that they are more important. If you had to choose one type of testing only, that would be it. Unit tests are also important, as you said.
bluGill 7y ago
I too prefer higher level tests, but for a different reason: the deep unit tests tend to make it harder to change the code in non-functional ways. If I decide my program structure is wrong most of my unit tests will not apply to the new structure, but the high level tests need to pass.