6 ms·

> "Use Fixed Data Instead of Randomized Data Avoid randomized data as it can lead to toggling tests which can be hard to debug and omit error messages that make

> "Use Fixed Data Instead of Randomized Data Avoid randomized data as it can lead to toggling tests which can be hard to debug and omit error messages that make tracing the error back to the code harder. They will create highly reproducible tests, which are easy to debug and create error messages that can be easily traced back to the relevant line of code." Good article, but I don't get this one at all; it almost seems like an anti-pattern. Choosing fixed data instead of random because the results are more "reproducible" seems to miss the point. If random data eventually helps uncover more bugs, then it's worth using!

Something1234 6y ago
It also adds in the assumption that your randomized setup is correct. That is not always the case, and in a lot of situations it's not trivial.
jayd16 6y ago
You should test the edge cases explicitly instead of hoping the randomization will save. If there's some bounded set of values, like an enum, then test every value instead of randomly picking things and hoping for the worst. I don't want to know eventually. I want to know now.
- chriswarbo 6y ago
  > I don't want to know eventually. I want to know now. That would be nice, but it's not the choice we're facing. The choice is between knowing eventually (by randomising) or knowing never (with determinism); or alternatively, between definitely finding out at 3AM when there's a production outage, or possibly finding out during testing. Test suites get run a lot, so even a small chance of spotting cases we hadn't thought of can be worthwhile. Also, it's much easier to run a randomised test with specific examples than it is to run a hard-coded test with randomised inputs (this is because "randomised tests" are actually parameterised tests, which take test data as arguments). Hence we might as well write the randomised version, then also call it with a bunch of known edge-cases (in QuickCheck-style frameworks this is just a function call, in Hypothesis we can also use the '@example' decorator). If we go down this route there are also automated approaches to make things easier, e.g. Hypothesis maintains a database of counterexamples that it's found in the past, which it mixes into its random data generators. We can ask for these values and use them as explicit examples if we like. In my experience writing "randomised tests" (i.e. property checking) is much easier and far more powerful than writing lots of hard-coded examples. I've done this in Haskell with QuickCheck, Scala with Scalacheck, Python with Hypothesis, Javascript with JSVerify and I hand-rolled a simple framework when I wrote PHP many years ago. Occasionally I find the urge to sprinkle a few hard-coded tests into the suite, but it rarely seems worth it.
  - jayd16 6y ago
    I'm not sure who told you that non-random tests means non-parameterized tests but I don't think you'll get a lot of push back on parameterized tests from any of the commentators here. The point is you should write exhaustive tests such that you couldn't imagine the randomized test finding anything new, especially on bounded sets of inputs. If you're not writing exhaustive tests because of a random strategy then yes, the choice is exactly between knowing immediately or later. I've seen novices assume random tests are exhaustive even when they could think of several edge cases.
OnlyOneCannolo 6y ago
I think better advice would be to use random data as a supplement, rather than as a replacement for fixed-data tests.
- lubonay 6y ago
  +1. I personally like using random data if the test is concerned with only one input, but I avoid random data when checking e.g. pairs of numbers because of the slight chance that the random numbers would be unexpectedly equal.
theptip 6y ago
I think the general idea is that you should be picking your input data to be as evil as possible, so that you're explicitly testing the edge cases; if you use random data you're not thinking about this aspect. Fuzz testing is a separate thing, and should be done too. But it's (in my experience) supplemental to well-chosen hand-crafted input. It's hard to do TDD or ensure you have all your edge-cases covered if you're using randomized inputs. For example, for datetimes, you can pick something nasty and unlikely like 1999-12-31T23:00 Pacific Time, which will read as a different day, month, and year if you have your timezone logic broken somewhere (e.g. you're parsing it as a naive UTC datetime somewhere). If you test randomly, that "dates at the end of the year parse incorrectly for 8 hours" bug is unlikely to be found.
- tincholio 6y ago
  If you're using generative testing, you'll most likely hit all those edge cases, and then some. Not sure what's the situation like in Java, for doing these types of tests, though, but something Quickcheck-like is a godsend in general.
rzzzt 6y ago
Lucene has randomized test cases, but the framework also prints out the (random) seed used at the beginning, so if you find an interesting scenario, you can reproduce it by using the same seed again.
slavik81 6y ago
Tests that use random input data are much more difficult to write correctly. Your test needs to know the expected result not just for one case, but for every possible case. That will vastly increase the number of bugs in your test code, leading to a seemingly endless stream of false-positives. The worst part is that the feedback is late. The test will randomly fail long after it was written. There's a lot of needless overhead in relearning the context of the failing code just so you can fix a test that is not general enough for the data it was provided. There are ways to effectively use randomly-generated test data, but it's harder than you'd think to do it right.
- chriswarbo 6y ago
  Tests with random inputs can be much easier to write. For example here's a test for a single hard-coded example: public bool canSetEmail() { User u = new User( "John", "Smith", new Date(2000, Date.JANUARY, 1), new Email("john@example.com"), Password.hash("password123", new Password.Salt("abc")) ); String newEmail = new Email("smith@example.com"); u.setEmail(newEmail); return u.getEmail().equals(newEmail); } Phew! Here's a randomised alternative: public bool canSetEmail(User u, Email newEmail) { u.setEmail(newEmail); return u.getEmail().equals(newEmail); } Not only is the test logic simpler and clearer, it's much more general. As an added bonus, when we write the data generators for User, Email, etc. we can include a whole load of nasty edge cases and they'll be used by all of our tests. I've not used Java for a while, but in Scalacheck I'd do something like this: def genEmail = { val genStr = arbitrary[String] .map(_.replace("@", "")) .retryUntil(_.nonEmpty) for { user <- genStr domain <- genStr tlds <- Gen.listOf(genStr) } yield (user + "@" + (domain :: tlds).mkString(".")) } Even using the default String generators, this still gives pretty good examples, e.g. scala> genEmail.sample res1: Option[String] = Some(뀩貲㳱誷Ⓣ壟獝鈂ᗘ䮕鹍尛斿績線孞왁궽偔ቫ﯃쥑瞴䒨䏵艥꿊狿냩쩈簋㷡泪伺鑬鮘䦚벛妹乘饡㙧ꐤ큫ᯕ肳铬呢ூ靪୒틯鏱滄㧄Ｐ莶寕상䐀鹭ᚿ㌤ᇹ䶂攔ꅥ㶓⽌ꅂ뀏뛶盏⍸㇗ɧ⇽줓嵛@퇐䎁鱎馉琿䏍㔰蘿⠶㘮큵휕炠᭑㯠ꇷ氕瑤镦碋䓯鄓헛㥝籆찷舊⸦䁜⯞௵籋㟨㨴鯅鿸刡㽴ﮈ耾刏碓辁亁ᦨ氦ব꧓ꌝ飖䒱찮刍ฤ﯉⯛ஃ颰溯쨚ই媐䄇延醴熜䢐필㉕徫澬폁횱坠줊埬າ⟂.὆蹵箅䀈떥喹뭡㘺ꐟ⯉쉂㯻牏䢺梘쭸칹总⻎恵꣥ﴈ宎嶲ꎮ쌊䯧挤ꓯ⻯ꢠ鏿㰔操駚樅졇੩풻殜ᶝ팱.瘽៣뿔뱡䞿愝滠蛊妏뷩먰⦹挸긖᩿㣽ﮀ嫶ꚸ㗃鈴䇡⋥梏ڝ晡烡.줼欺ࠞ娫) The other advantage is that automated shrinking does a pretty good job of homing in on bugs. For example, if our test breaks when there's no top-level domain (e.g. 'root@localhost') then (a) that will be found pretty quickly, since 'tlds' will begin with a high probability of being empty and (b) the other components will be shrunk as far as possible, e.g. 'user' and 'domain' will shrink down to a single null byte (the "smallest" String which satisfies out 'nonEmpty' test); hence we'll be told that this test fails for \0@\0 (the simplest counterexample). We'll also be given the random seeds used by the generators.
asgard1024 6y ago
Actually, you can get reproducible tests even with randomized tests, by setting the random seed (I always do that in e.g. Quickcheck). So this is really a moot point, not a good excuse not to do random testing (or not to use QuickCheck ;-P).
- twic 6y ago
  I don't believe any test frameworks in Java have built-in support for randomisation using a seed, so this is a foreign concept to most Java programmers. Which is a shame, because it's useful. It would actually be really easy to package up seeded randomisation as a JUnit rule / extension. As far as i can tell, nobody has done that.
deleted 6y ago
[deleted]
mabbo 6y ago
Reproducibility is a subtle but important part of all parts of a build, including tests. What happens, in my own experience, is that randomized data in tests has a few unplanned downsides. First, it makes you think your testing all possible values (since they all might be used, right?). But you aren't- you'll never run the test enough to see all values. And what you really should do is think for a few minutes about the edge cases and ranges and write multiple tests covering each edge. The second downside I saw was "oh the tests failed so I just reran it and now it passes". This is especially difficult as your team brings in new people, less experienced people. Everyone is in a rush and yeah, that test fails like 1 in 30 of its runs but we don't have time to look into that right now. So you end up with this frustrating build that lets you continue when you have bugs, but sometimes fails. Your unit tests are a spec. They specify your expectations of the system under test. If you use random data, then the only way you can make your spec accurate is to reimplement the program in your unit tests, also matching the spec. But this will incredibly tightly couple your code and your test, as changing one implementation will require you to change the other. Tests shouldn't be like that, imho.
mumblemumble 6y ago
I have seen that peoples response to randomized testing hinges on the answer to one failing question: Does the test log the input that led to a failure? If the answer is no, then it's an immediate slam on the brakes. Tests aren't reproducible, they're flaky, this is impossible, etc. If the answer is yes, then it's a revelation. The tests are finding me corner and edge cases I never would have thought of on my own, and I can enshrine and document them by creating a fixed-data unit test to cover them even before I start to work the bug (I am still doing TDD, after all), and hey, maybe I don't even have to think that much about the specific data, just its shape, and that will keep me honest and protect me from accidentally writing tautological tests. A short while later, you discover property testing, start achieving ever increasing test coverage with ever decreasing test code, have some sort of satori moment, tell your friends what you've discovered, and they tell you you're insane, randomize tests are flaky by definition, and flaky tests are Taboo. You slink away and change your legal name to Jonathan Livingston Tester.

Something1234 6y ago

It also adds in the assumption that your randomized setup is correct. That is not always the case, and in a lot of situations it's not trivial.

jayd16 6y ago

You should test the edge cases explicitly instead of hoping the randomization will save. If there's some bounded set of values, like an enum, then test every value instead of randomly picking things and hoping for the worst. I don't want to know eventually. I want to know now.

chriswarbo 6y ago
> I don't want to know eventually. I want to know now. That would be nice, but it's not the choice we're facing. The choice is between knowing eventually (by randomising) or knowing never (with determinism); or alternatively, between definitely finding out at 3AM when there's a production outage, or possibly finding out during testing. Test suites get run a lot, so even a small chance of spotting cases we hadn't thought of can be worthwhile. Also, it's much easier to run a randomised test with specific examples than it is to run a hard-coded test with randomised inputs (this is because "randomised tests" are actually parameterised tests, which take test data as arguments). Hence we might as well write the randomised version, then also call it with a bunch of known edge-cases (in QuickCheck-style frameworks this is just a function call, in Hypothesis we can also use the '@example' decorator). If we go down this route there are also automated approaches to make things easier, e.g. Hypothesis maintains a database of counterexamples that it's found in the past, which it mixes into its random data generators. We can ask for these values and use them as explicit examples if we like. In my experience writing "randomised tests" (i.e. property checking) is much easier and far more powerful than writing lots of hard-coded examples. I've done this in Haskell with QuickCheck, Scala with Scalacheck, Python with Hypothesis, Javascript with JSVerify and I hand-rolled a simple framework when I wrote PHP many years ago. Occasionally I find the urge to sprinkle a few hard-coded tests into the suite, but it rarely seems worth it.
- jayd16 6y ago
  I'm not sure who told you that non-random tests means non-parameterized tests but I don't think you'll get a lot of push back on parameterized tests from any of the commentators here. The point is you should write exhaustive tests such that you couldn't imagine the randomized test finding anything new, especially on bounded sets of inputs. If you're not writing exhaustive tests because of a random strategy then yes, the choice is exactly between knowing immediately or later. I've seen novices assume random tests are exhaustive even when they could think of several edge cases.

OnlyOneCannolo 6y ago

I think better advice would be to use random data as a supplement, rather than as a replacement for fixed-data tests.

lubonay 6y ago
+1. I personally like using random data if the test is concerned with only one input, but I avoid random data when checking e.g. pairs of numbers because of the slight chance that the random numbers would be unexpectedly equal.

theptip 6y ago

I think the general idea is that you should be picking your input data to be as evil as possible, so that you're explicitly testing the edge cases; if you use random data you're not thinking about this aspect. Fuzz testing is a separate thing, and should be done too. But it's (in my experience) supplemental to well-chosen hand-crafted input. It's hard to do TDD or ensure you have all your edge-cases covered if you're using randomized inputs. For example, for datetimes, you can pick something nasty and unlikely like 1999-12-31T23:00 Pacific Time, which will read as a different day, month, and year if you have your timezone logic broken somewhere (e.g. you're parsing it as a naive UTC datetime somewhere). If you test randomly, that "dates at the end of the year parse incorrectly for 8 hours" bug is unlikely to be found.

tincholio 6y ago
If you're using generative testing, you'll most likely hit all those edge cases, and then some. Not sure what's the situation like in Java, for doing these types of tests, though, but something Quickcheck-like is a godsend in general.

rzzzt 6y ago

Lucene has randomized test cases, but the framework also prints out the (random) seed used at the beginning, so if you find an interesting scenario, you can reproduce it by using the same seed again.

slavik81 6y ago

Tests that use random input data are much more difficult to write correctly. Your test needs to know the expected result not just for one case, but for every possible case. That will vastly increase the number of bugs in your test code, leading to a seemingly endless stream of false-positives. The worst part is that the feedback is late. The test will randomly fail long after it was written. There's a lot of needless overhead in relearning the context of the failing code just so you can fix a test that is not general enough for the data it was provided. There are ways to effectively use randomly-generated test data, but it's harder than you'd think to do it right.

chriswarbo 6y ago
Tests with random inputs can be much easier to write. For example here's a test for a single hard-coded example: public bool canSetEmail() { User u = new User( "John", "Smith", new Date(2000, Date.JANUARY, 1), new Email("john@example.com"), Password.hash("password123", new Password.Salt("abc")) ); String newEmail = new Email("smith@example.com"); u.setEmail(newEmail); return u.getEmail().equals(newEmail); } Phew! Here's a randomised alternative: public bool canSetEmail(User u, Email newEmail) { u.setEmail(newEmail); return u.getEmail().equals(newEmail); } Not only is the test logic simpler and clearer, it's much more general. As an added bonus, when we write the data generators for User, Email, etc. we can include a whole load of nasty edge cases and they'll be used by all of our tests. I've not used Java for a while, but in Scalacheck I'd do something like this: def genEmail = { val genStr = arbitrary[String] .map(_.replace("@", "")) .retryUntil(_.nonEmpty) for { user <- genStr domain <- genStr tlds <- Gen.listOf(genStr) } yield (user + "@" + (domain :: tlds).mkString(".")) } Even using the default String generators, this still gives pretty good examples, e.g. scala> genEmail.sample res1: Option[String] = Some(뀩貲㳱誷Ⓣ壟獝鈂ᗘ䮕鹍尛斿績線孞왁궽偔ቫ﯃쥑瞴䒨䏵艥꿊狿냩쩈簋㷡泪伺鑬鮘䦚벛妹乘饡㙧ꐤ큫ᯕ肳铬呢ூ靪୒틯鏱滄㧄Ｐ莶寕상䐀鹭ᚿ㌤ᇹ䶂攔ꅥ㶓⽌ꅂ뀏뛶盏⍸㇗ɧ⇽줓嵛@퇐䎁鱎馉琿䏍㔰蘿⠶㘮큵휕炠᭑㯠ꇷ氕瑤镦碋䓯鄓헛㥝籆찷舊⸦䁜⯞௵籋㟨㨴鯅鿸刡㽴ﮈ耾刏碓辁亁ᦨ氦ব꧓ꌝ飖䒱찮刍ฤ﯉⯛ஃ颰溯쨚ই媐䄇延醴熜䢐필㉕徫澬폁횱坠줊埬າ⟂.὆蹵箅䀈떥喹뭡㘺ꐟ⯉쉂㯻牏䢺梘쭸칹总⻎恵꣥ﴈ宎嶲ꎮ쌊䯧挤ꓯ⻯ꢠ鏿㰔操駚樅졇੩풻殜ᶝ팱.瘽៣뿔뱡䞿愝滠蛊妏뷩먰⦹挸긖᩿㣽ﮀ嫶ꚸ㗃鈴䇡⋥梏ڝ晡烡.줼欺ࠞ娫) The other advantage is that automated shrinking does a pretty good job of homing in on bugs. For example, if our test breaks when there's no top-level domain (e.g. 'root@localhost') then (a) that will be found pretty quickly, since 'tlds' will begin with a high probability of being empty and (b) the other components will be shrunk as far as possible, e.g. 'user' and 'domain' will shrink down to a single null byte (the "smallest" String which satisfies out 'nonEmpty' test); hence we'll be told that this test fails for \0@\0 (the simplest counterexample). We'll also be given the random seeds used by the generators.

asgard1024 6y ago

Actually, you can get reproducible tests even with randomized tests, by setting the random seed (I always do that in e.g. Quickcheck). So this is really a moot point, not a good excuse not to do random testing (or not to use QuickCheck ;-P).

twic 6y ago
I don't believe any test frameworks in Java have built-in support for randomisation using a seed, so this is a foreign concept to most Java programmers. Which is a shame, because it's useful. It would actually be really easy to package up seeded randomisation as a JUnit rule / extension. As far as i can tell, nobody has done that.

deleted 6y ago

[deleted]

mabbo 6y ago

Reproducibility is a subtle but important part of all parts of a build, including tests. What happens, in my own experience, is that randomized data in tests has a few unplanned downsides. First, it makes you think your testing all possible values (since they all might be used, right?). But you aren't- you'll never run the test enough to see all values. And what you really should do is think for a few minutes about the edge cases and ranges and write multiple tests covering each edge. The second downside I saw was "oh the tests failed so I just reran it and now it passes". This is especially difficult as your team brings in new people, less experienced people. Everyone is in a rush and yeah, that test fails like 1 in 30 of its runs but we don't have time to look into that right now. So you end up with this frustrating build that lets you continue when you have bugs, but sometimes fails. Your unit tests are a spec. They specify your expectations of the system under test. If you use random data, then the only way you can make your spec accurate is to reimplement the program in your unit tests, also matching the spec. But this will incredibly tightly couple your code and your test, as changing one implementation will require you to change the other. Tests shouldn't be like that, imho.

mumblemumble 6y ago

I have seen that peoples response to randomized testing hinges on the answer to one failing question: Does the test log the input that led to a failure? If the answer is no, then it's an immediate slam on the brakes. Tests aren't reproducible, they're flaky, this is impossible, etc. If the answer is yes, then it's a revelation. The tests are finding me corner and edge cases I never would have thought of on my own, and I can enshrine and document them by creating a fixed-data unit test to cover them even before I start to work the bug (I am still doing TDD, after all), and hey, maybe I don't even have to think that much about the specific data, just its shape, and that will keep me honest and protect me from accidentally writing tautological tests. A short while later, you discover property testing, start achieving ever increasing test coverage with ever decreasing test code, have some sort of satori moment, tell your friends what you've discovered, and they tell you you're insane, randomize tests are flaky by definition, and flaky tests are Taboo. You slink away and change your legal name to Jonathan Livingston Tester.