11 ms·

Best Practices for Testing in Java (2019)

103 points by moks 6y ago

beders 6y ago
Mandatory link to https://www.youtube.com/watch?v=EZ05e7EMOLM https://www.youtube.com/watch?v=EZ05e7EMOLM It clarifies a few of the points made in the article. Personally I would add: Don't overuse mocks, only use dependency injection on the module level (Ludicrous use of singleton DI will hinder your design, not help it)
commandlinefan 6y ago
I'd be happy if we could just get to the point where programmers wrote code that could be unit tested. i.e. don't write static initializers that connect to live databases for a start.
- eecc 6y ago
  Haven't seen anything that bad in ages, but yeah... I'd lose my cool if I saw that. Last bad thing I saw was - quite recently actually - some business code in a request handler mutating state on a controller field (which are singletons.) Lots of fun once we started load testing with concurrent sessions...
- exabrial 6y ago
  Dependency Injection + Mockito allows one to do that. There's a whole study on writing code that's easy tested. Our rule is no more than 4 layers: initiator, business logic, services, anemic domain model. Initiators abstract away the inbound protocol and serialize to the common model. Business logic controller handle all of the branching. Services effectuate single actions. Services can't call other services. And the domain model is how everything talks. We all build our apps this way and it's really easy to move people between projects. Not perfect but works for about 85% of stuff one has to write.
  - nytgop77 6y ago
    is database access only in service?
    - exabrial 6y ago
      Correct. A business logic controller can not directly corner the outside world, it must talk through a service. This is nice because it prevents leaky abstractions into the controller layer. Actions across services that need to be atomic can be grouped in an XA transaction: // Xa start databaseService.reset password(user); emailService.notifyPasswordChange(user); // Xa commit
  - rmdashrfstar 6y ago
    I'm not sure calling the domain layer "anemic" by default is correct, as it's typically a (negative) descriptions of models which are too data-driven instead of behavior-driven. I would suggest an alternative layer structure: Initiators/Controllers -> Application/Domain/Infrastructure Services -> Domain Models/Business Rules/Invariant Checking
    - exabrial 6y ago
      We've thought about that; the style I've described is very non-OO. It is easy to teach though and it makes unit testing a breeze. I like the architecture you've described. I think we arrived at where did because it's a natural for when using a DI container, which manages state and transaction groups for you.
- Spivak 6y ago
  I'm kinda confused about that one. Because if the value isn't coming from some external source like a db, api, file, whatever then why do you even need a static initalizer? If you mean they hardcoded the prod database then I'm so sorry.
  - commandlinefan 6y ago
    They hardcode the code that reads a config file and gets the host name of the DB and then connects to the DB. This initializes a singleton object that’s accessed by nearly every file in the rest of the system. They do this for every external source, not just databases. In my experience, this is the most common way Java “developers” design systems, and they usually get angry with me if I try to fix it because their way is “faster”.
    - cmckn 6y ago
      I've run into this type of thing too, it's soul-sucking because it's so easy to avoid. I wouldn't say I do TDD per se, but I definitely write tests for many things to prove to myself that it's working. Many devs just build + run the code and poke at it, which enables all types of heinous patterns like this (and in my opinion is a super inefficient way to code).
    - Spivak 6y ago
      So why is this design bad? It seems like when you actually start running into scaling issues that singleton translates naturally into borrowing from a shared connection pool.
      - commandlinefan 6y ago
        ... because nothing can be mocked out.
        Spivak 6y ago
        Do you not just mock the singleton? I mean don't get me wrong, it has all the usual global variable downsides but I don't see too much a meaningful difference between every method passing around the same connection pool handle explicitly vs ambiently. And either you can mock the connection objects the singleton returns or you can't. In most apps I've seen "services" like Redis/Rabbit/Memcache/Postgres/External APIs/Storage are cross-cutting concerns and you make your life a nightmare by having to pass myfunc(actual, params, db, redis, memcache, rabbit,...) because if you realize deep in your call stack you actually need data from Postgres now you have change all the callers recursively to pass the handle down. If you have this global "service catalog" it's almost always to eliminate the need for passing the same effectively global connection pools to hundreds of call-sites. It is annoying that it makes every test require a bunch of ambient service mocks but you only really have to set it up once.
- cgh 6y ago
  Static methods in general that create side-effects suck. PowerMock allows you to deal with these troublesome classes though so unit testing is possible.
  - hackerm0nkey 6y ago
    Yes, surely it's possible. But the resultant test is highly tightly coupled to the implementation of the SUT which gives the test more than one reason to change. We should strive generally to have tests that only change if the business requirements change. But if I want to refactor my unit (whatever that might be) then the test should not change, or at least should not change __much__
- lmm 6y ago
  Where the hell are you working where people think that's ok? I haven't seen anyone defending that for 10+ years.
vemv 6y ago
> KISS > DRY I'd agree elsewhere, but not for tests. Non-DRY tests often means that there's an ad-hoc similarity between a series of tests. Without DRYness, as time passes, domain knowledge fades away, and more maintainers touch the same test code, it becomes quite likely that this similarity will be broken. Which is a perfect recipe for false negatives: one fixes one test but not all other tests, which might start exercising a useless code path due to drift.
- joshka 6y ago
  The more canonical form is DAMP rather than KISS. Descriptive and Meaningful Phrases. DRY is fine in moderation in tests. E.g. instead of `Person.builder().withName("Alice").withAge(100).build();` using `People.grandmother()` is DRY-ish, but certainly something you'd see less of in production code.
r0s 6y ago
Good stuff. I especially appreciate the entreaty to limit assertions and corner cases to one test concern. The only thing I didn't like was the Given When Then recommendation. BDD style story language is really meant for key specification examples, which can be automated in functional tests, but not really appropriate for unit testing.
- noisem4ker 6y ago
  Even for unit testing, I find that a standard arrange-act-assert structure is helpful.
  - rmdashrfstar 6y ago
    While Given When Then rolls off the tongue easier, I think Arrange Act Assert better describes the structure
- joshka 6y ago
  This article says GWT, but all the examples are showing AAA (Arrange Act Assert). Subtle semantic difference, likely not the core of the article though.
- zoover2020 6y ago
  Why not BDD style in unit tests though? Our team adopted it and I really like it over triple A.
  - UK-Al05 6y ago
    I don't really understand his argument against it.
  - lmm 6y ago
    I've found it just obscures what the test is doing. If I'm looking at a test that failed, I want to see the code that caused the failure, as plainly as possible; I'm quite capable of reading code (indeed I find code is better at communicating clearly than English is), so I'd rather see a minimum of ceremony than some misguided effort at making tests more readable for imaginary business users.
bvanderveen 6y ago
Great article. The only thing I would add is that anything other than `assertEquals(expectedValue, actualValue)` is not really permissible. Don't make assertions about 'portion' or 'subset' of your results, make assertions about the entire value! Otherwise, you're throwing away an opportunity to encode a constraint; sooner or later the undefined behavior will make itself known. We are writing tests to avoid that scenario in the first place, no?
- joshka 6y ago
  Partial Disagreement on the phrasing of this - there are many classes of assertion like `assertThat(expected).containsInAnyOrder(foo,bar,baz);` that would not meet this. Order often does not matter and cannot be constrained. There are other things similar to this.
  - bvanderveen 6y ago
    Value should probably be a `Set` then!
    - sk5t 6y ago
      There are ordered sets, immutable sets... infinite sequences, lazy lists, sometimes collections are hairy to test.
      - bvanderveen 6y ago
        If find that if I am trying to test values that are not 'fully realizable' (infinite, lazy, etc) there's typically a temporal aspect involved, and I have a 'mock clock and timer' implementation so that I can call a `time.advance(t)` from within my tests and time advances in my system under test. This works because (as OP also noted) I only use my own mockable abstractions for answering the question 'what time is it?' and for instructing the system 'call me back at time t'. The net effect being I can fully define the behavior of the system within an arbitrary window of time—and collect the resulting values or behavior into some data structure which encodes what happened and when, and then compare it against expected results. Re immutable: the values you're getting from your system under test and the expected values you're constructing in tests should always be immutable!
codeulike 6y ago
"Do programmers have any specific superstitions?" "Yeah, but we call them best practices." (from https://twitter.com/dbgrandi/status/508329463990734848 https://twitter.com/dbgrandi/status/508329463990734848 but thats probably not the original source)
- brabel 6y ago
  Good luck coming up with your own way to write every test, making every decision as if it were the first time!
  - codeulike 6y ago
    I'm happy to take advice, but given how debatable everything is with programming I'm skeptical of anything labelled 'best practice' ... Over the decades, fads come and go, todays best practice is often tomorrows antipattern ...
splittingTimes 6y ago
To easily see in test failures what went wrong we follow the naming convention: UnitOfWorkUnderTest_Scenario_ExpectedBehavior Which mimics the internal GWT structure.
- hackerm0nkey 6y ago
  I love this. I started with trying to have simple, short and descriptive names for the test methods but then it doesn't work quite well with more complicated test cases. So I grew to like something along these lines: given_someInput_someOtherInput_when_doX_then_thisThingHappen_thatThingHappen which has the advantage to be optimised for reading. When I want to skim over the test class quickly I can simply look at the method names to get an idea on what the test is doing and if I want more details I can look further at it's implementation
- joshka 6y ago
  I have suggested the same previously, but the display name approach is better. Now test names should be short: @DisplayName("throws IllegalArgumentException when given bad data") void badData() {
sabujp 6y ago
Java testing at google: * Don't mock if you have access to the remote server code, create fakes using calls to class methods being tested or use an in memory version of the entire thing if possible. I know it's not easy and it takes time, but it's worth it in the long run. If something changes you'll detect errors quickly. * When using guice injections, override explicitly what you need directly in the test setup. Don't create huge test modules with everything and the kitchen sink, eventually these will become unmaintainable and become difficult for future maintainters to understand where overrides and injections are coming from. Integration testing is whole another subject but I would highly recommend it for mission critical production code.
dlandis 6y ago
> "Use Fixed Data Instead of Randomized Data Avoid randomized data as it can lead to toggling tests which can be hard to debug and omit error messages that make tracing the error back to the code harder. They will create highly reproducible tests, which are easy to debug and create error messages that can be easily traced back to the relevant line of code." Good article, but I don't get this one at all; it almost seems like an anti-pattern. Choosing fixed data instead of random because the results are more "reproducible" seems to miss the point. If random data eventually helps uncover more bugs, then it's worth using!
- Something1234 6y ago
  It also adds in the assumption that your randomized setup is correct. That is not always the case, and in a lot of situations it's not trivial.
- jayd16 6y ago
  You should test the edge cases explicitly instead of hoping the randomization will save. If there's some bounded set of values, like an enum, then test every value instead of randomly picking things and hoping for the worst. I don't want to know eventually. I want to know now.
  - chriswarbo 6y ago
    > I don't want to know eventually. I want to know now. That would be nice, but it's not the choice we're facing. The choice is between knowing eventually (by randomising) or knowing never (with determinism); or alternatively, between definitely finding out at 3AM when there's a production outage, or possibly finding out during testing. Test suites get run a lot, so even a small chance of spotting cases we hadn't thought of can be worthwhile. Also, it's much easier to run a randomised test with specific examples than it is to run a hard-coded test with randomised inputs (this is because "randomised tests" are actually parameterised tests, which take test data as arguments). Hence we might as well write the randomised version, then also call it with a bunch of known edge-cases (in QuickCheck-style frameworks this is just a function call, in Hypothesis we can also use the '@example' decorator). If we go down this route there are also automated approaches to make things easier, e.g. Hypothesis maintains a database of counterexamples that it's found in the past, which it mixes into its random data generators. We can ask for these values and use them as explicit examples if we like. In my experience writing "randomised tests" (i.e. property checking) is much easier and far more powerful than writing lots of hard-coded examples. I've done this in Haskell with QuickCheck, Scala with Scalacheck, Python with Hypothesis, Javascript with JSVerify and I hand-rolled a simple framework when I wrote PHP many years ago. Occasionally I find the urge to sprinkle a few hard-coded tests into the suite, but it rarely seems worth it.
closeparen 6y ago
>If this test fails, it’s hard to see what exactly is broken. Breaking existing tests is rare. I don't mind some detective work when it happens. You start with a pretty strong lead, anyway: whatever you touched since the last time the test passed. In my workflow this is unlikely to be more than five minutes ago. Half the time I know what I broke without even looking. I can see how specificity in test results would be important if you're running tests very infrequently and accumulating a lot of change between runs. But in that case I would probably invest in faster feedback before I'd invest in specificity. Huge, wordy test suites are expensive and can ossify bad design much more strongly than they preserve correct behavior.
- ThePadawan 6y ago
  > Breaking existing tests is rare. I don't mind some detective work when it happens. You start with a pretty strong lead, anyway: whatever you touched since the last time the test passed. You describe the exact situation where the advice is not helpful at all. Now imagine a large, very tightly interwoven codebase where in fact, making changes to a section of code breaks tests on the complete other side of the application. If you don't follow that advice, you are immediately blocked, and pretty much at the mercy of someone else fixing their test (probably to be less needlessly specific or less assertive after changing requirements). If they structured their test in a way that makes it clear why it should fail, you have the chance to investigate yourself how this contradicts with your changes.
  - rightbyte 6y ago
    That's a organization problem (bunkers, are they called silos?) not a coding problem. I wish people would specify their coding advice with "advice for big projects with square process", "advice for dysfunctional planning", "advice for small team" etc.
  - throwaway_pdp09 6y ago
    > very tightly interwoven Perhaps this is your problem.
lmilcin 6y ago
Hardly "best" practices. "Best" would suggest there are no better practices. I am fed up with unit testing. There is no feedback to tell if unit testing setup is complete and if unit tests are comprehensive. I am more and more turning to functional regression testing when entire service is tested as a black box against spec. Since we are testing API this tends to be specified much better than individual components. There is usually already some kind of documentation. API changes much less than individual components which means that if I start doing some kind of refactoring I may change a lot of internals but don't touch API at all. If I touch API then I need to update documentation and inform users so writing tests for that is not a huge additional cost.
- chii 6y ago
  > unit testing setup is complete coverage reports. They tell you which branch wasnt executed in your test, which can tell you what may be missing in your unit testing. > more turning to functional regression testing when entire service is tested as a black box against spec. functional/regression testing is important, but it can be difficult to use if your app is huge. And you can not easily isolate an area to test, so you always end up testing a common component repeatedly, and thus is slower to run. Not to mention some functional testing mechanisms (such as using a web browser) can be flaky. Both have their place in the automated testing paradigm.
  - 2rsf 6y ago
    Although coverage reports are misleading, 100% coverage means that you passed through every line of code, that's it. You don't know if you covered edge cases, it cannot cover concurrency well and although possible it will not tell you whether you used a series of values or long-paths in your code.
    - StevenWaterman 6y ago
      You're right about edge cases in terms of input values, and right about issues with concurrency, but many of the issues you mention are solved by changing your metric. Branch-level coverage tells you what percentage of codepaths are covered. In other words, it would report 50% coverage for: if a: print "a yes" else: print "a no" if b: print "b yes" else: print "b no" test 1: a = false, b = false test 2: a = true, b = true
twic 6y ago
Recommending AssertJ and not mentioning Hamcrest? Hamcrest is more flexible, more composable, and more extensible.
- joshka 6y ago
  Having used both extensively, AssertJ's API is more obvious and generally useful out of the box. It's replaced all the places where I'd previously used custom Hamcrest matchers with generally smaller amounts of code that spit out better errors when tests fail. It's not a either or though, sometimes it's worth looking at whether hamcrest is the right approach.
Nihilartikel 6y ago
I'm pretty lazy, but I like testing.. The habit that I've repeated on my last few project is this: Work out a way to gracefully serialize/deserialize data structures in your code into a human readable format, like EDN or pretty printed JSON. (this is easier in more civilized languages... -winks at Clojure-) Pick high level functions that exercise a lot of code for testing. Generate input data to exercise that function either by hand if it's small and tractable, or by instrumenting real runs of the program and outputting serialized copies of that data (to file or console). Write test helpers that call the function with the input data, and depending on an environment variable, read the expected 'golden' data from disk, or if UPDATE_TEST_RESULTS=true, write the function output back to disk with a filename unique to the test, and mark the test as passed. If the test fails, print a nice readable, structural diff of the expected/actual data - e.g. python datadiff, or clojure.data/diff so the exact change is visible. On the first run you set UPDATE_TEST_RESULTS=true and generate the expected test results. Check your 'golden' result data into git. Now you can easily keep expected results up to date even if logic changes, and get a git diff of exactly how the data has changes. Acknowledged that this doesn't speak to side effect riddled code that needs mocks.. and doesn't buy into the build-the-test-first style of development, but I've found it's a good return on investment for regression testing, and provides visibility into subtle logic changes, expected or otherwise, introduced by code changes.
- soedirgo 6y ago
  Are you referring to snapshot testing[0][1]? i.e. you first "snapshot" the output of the function and commit it to VC, and each test run will run the same input and compare it against the "snapshot", failing and giving a diff if it differs. I'm about to try it soon, seems like a good ROI as you said. [0] https://jestjs.io/docs/en/22.x/snapshot-testing https://jestjs.io/docs/en/22.x/snapshot-testing [1] https://github.com/mitsuhiko/insta https://github.com/mitsuhiko/insta
  - Nihilartikel 6y ago
    I think I am! Through convergent evolution at least - I hadn't found essays explicitly advocating it at the time. My use case was comparing results from chains of Spark RDD & Dataframe transformations, so having fairly large realistc input/output datasets was part of the game, and the main reason that manually writing all expected results wasn't feasible.
aabbcc1241 6y ago
The table of content is very helpful. However, I'm curious why this hugo generated static site is consuming over 130MB memory? For reference, HN is consuming 5MB memory only. (the numbers are observed from about:performance in Firefox)

103 points by moks 6y ago

Mandatory link to https://www.youtube.com/watch?v=EZ05e7EMOLM https://www.youtube.com/watch?v=EZ05e7EMOLM It clarifies a few of the points made in the article. Personally I would add: Don't overuse mocks, only use dependency injection on the module level (Ludicrous use of singleton DI will hinder your design, not help it)

commandlinefan 6y ago

I'd be happy if we could just get to the point where programmers wrote code that could be unit tested. i.e. don't write static initializers that connect to live databases for a start.

eecc 6y ago
Haven't seen anything that bad in ages, but yeah... I'd lose my cool if I saw that. Last bad thing I saw was - quite recently actually - some business code in a request handler mutating state on a controller field (which are singletons.) Lots of fun once we started load testing with concurrent sessions...
exabrial 6y ago
Dependency Injection + Mockito allows one to do that. There's a whole study on writing code that's easy tested. Our rule is no more than 4 layers: initiator, business logic, services, anemic domain model. Initiators abstract away the inbound protocol and serialize to the common model. Business logic controller handle all of the branching. Services effectuate single actions. Services can't call other services. And the domain model is how everything talks. We all build our apps this way and it's really easy to move people between projects. Not perfect but works for about 85% of stuff one has to write.
- nytgop77 6y ago
  is database access only in service?
  - exabrial 6y ago
    Correct. A business logic controller can not directly corner the outside world, it must talk through a service. This is nice because it prevents leaky abstractions into the controller layer. Actions across services that need to be atomic can be grouped in an XA transaction: // Xa start databaseService.reset password(user); emailService.notifyPasswordChange(user); // Xa commit
- rmdashrfstar 6y ago
  I'm not sure calling the domain layer "anemic" by default is correct, as it's typically a (negative) descriptions of models which are too data-driven instead of behavior-driven. I would suggest an alternative layer structure: Initiators/Controllers -> Application/Domain/Infrastructure Services -> Domain Models/Business Rules/Invariant Checking
  - exabrial 6y ago
    We've thought about that; the style I've described is very non-OO. It is easy to teach though and it makes unit testing a breeze. I like the architecture you've described. I think we arrived at where did because it's a natural for when using a DI container, which manages state and transaction groups for you.
Spivak 6y ago
I'm kinda confused about that one. Because if the value isn't coming from some external source like a db, api, file, whatever then why do you even need a static initalizer? If you mean they hardcoded the prod database then I'm so sorry.
- commandlinefan 6y ago
  They hardcode the code that reads a config file and gets the host name of the DB and then connects to the DB. This initializes a singleton object that’s accessed by nearly every file in the rest of the system. They do this for every external source, not just databases. In my experience, this is the most common way Java “developers” design systems, and they usually get angry with me if I try to fix it because their way is “faster”.
  - cmckn 6y ago
    I've run into this type of thing too, it's soul-sucking because it's so easy to avoid. I wouldn't say I do TDD per se, but I definitely write tests for many things to prove to myself that it's working. Many devs just build + run the code and poke at it, which enables all types of heinous patterns like this (and in my opinion is a super inefficient way to code).
  - Spivak 6y ago
    So why is this design bad? It seems like when you actually start running into scaling issues that singleton translates naturally into borrowing from a shared connection pool.
    - commandlinefan 6y ago
      ... because nothing can be mocked out.
      - Spivak 6y ago
        Do you not just mock the singleton? I mean don't get me wrong, it has all the usual global variable downsides but I don't see too much a meaningful difference between every method passing around the same connection pool handle explicitly vs ambiently. And either you can mock the connection objects the singleton returns or you can't. In most apps I've seen "services" like Redis/Rabbit/Memcache/Postgres/External APIs/Storage are cross-cutting concerns and you make your life a nightmare by having to pass myfunc(actual, params, db, redis, memcache, rabbit,...) because if you realize deep in your call stack you actually need data from Postgres now you have change all the callers recursively to pass the handle down. If you have this global "service catalog" it's almost always to eliminate the need for passing the same effectively global connection pools to hundreds of call-sites. It is annoying that it makes every test require a bunch of ambient service mocks but you only really have to set it up once.
cgh 6y ago
Static methods in general that create side-effects suck. PowerMock allows you to deal with these troublesome classes though so unit testing is possible.
- hackerm0nkey 6y ago
  Yes, surely it's possible. But the resultant test is highly tightly coupled to the implementation of the SUT which gives the test more than one reason to change. We should strive generally to have tests that only change if the business requirements change. But if I want to refactor my unit (whatever that might be) then the test should not change, or at least should not change __much__
lmm 6y ago
Where the hell are you working where people think that's ok? I haven't seen anyone defending that for 10+ years.

vemv 6y ago

> KISS > DRY I'd agree elsewhere, but not for tests. Non-DRY tests often means that there's an ad-hoc similarity between a series of tests. Without DRYness, as time passes, domain knowledge fades away, and more maintainers touch the same test code, it becomes quite likely that this similarity will be broken. Which is a perfect recipe for false negatives: one fixes one test but not all other tests, which might start exercising a useless code path due to drift.

joshka 6y ago
The more canonical form is DAMP rather than KISS. Descriptive and Meaningful Phrases. DRY is fine in moderation in tests. E.g. instead of `Person.builder().withName("Alice").withAge(100).build();` using `People.grandmother()` is DRY-ish, but certainly something you'd see less of in production code.

r0s 6y ago

Good stuff. I especially appreciate the entreaty to limit assertions and corner cases to one test concern. The only thing I didn't like was the Given When Then recommendation. BDD style story language is really meant for key specification examples, which can be automated in functional tests, but not really appropriate for unit testing.

noisem4ker 6y ago
Even for unit testing, I find that a standard arrange-act-assert structure is helpful.
- rmdashrfstar 6y ago
  While Given When Then rolls off the tongue easier, I think Arrange Act Assert better describes the structure
joshka 6y ago
This article says GWT, but all the examples are showing AAA (Arrange Act Assert). Subtle semantic difference, likely not the core of the article though.
zoover2020 6y ago
Why not BDD style in unit tests though? Our team adopted it and I really like it over triple A.
- UK-Al05 6y ago
  I don't really understand his argument against it.
- lmm 6y ago
  I've found it just obscures what the test is doing. If I'm looking at a test that failed, I want to see the code that caused the failure, as plainly as possible; I'm quite capable of reading code (indeed I find code is better at communicating clearly than English is), so I'd rather see a minimum of ceremony than some misguided effort at making tests more readable for imaginary business users.

bvanderveen 6y ago

Great article. The only thing I would add is that anything other than `assertEquals(expectedValue, actualValue)` is not really permissible. Don't make assertions about 'portion' or 'subset' of your results, make assertions about the entire value! Otherwise, you're throwing away an opportunity to encode a constraint; sooner or later the undefined behavior will make itself known. We are writing tests to avoid that scenario in the first place, no?

joshka 6y ago
Partial Disagreement on the phrasing of this - there are many classes of assertion like `assertThat(expected).containsInAnyOrder(foo,bar,baz);` that would not meet this. Order often does not matter and cannot be constrained. There are other things similar to this.
- bvanderveen 6y ago
  Value should probably be a `Set` then!
  - sk5t 6y ago
    There are ordered sets, immutable sets... infinite sequences, lazy lists, sometimes collections are hairy to test.
    - bvanderveen 6y ago
      If find that if I am trying to test values that are not 'fully realizable' (infinite, lazy, etc) there's typically a temporal aspect involved, and I have a 'mock clock and timer' implementation so that I can call a `time.advance(t)` from within my tests and time advances in my system under test. This works because (as OP also noted) I only use my own mockable abstractions for answering the question 'what time is it?' and for instructing the system 'call me back at time t'. The net effect being I can fully define the behavior of the system within an arbitrary window of time—and collect the resulting values or behavior into some data structure which encodes what happened and when, and then compare it against expected results. Re immutable: the values you're getting from your system under test and the expected values you're constructing in tests should always be immutable!

codeulike 6y ago

"Do programmers have any specific superstitions?" "Yeah, but we call them best practices." (from https://twitter.com/dbgrandi/status/508329463990734848 https://twitter.com/dbgrandi/status/508329463990734848 but thats probably not the original source)

brabel 6y ago
Good luck coming up with your own way to write every test, making every decision as if it were the first time!
- codeulike 6y ago
  I'm happy to take advice, but given how debatable everything is with programming I'm skeptical of anything labelled 'best practice' ... Over the decades, fads come and go, todays best practice is often tomorrows antipattern ...

splittingTimes 6y ago

To easily see in test failures what went wrong we follow the naming convention: UnitOfWorkUnderTest_Scenario_ExpectedBehavior Which mimics the internal GWT structure.

hackerm0nkey 6y ago
I love this. I started with trying to have simple, short and descriptive names for the test methods but then it doesn't work quite well with more complicated test cases. So I grew to like something along these lines: given_someInput_someOtherInput_when_doX_then_thisThingHappen_thatThingHappen which has the advantage to be optimised for reading. When I want to skim over the test class quickly I can simply look at the method names to get an idea on what the test is doing and if I want more details I can look further at it's implementation
joshka 6y ago
I have suggested the same previously, but the display name approach is better. Now test names should be short: @DisplayName("throws IllegalArgumentException when given bad data") void badData() {

sabujp 6y ago

Java testing at google: * Don't mock if you have access to the remote server code, create fakes using calls to class methods being tested or use an in memory version of the entire thing if possible. I know it's not easy and it takes time, but it's worth it in the long run. If something changes you'll detect errors quickly. * When using guice injections, override explicitly what you need directly in the test setup. Don't create huge test modules with everything and the kitchen sink, eventually these will become unmaintainable and become difficult for future maintainters to understand where overrides and injections are coming from. Integration testing is whole another subject but I would highly recommend it for mission critical production code.

dlandis 6y ago

> "Use Fixed Data Instead of Randomized Data Avoid randomized data as it can lead to toggling tests which can be hard to debug and omit error messages that make tracing the error back to the code harder. They will create highly reproducible tests, which are easy to debug and create error messages that can be easily traced back to the relevant line of code." Good article, but I don't get this one at all; it almost seems like an anti-pattern. Choosing fixed data instead of random because the results are more "reproducible" seems to miss the point. If random data eventually helps uncover more bugs, then it's worth using!

Something1234 6y ago
It also adds in the assumption that your randomized setup is correct. That is not always the case, and in a lot of situations it's not trivial.
jayd16 6y ago
You should test the edge cases explicitly instead of hoping the randomization will save. If there's some bounded set of values, like an enum, then test every value instead of randomly picking things and hoping for the worst. I don't want to know eventually. I want to know now.
- chriswarbo 6y ago
  > I don't want to know eventually. I want to know now. That would be nice, but it's not the choice we're facing. The choice is between knowing eventually (by randomising) or knowing never (with determinism); or alternatively, between definitely finding out at 3AM when there's a production outage, or possibly finding out during testing. Test suites get run a lot, so even a small chance of spotting cases we hadn't thought of can be worthwhile. Also, it's much easier to run a randomised test with specific examples than it is to run a hard-coded test with randomised inputs (this is because "randomised tests" are actually parameterised tests, which take test data as arguments). Hence we might as well write the randomised version, then also call it with a bunch of known edge-cases (in QuickCheck-style frameworks this is just a function call, in Hypothesis we can also use the '@example' decorator). If we go down this route there are also automated approaches to make things easier, e.g. Hypothesis maintains a database of counterexamples that it's found in the past, which it mixes into its random data generators. We can ask for these values and use them as explicit examples if we like. In my experience writing "randomised tests" (i.e. property checking) is much easier and far more powerful than writing lots of hard-coded examples. I've done this in Haskell with QuickCheck, Scala with Scalacheck, Python with Hypothesis, Javascript with JSVerify and I hand-rolled a simple framework when I wrote PHP many years ago. Occasionally I find the urge to sprinkle a few hard-coded tests into the suite, but it rarely seems worth it.

closeparen 6y ago

>If this test fails, it’s hard to see what exactly is broken. Breaking existing tests is rare. I don't mind some detective work when it happens. You start with a pretty strong lead, anyway: whatever you touched since the last time the test passed. In my workflow this is unlikely to be more than five minutes ago. Half the time I know what I broke without even looking. I can see how specificity in test results would be important if you're running tests very infrequently and accumulating a lot of change between runs. But in that case I would probably invest in faster feedback before I'd invest in specificity. Huge, wordy test suites are expensive and can ossify bad design much more strongly than they preserve correct behavior.

ThePadawan 6y ago
> Breaking existing tests is rare. I don't mind some detective work when it happens. You start with a pretty strong lead, anyway: whatever you touched since the last time the test passed. You describe the exact situation where the advice is not helpful at all. Now imagine a large, very tightly interwoven codebase where in fact, making changes to a section of code breaks tests on the complete other side of the application. If you don't follow that advice, you are immediately blocked, and pretty much at the mercy of someone else fixing their test (probably to be less needlessly specific or less assertive after changing requirements). If they structured their test in a way that makes it clear why it should fail, you have the chance to investigate yourself how this contradicts with your changes.
- rightbyte 6y ago
  That's a organization problem (bunkers, are they called silos?) not a coding problem. I wish people would specify their coding advice with "advice for big projects with square process", "advice for dysfunctional planning", "advice for small team" etc.
- throwaway_pdp09 6y ago
  > very tightly interwoven Perhaps this is your problem.

lmilcin 6y ago

Hardly "best" practices. "Best" would suggest there are no better practices. I am fed up with unit testing. There is no feedback to tell if unit testing setup is complete and if unit tests are comprehensive. I am more and more turning to functional regression testing when entire service is tested as a black box against spec. Since we are testing API this tends to be specified much better than individual components. There is usually already some kind of documentation. API changes much less than individual components which means that if I start doing some kind of refactoring I may change a lot of internals but don't touch API at all. If I touch API then I need to update documentation and inform users so writing tests for that is not a huge additional cost.

chii 6y ago
> unit testing setup is complete coverage reports. They tell you which branch wasnt executed in your test, which can tell you what may be missing in your unit testing. > more turning to functional regression testing when entire service is tested as a black box against spec. functional/regression testing is important, but it can be difficult to use if your app is huge. And you can not easily isolate an area to test, so you always end up testing a common component repeatedly, and thus is slower to run. Not to mention some functional testing mechanisms (such as using a web browser) can be flaky. Both have their place in the automated testing paradigm.
- 2rsf 6y ago
  Although coverage reports are misleading, 100% coverage means that you passed through every line of code, that's it. You don't know if you covered edge cases, it cannot cover concurrency well and although possible it will not tell you whether you used a series of values or long-paths in your code.
  - StevenWaterman 6y ago
    You're right about edge cases in terms of input values, and right about issues with concurrency, but many of the issues you mention are solved by changing your metric. Branch-level coverage tells you what percentage of codepaths are covered. In other words, it would report 50% coverage for: if a: print "a yes" else: print "a no" if b: print "b yes" else: print "b no" test 1: a = false, b = false test 2: a = true, b = true

twic 6y ago

Recommending AssertJ and not mentioning Hamcrest? Hamcrest is more flexible, more composable, and more extensible.

joshka 6y ago
Having used both extensively, AssertJ's API is more obvious and generally useful out of the box. It's replaced all the places where I'd previously used custom Hamcrest matchers with generally smaller amounts of code that spit out better errors when tests fail. It's not a either or though, sometimes it's worth looking at whether hamcrest is the right approach.

Nihilartikel 6y ago

I'm pretty lazy, but I like testing.. The habit that I've repeated on my last few project is this: Work out a way to gracefully serialize/deserialize data structures in your code into a human readable format, like EDN or pretty printed JSON. (this is easier in more civilized languages... -winks at Clojure-) Pick high level functions that exercise a lot of code for testing. Generate input data to exercise that function either by hand if it's small and tractable, or by instrumenting real runs of the program and outputting serialized copies of that data (to file or console). Write test helpers that call the function with the input data, and depending on an environment variable, read the expected 'golden' data from disk, or if UPDATE_TEST_RESULTS=true, write the function output back to disk with a filename unique to the test, and mark the test as passed. If the test fails, print a nice readable, structural diff of the expected/actual data - e.g. python datadiff, or clojure.data/diff so the exact change is visible. On the first run you set UPDATE_TEST_RESULTS=true and generate the expected test results. Check your 'golden' result data into git. Now you can easily keep expected results up to date even if logic changes, and get a git diff of exactly how the data has changes. Acknowledged that this doesn't speak to side effect riddled code that needs mocks.. and doesn't buy into the build-the-test-first style of development, but I've found it's a good return on investment for regression testing, and provides visibility into subtle logic changes, expected or otherwise, introduced by code changes.

soedirgo 6y ago
Are you referring to snapshot testing[0][1]? i.e. you first "snapshot" the output of the function and commit it to VC, and each test run will run the same input and compare it against the "snapshot", failing and giving a diff if it differs. I'm about to try it soon, seems like a good ROI as you said. [0] https://jestjs.io/docs/en/22.x/snapshot-testing https://jestjs.io/docs/en/22.x/snapshot-testing [1] https://github.com/mitsuhiko/insta https://github.com/mitsuhiko/insta
- Nihilartikel 6y ago
  I think I am! Through convergent evolution at least - I hadn't found essays explicitly advocating it at the time. My use case was comparing results from chains of Spark RDD & Dataframe transformations, so having fairly large realistc input/output datasets was part of the game, and the main reason that manually writing all expected results wasn't feasible.

aabbcc1241 6y ago

The table of content is very helpful. However, I'm curious why this hugo generated static site is consuming over 130MB memory? For reference, HN is consuming 5MB memory only. (the numbers are observed from about:performance in Firefox)