28 ms·

Multiple assertions are fine in a unit test

• lodovic 4y ago
  Wait - people are doing real http calls in unit tests, over a network, and complaining about multiple asserts in the test code?
  • mollerhoj 4y ago
    Hopefully the responses are stored locally and replayed on subsequent runs.
    • tinyspacewizard 4y ago
      Still a bit flaky. In an OOP language mocking is appropriate or in an FP language defunctionalization.
      • mollerhoj 4y ago
        Heh yeah but it can be used to write tests that check your assumptions about a 3 party api. Granted, It'll only fail once the test are rerun without the cache, but it can still be a valuable technique. It can be valuable to have a testsuite that a) helps check assumptions when implementing the connection and b) helps locate what part of it later starts to behave unexpectedly.
  • deleted 4y ago
    [deleted]
  • chopin 4y ago
    Where I work milestone and release builds cannot make http calls other than to our maven repo. It's meant to further reproducible builds but your tests can't do this as well. I fire up an in-memory database to make my tests self containing.
  • batiste 4y ago
    I have met people that thinks HTTP calls are great! - "But it tests more things!" Well ok, but those are integration tests, not unit tests... It is unacceptable that a unit tests can fail because of external system...
    • int_19h 4y ago
      It's far better to have integration tests without unit tests than the other way around.
  • flqn 4y ago
    It's not unusual to spin up a local dev server in the same environment (or on the same machine, ie at localhost) as the tests. There's an argument to say these aren't "unit" tests but your definition of "unit" may vary.
  • warrenmiller 4y ago
    They can be mocked surely?
    • drewcoo 4y ago
      It's not nice to mock people "doing real http calls in unit tests," even if they deserve it.
      • warrenmiller 4y ago
        Oh dear
      • nsoonhui 4y ago
        For mocking, the parent comment means "swapping out the external calls with dummy calls", not "laughing at the developer"
        • kevincox 4y ago
          I'm 98% sure that they understood that and it was a joke.
        • BurningFrog 4y ago
          You failed a humor unit test.
        • ThunderSizzle 4y ago
          Woops, did you see the joke fly right past you?
  • tester756 4y ago
    define "real" yes, we perform "somewhat real tests" - tests start the app, fake db, and call HTTP APIs it's really decent
• wccrawford 4y ago
  First off, I do put more than 1 assertion in a test. But it definitely leads to situations where you have to investigate why a test failed, instead of it just being obvious. Like the article, I test 1 thing per test, but sometimes that means multiple assertions about the outcome of a test. IMO there's no point in checking that you got a response in 1 test, and then checking the content/result of that response in another test. The useful portion of that test is the response bit.
  • _lambdalambda 4y ago
    Even with multiple assertions the test failure reason should be quite clear as most testing frameworks allow to specify a message which is then output in the testing summary. E.g. `assertEqual(actual_return_code, 200, "bad status code")` should lead to output like `FAILED: test_when_delete_user_then_ok (bad status code, expected 200 got 404)`
    • TeMPOraL 4y ago
      Even if you don't specify the message, at the very minimum, the output should look like: FAILED: test_when_delete_user_then_ok Assertion failed: `actual_return_code' expected `200', got `400'. Note it mentions the actual expression put in the assert. Which makes it almost always uniquely identifiable within the test. That's the bare minimum I'd expect of a testing framework - if it can't do that, then what's the point of having it? It's probably better to just write your own executable and throw exceptions in conditionals. What I expect from a testing framework is at least this: FAILED: test_when_delete_user_then_ok Assertion failed: `actual_return_code' expected `200', got `400'. In file: '/src/blah/bleh/blop/RestApiTests.cs:212'. I.e. to also identify the file and the line containing the failing assertion. If your testing framework doesn't do that, then again, what's even the point of using it? Throwing an exception or calling language's built-in assert() on a conditional will likely provide at least the file+line.
  • iLoveOncall 4y ago
    > IMO there's no point in checking that you got a response in 1 test, and then checking the content/result of that response in another test. The useful portion of that test is the response bit. If I understood this part correctly, you are making the dangerous assumption that your tests will run in a particular order.
• choeger 4y ago
  What? People really would criticize that code because it has two assertions? How are they ever testing any state changes? And to the author: Your bubble is significantly different from mine. Pretty much every competent developer I've worked with would laugh at you for the idea that the second test case would not be perfectly fine. (But that first iteration would never pass code review either because it does nothing and thus is a waste of effort.)
  • mollerhoj 4y ago
    To answer your question: We zealots test for the fact that something changes to some degree. E.g with rubys rspec library: expect { foo.call() }.to change { bar.value }.by(2) That is, regardless of the absolute value of bar.value, I expect foo.call() to increment it by 2. The point of the 1 assertion per test guideline is to end up with tests that are more focused. Giving that you did not seem to think of the above technique, I'd say that this guideline might just have helped you discover a way to write better specs ;-) Guidelines (that is, not rules) are of course allowed to be broken if you have a good reason to do so. But not knowing about common idioms is not a good reason. You might argue that the above code is just sugar for 2 assertions, but thats beside the point: The test is more focused, there -appears- to be only one assertion, and thats what matters.
    • philliphaydon 4y ago
      That’s because the example test only requires 1 assertion. Any rule that says there should be only 1 assertion ever is stupid.
      • mollerhoj 4y ago
        OP asked how any state change would be tested with a single 'assertion' and I provided an answer. Absolute rules are stupid, but our codebase has just short of 10k tests, and very few have more than one assertion. The only reason I can really see to have more than one assertion would be to avoid having to run the setup/teardown multiple times. However, its usually a desirable goal to write code that require little setup/teardown to test anyways because that comes with other benefits. Again, it might not be practical or even possible, but that goes of almost all programming "rules"..
• jmillikin 4y ago
  Is it weird that not only have I never heard of the "rule" this post argues against, but I can't even conceive of a code structure where it would make sense? How would a test suite with one assertion per test work? Do you have all the test logic in a shared fixture and then dozens of single-assertion tests? And does that rule completely rule out the common testing pattern of a "golden checkpoint"? I tried googling for that rule and just came up with page after page of people arguing against it. Who is for it?
  • littlecranky67 4y ago
    > Is it weird that not only have I never heard of the "rule" this post argues against This "rule" is known mostly because it is featured in the "Clean Code" book by Robert C. Martin (Uncle Bob). You should have heard of it ;)
    • TeMPOraL 4y ago
      Perhaps the author is better off not having heard of it then, and by implication, not having read "Clean Code" in the first place. The book is full of anti-patterns.
    • Kwpolska 4y ago
      The book is so full of bad advice I'm not surprised this "rule" comes from there as well.
      • teddyh 4y ago
        As MikeDelta reports¹, the book doesn’t actually say that. I’ve come to learn to completely disregard any non-specific criticism of that book (and its author). There is apparently a large group of people who hate everything he does and also, seemingly, him personally. Everywhere he (or any of his books) is mentioned, the haters come out, with their vague “it’s all bad” and the old standard “I don’t know where to begin”. Serious criticism can be found (if you look for it), and the author himself welcomes it, but the enormous hate parade is scary to see. 1. https://news.ycombinator.com/item?id=33480517 https://news.ycombinator.com/item?id=33480517
• latch 4y ago
  I haven't heard of the single-assertion thing in at least 10 years, probably 15. In the early 2000s, when I was starting out and doing .NET, it used to be something you'd hear in the community as a very general guideline, more like "there's something to be said about very focused tests, and too many assertions might be a smell." At the time, I got the impression that the practice had come over from Java and converted from a rule to a guideline (hardly the only bad practice that the .NET community adopted from Java, but thankfully they largely did move the needle forward in most cases). (I wrote Foundations of Programming for any 2000s .NET developer out there!) To hear this is still a fight people are having...It really makes me appreciate the value of having deep experience in multiple languages/communities/frameworks. Some people are really stuck in the same year of their 10 (or 20, or 30) years of experience.
  • asabla 4y ago
    > I wrote Foundations of Programming for any 2000s .NET developer out there! Holy moly! Think I still have your book somewhere. So thank you for that. In my last +10 years of .net development I haven't heard anything about single-assertion. > Some people are really stuck in the same year of their 10 (or 20, or 30) years of experience. I think this has manifested even more with the transition into .net core and now .net 5 and beyond. There are so many things changing all the time (not that I complain), which can make it difficult to pick up what's the current mantra for the language and framework.
  • ed25519FUUU 4y ago
    Like you I was surprised to hear this is a thing or is even controversial. Admittedly I've only been programming for about 10 years, but I haven't heard (or seen) this come up even one time. Every test I've ever seen has usually had multiple mutations and assertions, all of them testing the same premise.
• BlargMcLarg 4y ago
  Out of everything coming out of testing and the pain of testing old code, this seems like such a trivial thing to discuss about. Then again, automated testing seems like a breeding ground for inane discussions wasting more time than picking a less-than-ideal solution and moving on.
• devit 4y ago
  The problem seems to be that assert is implemented as a regular function. It must be implemented as a macro so that the line number and assertion expressions are printed, allowing to easily identify the failed assertion. If a language doesn't support such macros and has no ad-hoc mechanism for this case, it should not be used, or if it must the assert function must take a string parameter identifying the assertion.
  • jmillikin 4y ago
    Some languages allow a stack trace to be obtained in normal code, which enables position reporting without macros. Python and Go are good examples. If you know the file and line of the assertion, plus the values that are being checked, there's not as much need for a stringified version of the expression.
    • TeMPOraL 4y ago
      > If you know the file and line of the assertion, plus the values that are being checked, there's not as much need for a stringified version of the expression. It does save time. With the actual condition reproduced, half the time I don't even need to check the source of the failed test to know what went bad and where to fix it. Consider the difference between: FAILED: Expected 1, got 4 In /src/foo/ApiTest.cpp:123 vs. FAILED: response.status evaluated to 4 expected: 1 In /src/foo/ApiTest.cpp:123 vs. FAILED: response.status evaluated to Response::invalidArg (4) expected: Response::noData (1) In /src/foo/ApiTest.cpp:123 This is also why I insist on adding custom matchers and printers in Google Test for C++. Without it, 90% of the time a failed assertion/expectation just prints "binary objects differ" and spews a couple lines of hexadecimal digits. Adding a custom printer or matcher takes little work, but makes all such failures print meaningful information instead, allowing one to just eyeball the problem from test output (useful particularly with CI logs).
      • jmillikin 4y ago
        Ah, this might be a difference of idiom. I would expect that assertion to print output like this: FAILED: Expected RESPONSE_NO_DATA, got RESPONSE_INVALID_ARG In /src/foo/ApiTest.cpp:123 For non-enumerated integers, the output would be something like: FAILED: Expected ResponseCode(200), got ResponseCode(404) In /src/foo/ApiTest.cpp:123 If values are formatted with some amount of self-description, then printing the name of the variable being evaluated is not usually informative.
• TeMPOraL 4y ago
  > The excellent book xUnit Test Patterns describes a test smell named Assertion Roulette. It describes situations where it may be difficult to determine exactly which assertion caused a test failure. How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". In programming languages and test frameworks I worked with, this typically includes providing at minimum the expression put in the assertion, verbatim, and precise coordinates of the assertion - i.e. name of the source file + line number. I've never seen a case where it would be hard to tell which assertion failed. On the contrary, the most common problem I see is knowing which assertion failed, but not how the code got there, because someone helpfully stuffed it into a helper function that gets called by other helper functions in the test suite, and the testing framework doesn't report the call stack. But it's not that big of a deal anyway; the main problem I have with it is that I can't gleam the exact source of failure from CI logs, and have to run the thing myself.
  • danuker 4y ago
    I use the following pattern for testing regexes: expected_positive = [ 'abc', 'def', ...] for text in expected_positive: self.assertTrue(matcher(text), f"Failed: {text}") Before I added the assertion error message, `f"Failed: {text}"`, it was quite difficult to tell WHICH example failed.
    • profunctor 4y ago
      If you’re using pytest you just paramaterize the tests and it tells you the exact failing case. Seems to be a basic feature I would be surprised to know doesn’t exist across almost all commonly used frameworks.
      • danuker 4y ago
        We are using unittest. But thanks for bringing it up, it seems it also has "subtest" support, which might be easier than interpolating the error message in some cases: https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests https://docs.python.org/3/library/unittest.html#distinguishi...
• TrianguloY 4y ago
  The one assertion per test doesn't mean you need to use only one assertion call but rather that you only need to do one assertion block. Checking everything after a response is considered 1 assertion, no matter how many assert calls you need. The issue is when you use multiple assertions for multiple logic statements: do > assert > do > assert... In that example imagine that you were also checking that the reservation was successful. That would be considered bad, you should create a different test that checks for that (testCreate + testDelete) and just have the precondition that the delete test has a valid thing to delete (usually added to the database on the setup).
  • christophilus 4y ago
    Sometimes, it takes a lot less code to test a particular code by doing it in a long test with multiple assertions. Migrate up -> assert ok -> rollback 1 -> assert ok -> rollback 2 -> assert ok I don’t see much benefit to breaking it up, and you’re testing state changes between each transition, so the entire test is useful and simpler, shorter, and clearer than the alternative.
    • twic 4y ago
      Sometimes it's also more meaningful to the reader this way. Imagine an object which sees numbers, tries to pair up identical numbers, and reports the set of unpaired numbers. A good test would be: Create object Assert it has no unpaired numbers Show it 1, 2, and 3 Assert it has 1, 2, and 3 as the unpaired numbers Show it 1 and 3 Assert it has 2 as the unpaired number This test directly illustrates how the state changes over time. You could split it into three tests, but then someone reading the tests would have to read all three and infer what is going on. I consider that strictly worse.
    • TrianguloY 4y ago
      That test is asserting a flow, not features. How can you be sure that the second rollback fails because that rollback code is wrong and not because the previous two functions made some unexpected changes? Or rather, how can you be sure that the second rollback is ok if you dont know the state it was run from? Maybe the migration set an unexpected flag that made the rollbacks pass that test without working properly. This is also the reason why tests should be run in arbitrary order, to avoid unexpected interactions due to order. Flow tests can be useful in some situations, but they should never replace individual feature tests.
• quickthrower2 4y ago
  Multiple assertions make sense in the case where otherwise you’d have to copy most of test A again to make test B
• lifeisstillgood 4y ago
  Things I find awkward with unit test (and it might just be me) is Inwant to write a test like def testLotsOfWaysTofail(): d = {"Handle Null": (None, foobar), "Don't allow under 13 to do it": (11, foobar), "or old age pensioner": (77,wobble)} for ... generate a unit test dynamically here I have built metaclasses, I have tried many different options. I am sure there is a near solution. But I never seem to have it right
  • oweiler 4y ago
    What you are looking for is called property based testing
  • _dain_ 4y ago
    isn't this just a parametrized test? or do you want to generate the test cases automatically like in property-based testing?
    • lifeisstillgood 4y ago
      Yes parameterised testing. Just all the packages i have tried seemed to make it awkward to impossible - hence metaclasses
      • deleted 4y ago
        [deleted]
      • ckp95 4y ago
        Does this do what you want? https://github.com/ckp95/pytest-parametrize-cases https://github.com/ckp95/pytest-parametrize-cases @parametrize_cases( Case("handle null", age=None, x="foobar"), Case("don't allow under 13s", age=11, x="foobar"), Case("or old age pension", age=77, x="wobble"), ... # as many as you want ) def test_lots_of_ways_to_fail(age, x): with pytest.raises(ValueError): function_under_test(age, x)
        • 4y ago
• oxff 4y ago
  I mean, obviously. Obviously I'm not going to assert every header in their own unit test for a single request path in most cases. Assert what makes sense as a logical whole / unit.
• projektfu 4y ago
  This is such a bad example because the level of testing is somewhat between unit and acceptance/functional testing. I can't tell at a glance if "api" is something that will hit a database or not. The first part where he says you can check in the passing test of code that does nothing makes me twitch, but mainly because I see functional testing as the goal for a bit of functionality and unit tests as a way of verifying the parts of achieving that goal. Unit tests should verify the code without requiring integration and functional tests should confirm that the units integrate properly. I wouldn't recommend checking in a test that claims to verify that a deleted item no longer exists when it doesn't actually verify that. Deciding on the granularity of actual unit tests is probably something that is best decided through trial and error. I think when you break down the "rules", like one assertion per test, you need to understand the goals. In unit testing an API I might have lots of tiny tests that confirm things like input validation, status codes, debugging information, permissions, etc. I don't want a test that's supposed to check the input validation code to fall because it's also checking the logged-in state of the user and their permission to reach that point in the code. In unit tests, maybe you want to test the validation of the item key. You can have a "testItemKey" test that checks that the validation confirms that the key is not null, not an empty string, not longer than expected, valid base64, etc. Or you could break those into individual tests in a test case. It's all about the balance of ergonomics and the informativeness and robustness of the test suite. In functional testing, however, you can certainly pepper the tests with lots of assertions along the way to confirm that the test is progressing and you know at what point it broke. In that case, the user being unable to log in would mean that testing deleting an item would not be worthwhile.
• zabil 4y ago
  I like the one assertion per unit test case rule. It makes me design cleaner code and write readable test cases. I also use object equality vs asserting on individual fields.
• SideburnsOfDoom 4y ago
  This is correct - multiple asserts are OK, but there are still good guidelines: A good unit test has the phases Arrange, Act, Assert, end of unit test. You can use multiple assert statements in the "assert" phase, to check the specifics of the single logical outcome of the test. In fact, once I see the same group of asserts used 3 or more times, I usually extract a helper method, e.g. "AssertCacheIsPopulated" or "AssertHttpResponseIsSuccessContainingOrder" these might have method bodies that contain multiple assert statements, but the question of is this a "single assert" or not, is a matter of perspective and not all that important. The thing to look out for is - does the test both assert that e.g. the response is an order, and that the cache is populated? Those should likely be separate tests as they are logically distinct outcomes. The test ends after the asserts - You do not follow up the asserts with a second action. That should be a different test.
• ChrisMarshallNY 4y ago
  I always have multiple assertions in my unit tests. I test around areas of functionality, as opposed to individual functions. It's a bit arbitrary, but there you have it... I also use Apple's XCTest, which does a lot more than simple assertions. If an assertion is thrown, I seldom take the assertion's word for it. I debug trace, and figure out what happened. The assertion is just a flag, to tell me where to look.
• skatanski 4y ago
  This always rubbed me the wrong way. I think a better way would be to ensure the assertions to be readable, as simple as can be. The worst case of this being broken was the assertions were done in an helper method used in a base class of a test, and navigating to it required multiple hops, it took time to build the context of the test as well. The only downside of multiple assertions is that when the first one fails we won’t know if the subsequent ones are passing.
  • nharada 4y ago
    Totally agreed, I even think duplication is totally fine in tests if it brings more readability. You should be able to read the test to verify its correctness (after all you don't test the test code), and multiple helper functions and levels of abstraction hinder this goal.
• Jolter 4y ago
  I’ve seen code guidelines where this article’s asserts would be unacceptable. If I saw this error in a test output, I’d be pretty miffed that it doesn’t tell me what the expected and actual values are. Actual status code: NotFound. Expected: True Actual: False
• someweirdperson 4y ago
  A maximum of one assert per test seems even worse than the church of "a maximum of one test for every instruction that needs coverage". Both seem to be driven by misunderstanding through simplification of some otherwise meaningful ideas.
• willjp 4y ago
  I think part of the motivation for one assertion per test comes from the fact that you stop getting information from a test as soon as one assertion fails. I think it was a guidance, like the SRP where you should be testing one thing in each test case. I also think a growing number of assertions might be a sign your unit under test is wearing too many responsibilities. Maybe it’s better to say “few assertions, all related to testing one thing”
• deleted 4y ago
  [deleted]
• oytis 4y ago
  Who said multiple assertions are wrong in the first place? First time I hear about it really
• z9znz 4y ago
  Testing is hard when the code it tests is OOP, has mutations everywhere, and has big functions that do too many things. It's practically impossible to thoroughly test such code with one assertion per test; it would mean having dozens of tests just for one object method. Correspondingly, the fixtures/factories/setup for tests would balloon in number and complexity as well to be able to setup the exact circumstance being tested. But the example in TFA is, imo, bad because it is testing two entirely different (and unrelated) layers at once. It is testing that a business logic delete of a thing works correctly, and that a communication level response is correct. Those could be two separate tests, separating the concerns, and resulting in simpler code to reason about and less maintenance effort in the future. We want to know if the DeleteAsync(address) behaves correctly. Actually we want to know if DeleteReservation() works, irrespective of the async requirement. Testing whether AnythingAsync() works is something that is already done at the library or framework level, and we probably don't need to prove that it works again. Write a test for DeletReservation() which tests if a valid reservation gets deleted. Write another related test to ensure that a non-existent or invalid reservation does not get deleted, but rather returns some appropriate error value. That's two, probably quite simple tests. Now somewhere higher up, write the REST API tests. ApiDelete()... a few tests to establish that if an API delete is called, and the business logic function it calls internally returns a successful result, then does the ApiDelete() return an appropriate response? Likewise if the business logic fails, does the delete respond to the API caller correctly?
  • drewcoo 4y ago
    > Testing is hard when the code it tests is OOP, has mutations everywhere, and has big functions that do too many things. Yes, that means the software being tested is not very testable. And should probably be refactored. https://en.wikipedia.org/wiki/Testability https://en.wikipedia.org/wiki/Testability
  • commandlinefan 4y ago
    > when the code it tests is OOP In my experience when code isn’t OOP, that means all static functions with static (I.e. global) data which isn’t hard to test, it’s actually impossible because you can’t mock out the static data.
• indymike 4y ago
  I've seen this pattern often with teams that are out of their normal wheelhouse (i.e. Python shop doing Go or JS shop doing Java) and think they cannot extend a bad test library. The other place you see this a lot is where developer hours are being billed. I can sympathize with devs who are shallow on a test library, but the billing hours one is a dark pattern for optimizing cost plus billing.
• deleted 4y ago
  [deleted]
• batiste 4y ago
  It has been a long time I have met somebody stupid enough to push this kind of meaningless rules forward. Test setup is sometimes very complicated AND expensive. Enforcing only 1 assertion per test is moronic beyond description.
• cultofmetatron 4y ago
  I didn't know you were only supposed to put one assertion per unit test. some of mine have 5-10 assertions. why?. because I don't have time to write tests all day but I recognize there are tests. As long as it catches the bugs before it goes to prod and can be modified easily, who cares?
  • BlargMcLarg 4y ago
    Unfortunately and from experience, far more seem to care about the how than the what when it comes to testing. This comment section alone is pretty telling: people spending their Saturday to write dogmatic rules.
• alkonaut 4y ago
  An assert message says what went wrong, and on which code line. How on earth does it help to make just one? The arrange part might take seconds for a nontrivial test and that would need to be duplicated both in code and execution time to make two asserts. If you painstakingly craft a scenario where you create a rectangle of a specific expected size why wouldn’t it be acceptable to assert both the width and height of the the rectangle after you have created it? assert_equal(20, w, … assert_equal(10, h, … A dogmatic rule would just lead to an objectively worse test where you assert an expression containing both width and height in a single assert? assert_true(w == 20 && h == 10,…) So I can only assume the rule also prohibits any compound/Boolean expressions in the asserts then? Otherwise you can just combine any number of asserts into one (including mutating state within the expression itself to emulate multiple asserts with mutation between)!
  • TeMPOraL 4y ago
    > So I can only assume the rule also prohibits any compound/Boolean expressions in the asserts then? Otherwise you can just combine any number of asserts into one That's what's bound to happen under that rule. People just start writing their complex tests in helper functions, and then write assert_true(myTotallySimpleAndFocusedTestOn(result))
  • jeremyjh 4y ago
    Apparently this is yet another “best practice” preached by people using crappy tools.
  • ljm 4y ago
    I’ve seen people take a dogmatic approach to this in Ruby without really applying any critical thought, because one assertion per test means your test is ‘clean’. The part that is glossed over is that the test suite takes several hours to run on your machine, so you delegate it to a CI pipeline and then fork out for parallel execution (pun intended) and complex layers of caching so your suite takes 15 minutes rather than 2 and a half hours. It’s monumentally wasteful and the tests aren’t any easier to follow because of it. The suite doesn’t have to be that slow, but it’s inevitable when every single assertion requires application state to be rebuilt from scratch, even when no state is expected to change between assertions, especially when you’re just doing assertions like ‘assert http status is 201’ and ‘assert response body is someJson’.
• dicroce 4y ago
  also, only testing public interfaces is perfectly fine and may actually be preferable as it leaves you free to refactor internals freely without breaking tests. tbh unit testing is a balancing act between reaping code quality benefits and bogging yourself down with too much testing updating.
  • hgomersall 4y ago
    This is an interesting point IMO. We tend to focus at the API level far more, and implicitly test inner functionality (that is, the inner functionality must be correct for the outer tests to pass). Sometimes testing inner functionality explicitly is required when the outer tests are not complete, or when behaviour is defined by the inner code. We also as far as possible use defensive techniques and extensive type constraints (which is a joy in Rust). I'm constantly thinking where we need to put tests though, and in still not fully convinced I get it right. My rule of thumb is that each test should map to a specification point, and that spec is a necessary documentation line for the test.
• andrewallbright 4y ago
  I've spent some time testing and thinking about testing. I think the universal guidance that should be followed is this... Do whatever you need/want to that makes your software successful. The rules are meant to be broken if it means you shipping.
• throwaway0asd 4y ago
  My learning from test automation: * The primary goal of test automation is to prevent regression. A secondary goal can be performance tuning your product. * Tests are tech debt, so don’t waste time with any kind of testing that doesn’t immediately save you time in the near term. * Don’t waste your energy testing code unless you have an extremely good reason. Test the product and let me the product prove the quality of your code. Code is better tested with various forms of static analysis. * The speed with which an entire test campaign executes determines, more than all other factors combined, when and who executes the tests. If the test campaign takes hours nobody will touch it. Too painful. If it takes 10-30 minutes only your QA will touch it. When it takes less than 30 seconds to execute against a major percentage of all your business cases everybody will execute it several times a day.
  • jeremyjh 4y ago
    Tests are not tech debt. You could have bad, brittle tests that you could consider debt but just having tests isn’t debt. Debt implies there is something you could do about it in the future to pay it down, which isn’t the case for a good test suite.
    • ronnier 4y ago
      It’s debt. When you can’t add new features quickly because you have nightmarish tests to fix and you spend more time on the tests than the product, I’d say it’s debt. Especially with the insane mocking setups.
      • jeremyjh 4y ago
        Yes, those are the bad tests I was referring to. NOT having tests greatly increases the debt burden of your production code because you cannot refactor with any confidence and so you simply won’t.
        • hbrn 4y ago
          > Yes, those are the bad tests I was referring to This is the No True Scotsman issue with testing. When it fails, you just disregard the failure as "bad tests". But any company that has anything that resembles a testing culture will have a good amount of those "bad tests". And this amount is way higher than people are willing to admit. > you cannot refactor with any confidence Anecdotally, I've had way more cases where I wouldn't refactor because too many "bad tests" were breaking, not because I lacked confidence due to lack of tests. There are many things beyond tests that allow you refactor with confidence: simple interfaces, clear dependency hierarchy, modular design, etc. They are way more important than tests. Tests are often a last resort when all of the above is a disaster. When you're at a place where you need tests to keep your software stable you are probably already fucked, you're just not willing to recognize it. You shouldn't have zero tests, but tests should be treated as debt. The fewer tests you need to keep your software stable, the better your architecture is. Huge number of tests in a codebase is typically a signal of shitty architecture that crumbles without those crutches.
• rhdunn 4y ago
  I view it more as "only test one operation per unit test". If that needs multiple asserts (status code, response content, response mime type, etc.) to verify, that is fine. IIUC, the guideline is so that when a test fails you know what the issue is. Therefore, if you are testing more than one condition (missing parameter, invalid value, negative number, etc.) it is harder to tell which of those conditions is failing, whereas if you have each condition as a separate test it is clear which is causing the failure. Separate tests also means that the other tests run, so you don't have any hidden failures. You'll also get hidden failures if using multiple assertions for the condition, so will need to re-run the tests multiple times to pick up and fix all the failures. If you are happy with that (e.g. your build-test cycle is fast) then having multiple assertions if fine. Ultimately, structure your tests in a way that best conveys what is being tested and the conditions it is being tested under (e.g. partition class, failure condition, or logic/input variant).
  • ronnier 4y ago
    I’ve come to the conclusion that none of this matters for most parts of a system. I worked in the most horrendous code and systems you can imagine but it turned into a multi billion dollar company. Then everyone starts talking about code quality and rewrites etc and new features stall as beautiful systems are written and high test coverage met and competing companies surpass us and take market share with new and better features. We’ve gotten religious over code and tests in the software industry abd should probably shift back some.
    • dxdm 4y ago
      I've worked on shitty code with shitty tests that ran the core of the business. Even while doing that, it was horrible to work with, held important features back and drove talented people away, leaving everything to stagnate in a "this works enough" state. When the wind changed, it was hard to turn the ship around, important people got nervous, and things got into a bad spiral. None of this is the failure of code and tests alone; but both can be indicative of the structural health and resilience of the wider situation.
• SeriousM 4y ago
  I keep it simple by using the AAA pattern: Arrange: whatever you need for the setup. Act: a single line of code that is under test. Assert: whatever you want to assert, multiple statements allowed and usually required. We put the 3A in code as comments as boundaries and that works more than perfect for the whole team. Oh and it's readable!
• smhoff256 4y ago
  The reason for one assert pet test is that it forces you to test one thing at a time. It sets your mind in the mode of single responsibility.
  • marcosdumay 4y ago
    And what is the gain from testing a single thing at a time? On a specific case, what is the gain from not testing the pre and post-conditions of your important test?
• gorjusborg 4y ago
  The trouble is that the type of developer that blindly follows this sort of shaming/advice is exactly the type that blindly followed someone's rule of thumb to use one assertion per test. There are no hard and fast rules. I agree with keeping the number of assertions low, but it isn't the number that matters. Keeping the number of assertions low helps prevent the 'testItWorks()' syndrome. Oh, it broke, I guess 'it does not work' time to read a 2000 line test written 5 years ago.
• turtleyacht 4y ago
  Use mutation testing as well, if it's available for your language. This evaluates the quality of your test suite by automatically switching up your app logic and running the same tests. Nothing like removing one line, breezing through code review, and bringing down production. "But all tests passed!"
• mrhands556 4y ago
  > You may be trying to simulate a ‘session’ where a client performs many steps in order to achieve a goal. As Gerard Meszaros writes regarding the test smell, this is appropriate for manual tests, but rarely for automated tests. Integration tests are typically easier to write / maintain and thus are more valuable than small unit tests. Don’t know why the entire premise argues against that.
• ascendantlogic 4y ago
  I've always operated on the principal that each test should be testing one logical concept. That usually translates into one concrete assertion in the test code itself, but if you need to assert multiple concrete conditions to test the logical concept it's not a bad thing. At the end of the day tests are just there to make you more confident in code changes and they are a tax you have to pay for that comfort. However you arrive at "I feel comfortable making changes", go with it.
• bazoom42 4y ago
  Unit tests are one of the many great ideas in our industry which have been undermined by people treating it is a set of rituals rather than a tool.
  • seadan83 4y ago
    I wonder how much of this is the journeyman problem (aka, the expert beginner) I believe writing test code is its own skill. Hence, like a coder learning SRP and dogmatically applying it, so does a person that is forced to write unit tests without deep understanding. (And of course, bad abstractions are worse than code duplication) I think it's very possible to have a developer with 10 yes experience but effectively only 2 years experience building automated test suites. (Particularly if they came from a time before automated testing, or if the testing and automated tests were someone else's job)
• jameshart 4y ago
  The issue here appears to be imperative testing, with assertion failures as an early exit. If all your assertions are always evaluated, you don’t have this issue?
• kmac_ 4y ago
  Multiple asserts are fine, multiple aspects in a single test are not. For example like a transaction test that checks if account balance is ok and if an user gets a notification. It boils down to a single responsibility, so when there are multiple aspects then it means that the class/interface should be refactored.
• fnordpiglet 4y ago
  Stop writing unit tests and write comprehensive integration tests instead. Problem solved.
• 0x445442 4y ago
  A test should have as many assertions as needed to test an interface. If you find you’re needing a lot of assertions to do that then I suspect either the interface under test is too large or you’re testing multiple stack frames. In my experience, it’s usually the latter; I call it accidental testing. Those delegate calls should have their own tests.
• cratermoon 4y ago
  The Go idiom is to use table-driven tests[1]. It's still an evolving practice, so you'll see different variants, but the essence is that you have a slice of your inputs and expected outputs, iterate through the slice, and run the assert(s) on each element. var flagtests = []struct { in string out string }{ {"%a", "[%a]"}, {"%-a", "[%-a]"}, {"%+a", "[%+a]"}, // additional cases elided {"%-1.2abc", "[%-1.2a]bc"}, } func TestFlagParser(t *testing.T) { var flagprinter flagPrinter for _, tt := range flagtests { t.Run(tt.in, func(t *testing.T) { s := Sprintf(tt.in, &flagprinter) if s != tt.out { t.Errorf("got %q, want %q", s, tt.out) } }) } } Sometimes there will be an additional field in the test cases to give it a name or description, in which case the assertion will look something like: t.Fatalf("%s: expected: %v, got: %v", tt.name, tt.out, got) Another evolving practice is to use a map instead of a slice, with the map key being the name or description of the test case. This is nice because in Go, order is not specified in iterating over a map, so each time the test runs the cases will run in a different order, which can reveal any order-dependency in the tests. 1 https://dave.cheney.net/2019/05/07/prefer-table-driven-tests https://dave.cheney.net/2019/05/07/prefer-table-driven-tests
• inglor_cz 4y ago
  It is interesting how far purism can go. I wouldn't have thought that some people obsess about having just one assertion in a test. This seems to be a case of being mentally enslaved by your own categories.
• 8n4vidtmkvmk 4y ago
  i need to send this to all my coworkers. i wrote some tests kind of similar to this, except i had a bunch of setup code then asserted something was true, changed 1 thing and then verified it became false so that it was super clear that it was the 1 thing that causes it to become false. they were like no, copy paste the entire setup and change the field to false in the new setup. im like how are you supposed to tell which of the dozen conditions triggered it now? you have to diff the 2 tests in your head and hope they don't get out of sync? ridiculous
• msaltz 4y ago
  I think two important requirements for good unit tests are that 1) If you look at a particular test, you can easily tell exactly what it's testing and what the expected behavior is, and 2) You are able to easily look at the tests in aggregate and determine whether they're covering all the behavior that you want to test. To that end, I think a better guideline than "only have one assertion per test" would be "only test one behavior per test". So if you're writing a test for appending an element to a vector, it's probably fine to assert that the size increased by one AND assert that the last element in the vector is now the element that you inserted. The thing I see people do that's more problematic is to basically pile up assertions in a single test, so that the inputs and outputs for the behavior become unclear, and you have to keep track of the intermediate state of the object being tested in your head (assuming they're testing an object). For instance, they might use the same vector, which starts out empty, test that it's empty; then add an element, then test that its size is one; then remove the element, test that the size is 0 again; then resize it, etc. I think that's the kind of testing that the "one assertion per test" rule was designed to target. With a vector it's easy enough to track what's going on, but it's much harder to see what the discrete behaviors being tested are. With a more complex object, tracking the internal state as the tests go along can be way more difficult. It's a lot better IMO to have a bunch of different tests with clear names for what they're testing that properly set up the state in a way that's explicit. It's then easier to satisfy the above two requirements I listed. I want to be able to look at a test and know exactly what I don't mind if a little bit of code is repeated - you can make functions if you need to help with the test set up and tear down.
• jpollock 4y ago
  I work to "assert one thing", and the assertions are added to help with debugging. I sometimes even assert starting conditions if they can at all change. Without a "assert one thing", what tends to happen is there will be cut-and-paste between tests, and the tests overlap in what they assert. This means that completely unrelated tests will have an assertion failure when the code goes wrong. When you do a refactor, or change some behavior, you have to change _all_ of the tests. Not just the one or two that have the thing you're changing as their focus. Think of tests that over-assert like screenshot or other diff-based tests, they are brittle.
• 0xmarcin 4y ago
  I think it depends on the test. If you are checking that e.g. context.currentUser() returns current user name, email, id and roles I would probably wrote just 2 tests. One for the user attributes and one for checking roles. jUnit provides a helpful assertAll(...) method that allows us to check multiple assertions without stopping and the first failed one. I my tests I often use "thick" asserts like assertEqualsWithDiff(dtoA, dtoB), that compare 2 objects as a whole and prints property names with values that do not match. Not everyone likes this approach, but for me it is the best balance between time spend on the test and the benefit that I get from it.
• code_runner 4y ago
  I have a bad habit of writing a mondo-super-all-encompassing test, and I have to break it up when I realize it’s a house of cards. I think the better rule is “just test one thing, don’t use a lot of overlapping assertions between tests”. Number of assertions, as noted by this post and this comment section, is not a smell at all.
• kazinator 4y ago
  At work, I put in a change which allows multiple assertions in a C testing framework. There is no exception handling or anything. A macro like EXPECT_ASSERT(whatever(NULL)); will succeed if whatever(NULL) asserts (e.g. that its argument isn't null). If whatever neglects to assert, then EXPECT_ASSERT will itself assert. Under the hood it works with setjmp and longjmp. The assert handler is temporarily overridden to a function which performs a longjmp which changes some hidden local state to record that the assertion went off. This will not work with APIs that leave things in a bad state, because there is no unwinding. However, the bulk of the assertion being tested are ones that validate inputs before changing any state. It's quite convenient to cover half a dozen of these in one function, as a block of six one-liners. Previously, assertions had to be written as individual tests, because the assert handler was overriden to go to a function which exits the process successfully. The old style tests are then written to set up this handler, and also indicate failure if the bottom of the function is reached.
• nesarkvechnep 4y ago
  Are people so entrenched in dogmas that we need “Stop doing X” type articles every few months?
  • quonn 4y ago
    Unfortunately, yes. People with good judgement are difficult to find.
• zzzeek 4y ago
  "A foolish consistency is the hobgoblin of little minds" - whoever I definitely write tests with multiple assertions, the rule I try to follow is that the test is testing a single cause/effect. that is, a single set of inputs, run the inputs, then assert as many things as you want to ensure the end state is what's expected. there is no problem working this way.
  • deleted 4y ago
    [deleted]
• peheje 4y ago
  Is DeleteReservationActuallyDeletes a good test name?
• ziml77 4y ago
  It took me a long time to feel okay with all the times I broke the single assertion “rule” after reading Clean Code. In fact I only recently stopped feeling bad at all when I went to reimplement a bunch of abseil’s flat hash map tests in C#. All of the tests assert multiple things. If a project as big as that can have multiple asserts on a basic data structure then I can too
• dvh 4y ago
  My problem with assertions is that they can be turned off. I've seen things that could never happen happen in production on daily basis.
• dwgebler 4y ago
  It's not that there's anything inherently wrong with multiple assertions in a unit test, it's that it's often a smell you're testing multiple behaviours in a single test - which is the thing you should avoid.
• ImPleadThe5th 4y ago
  There is this team of fresher-to-the-field devs at my company who insist this is law. To the point where if they are testing object equality, they check each field in its own test. I have no idea where they got the notion from.
• eddsh1994 4y ago
  These aren't unit tests, they are integration tests (API calls to an external system). Integration tests have had multiple assertions forever (look at how Cypress works where each test step is an assertion), the idea of single assertions are for unit tests. The issue with multiple assertions for unit tests is that they hide errors by failing early extending the time to fix (fail at Assert 1, fix, re-run, fail at Assert 2, fix...). If you want multiple assertions you should have a single assertion that can report multiple issues by returning an array of error strings or something. The speed increase of multiple assertions vs multiple tests is usually tiny, but if you do have a long unit test then that's basically the only reason I can think of to use multiple assertions.
  • hamandcheese 4y ago
    The speed at which I can fix bugs is dramatically slower than the speed at which I can re-run a unit test. Even when I do have multiple failing tests, I focus on and fix them one test at a time.
• xg15 4y ago
  I think my dream test framework would have the following features: 1) Test suites are organized as trees, not as lists: I found one of the most common reasons to have many assertions in a test was that you want to share some complicated setup/teardown logic - or that one testable action depends on another testable action having happened before. (i.e., adding an item - asserting it's there, then removing it, asserting it's gone). The disadvantage is that you have to lower the granularity of your tests - if you want to debug a specific action, you still have to rerun the whole test. I think a better way to solve this would be to organize tests as a tree, maybe something like this: - A single unit test consists of a setup phase, a teardown phase, 0 or more assertions and 0 or more child tests. Each child test is organzed the same, i.e. can have child tests on its own, etc. - When running a test, first the setup phase and assertions are executed, then each child test recursively, then the teardown phase. Success/failure is tracked for each test separately, but child tests are ran in the same process/context as the parent test. - Each test can be started individually, including child tests. When a child test (or grandchild test, etc) is ran individually, the test runner will first run the setup phases of all ancestors, then run the test, then run the teardown phases of the ancestors. - Bonus: In the setup phase, a test can dynamically generate child tests (e.g. as lambdas/closures). Each test must have a unique ID with which it can tracked across different test runs or started individually. This could be useful for parameterized tests or if you want to test a loop invariant across multiple iterations. This would allow you to write your test script like one big multi-assert test, but still get fine-grained reports and control as if you'd have put each assert in a separate script. 2) Provide "metrics" and "change detection" as an alternative to assertions: I think one of the most involved parts of writing tests can often be to verify the results - think which particular state you want to assert, how you can access that state in your script, etc. A way to make this easier would be to provide a second kind of "output" for the test script: The test script simply outputs a list of key/value pairs without any notion whether or not the value is "correct" or "incorrect". The test runner stores the list and compares the values with the list from a previous test run - e.g. the previous commit. Every value that was changed between the runs is shown to the user and can be marked as "correct" or "incorrect". This way, you could sort of interactively "learn" which values are correct and which aren't instead of having to figure out all of it beforehand. The runner could also implement more complex conditions instead of "changed"/"did not change", such as "value may only change in one direction" e.g. for quality measures or "value must stay the same within a certain confidence interval" for flaky tests. This could also let you track more difficult to manage metrics in a test, such as runtime or memory consumption of particular method calls.