10 ms·
> The excellent book xUnit Test Patterns describes a test smell named Assertion Roulette. It describes situations where it may be difficult to determine exactly
by TeMPOraL 4y ago
> The excellent book xUnit Test Patterns describes a test smell named Assertion Roulette. It describes situations where it may be difficult to determine exactly which assertion caused a test failure.
How is that even possible in the first place?
The entire job of an assertion is to wave a flag saying "here! condition failed!". In programming languages and test frameworks I worked with, this typically includes providing at minimum the expression put in the assertion, verbatim, and precise coordinates of the assertion - i.e. name of the source file + line number.
I've never seen a case where it would be hard to tell which assertion failed. On the contrary, the most common problem I see is knowing which assertion failed, but not how the code got there, because someone helpfully stuffed it into a helper function that gets called by other helper functions in the test suite, and the testing framework doesn't report the call stack. But it's not that big of a deal anyway; the main problem I have with it is that I can't gleam the exact source of failure from CI logs, and have to run the thing myself.
- danuker 4y agoI use the following pattern for testing regexes: expected_positive = [ 'abc', 'def', ...] for text in expected_positive: self.assertTrue(matcher(text), f"Failed: {text}") Before I added the assertion error message, `f"Failed: {text}"`, it was quite difficult to tell WHICH example failed.
- profunctor 4y agoIf you’re using pytest you just paramaterize the tests and it tells you the exact failing case. Seems to be a basic feature I would be surprised to know doesn’t exist across almost all commonly used frameworks.
- danuker 4y agoWe are using unittest. But thanks for bringing it up, it seems it also has "subtest" support, which might be easier than interpolating the error message in some cases: https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests https://docs.python.org/3/library/unittest.html#distinguishi...
- Dunedan 4y agoThere is also `parameterized` [1] if you want to do it by using decorators. [1]: https://pypi.org/project/parameterized/ https://pypi.org/project/parameterized/
- masklinn 4y ago> We are using unittest. Unless you need the library style in order to drive it, switch to pytest. Seriously. - assert rewriting is stellar, so much more comfortable than having to find the right assert* method, and tell people they're using the wrong one in reviews - runner is a lot more practical and flexible: nodeids, marks, -k, --lf, --sw, ... - extensions further add flexibility e.g. timeouts, maxfail, xdist (though it has a few drawbacks) - no need for classes (you can have them, mind, but if functions are sufficient, you can use functions) - fixtures > setup/teardown - parameterized tests
- jjgreen 4y agoYou mean "subTest", I switched to pytest just to get away from the camel (and it's in the standard library!)
- ericvsmith 4y agoCame here to mention unittest.subtest. It's for exactly this case.
- lancebeet 4y agoThat looks like a case where I would use a parameterized test rather than a for loop inside the test.
- dd82 4y agothis has downsides if you're comparing attributes with a method result and checking whether said attrs match what you expect. Either you run each test N times for N attr comparisons, accepting the cost of setup/teardown, or do a loop and fire off an assert error with text on which comparison failed. Since you already have the object right there, why not do the latter approach?
- lancebeet 4y agoIf the setup/teardown is expensive I would do it in reusable fixtures. The reason I wouldn't choose the latter approach is that it would usually be less convenient in the long run. You'd need to replace your asserts with expects to avoid it throwing on the first error (if this isn't what you want), you'll often need to manually add data to the assertion (as GP did) that you would otherwise get for free, and you'll need to look at the assertion error rather than the test case to know what actually failed. This can be quite inconvenient if you e.g. export your test results in a CI/CD pipeline.
- dd82 4y agonormally in a CI/CD pipeline, you'll see what asserts failed in the log output. Github Actions with pytest shows the context of the failed asserts in the log output. TBH, thought this was standard behavior, do you have experience with a CI pipeline that differs? All the other points you make as negatives are all positives for me. Biggest thing is, if you're making this change that alters things so drastically, is that really a good approach. Also, fixtures aren't magic. If you can't scope the fixture to module or session, that means by default it runs in function scope, which would be the same thing as having expensive setup/teardown. And untangling fixtures can be a bgger PITA than untangling unexpected circular imports
- olex 4y agoThis seems like a test design issue to me. Best practice is to avoid for-each loops with assertions within tests - using parametrized tests and feeding the looped values as input is almost always a better option. Figuring out which one failed and why is one advantage it gives you in comparison. Another one is that all inputs will always be tested - your example stops on the first one that fails, and does not evaluate the others after that.
- danuker 4y ago> your example stops on the first one that fails, and does not evaluate the others after that. I don't think this is a big problem; trying to focus on multiple examples at once is difficult. It might be a problem if tests are slow and you are forced to work on all of them at once. But in that case I'd try to make the tests faster (getting rid of network requests, disk/DB access by faking them away or hoisting to the caller).
- olex 4y agoHm. I think my main issue there is not the speed, but rather seeing the whole picture at once. You mentioned you use this pattern to test regular expressions; say you modify the regexp in question with some new feature requirement, and now the very first of a dozen test inputs fails. You fix it, but then each one of the following keeps failing, and you can only find an elegant solution that works for all of them after seeing all the failures, having ran the test and modified the code a dozen times. Wouldn't it be nicer to see all fails right away and be able to find a solution to all of them, instead of fixing the inputs one-by-one?
- danuker 4y agoIn my experience, from doing some TDD Katas[0] and timing myself, I found coding slower and more difficult when focusing on multiple examples at once. I usually even comment out all the failing tests but the first one, after translating a bunch of specifications into tests, so I see the "green" when an example starts working. Maybe it would be easier to grok multiple regex examples than algorithmic ones, but at least for myself, I am skeptical, and I prefer taking them one at a time. [0] - https://kata-log.rocks/tdd https://kata-log.rocks/tdd
- deleted 4y ago[deleted]
- spiffytech 4y agoI've seen several test frameworks that don't abort a test case after the first failed assertion. When you get many tests each emitting multiple failures because one basic thing broke, the output gets hard to sort through. It's easier when the failures are all eager.
- codeflo 4y agoWhich ones? I’ve used at least a dozen at this point, across C++, C#, JavaScript, Rust — and all of them throw (the equivalent of) exceptions on assertion failures.
- k__ 4y agoMy experience with testing framework was that they do all tests and then mark the ones that failed.
- carnitine 4y agoThat has nothing to do with not knowing which assertion in a given test has failed.
- jmillikin 4y agoGoogleTest (C++) and Go's built-in testing framework both support non-fatal assertions. They're used for code like this: assert_eq(list.len(), 1) expect_eq(list[0].username, "jdoe") expect_eq(list[0].uid, 1000) The idea being that if multiple properties are incorrect, then all of them will be printed out to the test log.
- masklinn 4y agoThat seems like it'd easily get confusing, when the assertions are dependent. Which is often the case e.g. if the list is empty, testing the properties of the first item make no sense.
- 4y ago
- sschueller 4y agoIn phpunit you can send dataset through a test function . If you don't label them you will have a jolly time finding out which one of the sets caused the failure.
- magic_hamster 4y ago> How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". I envy you for never having seen tests atrocious enough where this is not only possible, but the common case. Depending on language, framework and obviously usage, assertions might not be as informative as providing the basic functionality of failing the test - and that's it. Now imagine this barebones use of assertions in tests which are entirely too long, not isolating the test cases properly, or even completely irrelevant to what's (supposedly) being tested! If that's not enough, imagine this nightmare failing not after it has been written, but, let's say 18 months later, while being part of a massive test suite running for a while. All you have is a the name of the test that failed, you look into it to find a 630 lines long test "case" with 22 nondescript assertions along the way. You might know which line failed the test, but not always. And of course debugging the test function line by line doesn't work because the test depends on intricate timing for some reason. The person who wrote this might not be around and now this is your dragon to slay. I think I should stop here before triggering myself any further. Therapy is expensive.
- ramraj07 4y agoSo because some idiot somewhere wrote a 100 assertion unit test we should ban anyone from writing even 2 assertions in one test?
- magic_hamster 4y agoNot at all. It makes sense in some tests. I addressed the part asking how it's even possible to not know what happened. As for multiple asserts, that is really meaningless. The test case should test one thing. If it requires several asserts that's okay. But having a very long test function with a lot of assertions, is strongly indicating that you're testing more than one thing, and when the test fails it will be harder to know what actually happened.
- Joeboy 4y agoI guess you might be talking about a different language / environment than I'm used to, but even in the 100 assertion test case you get useful tracebacks in python. Testing lots of things at the same time means strictly speaking you're writing an integration test rather than a unit test, but still I don't see how it's a bad test. It's easy and stops buggy PRs going into production. The test failures I see that are actually hard to debug are ones where the failures are difficult to reproduce due to random input, tests running in parallel and sharing the same filesystem etc. I don't think I've ever not known what assert was failing (although I guess in theory you could make that happen by catching AssertionError).
- 411111111111111 4y agoit('fails', async () => { expect(await somePromise()).toBe(undefined) expect(await someOtherPromise()).toBeDefined() }) No chance figuring out where it failed, it's likely just gonna run into a test suite timeout with no line reference or anything.
- torginus 4y agoI remember writing a small .NET test library for that exact problem - You could pass in a lambda with a complex condition, and it evaluated every piece of the expression separately and pretty printed what part of the condition failed. So essentially you could write Assert(()=>width>0 && x + width < screenWidth) And you would get: Assertion failed: x is 1500 width is 600 screenWidth is 1920 It used Expression<T> to do the magic. Amazing debug messages. No moralizing required. This was a huge boon for us as it was a legacy codebase and we ran tens of thousands of automated tests and it was really difficult to figure out why they failed.
- superjan 4y agoIs this shared somewhere?
- torginus 4y agoNah, it was some corporate project. But if you are interested, I could rewrite it. It would be a fun weekend project.
- mattmanser 4y agoIt's also worth figuring out for yourself though! They're suprisingly easy to use (look up Expression<>). It's incredibly useful once you know how, and encourages you to stop using reflection, passing magic strings as arguments, or having to use nameof().
- mananaysiempre 4y agoFWIW, in Python, this is pytest’s entire thing (although it ends up grovelling into bytecode to achieve it).
- throwaway6977 4y agoNot quite the same, but available on nuget- 'fluentAssertions' gives you something akin to this. I've had decent success with having our juniors use it vs less verbose assertion libraries. I don't know about evaluating individual expressions in a line separately, but it does give you clean syntax and similar error messages that are very readable- "GetProductPage().GetProductPrice().Should().Be().LessThan(...)"
- torginus 4y agoxUnit is terrible. It has a horrible culture, the author seems to have a god complex. It is overly complex and opinionated in the worst way possible. Many times I searched for 'how do I do something with xUnit' and found a github issue with people struggling with the same thing, and the author flat out refusing to incorporate the feature as it was against his principles. Other times I found that I needed to do was override some core xUnit class so it would do the thing I wanted it to do - sounds complex, all right lets see the docs. Oh there are none, 'just read the source' according to the author. Another thing that bit us in the ass is they refused to support .NET Standard, a common subset of .NET Framework and Core, making migration hell.
- HereBeBeasties 4y agoNUnit isn't much better. You'd think that they would have good test coverage and therefore high confidence to make changes, especially to fix actual bugs, but I gave up trying to get patches in because the core devs seem so afraid of breaking anything, even when it's an obviously-isolated private, hidden bug fix or performance improvement. "We can't land this tiny fix because we have a release planned within three months" sort of thing.
- torginus 4y agoTbh, one of the differences between xUnit and nUnit, is the way generated test cases work, like specifying test cases in an xml file. nUnit has the TestCaseSource attribute for this, while xUnit has the Fact attribute. One of the key differences, is there are no test cases generated, nUnit tests just wont run, while xUnit will throw. Since it's completely legal and sensible for a certain kind of test to have no entries in an xml, we needed to hack around this quirk. When I (and countless others) have mentioned this on the xunit github, the author berated us that how dare we request this. So nUnit might be buggy, but xUnit is fundamentally unfixable.
- zdragnar 4y agoHow ironic, a unit test framework is so fragile and poorly covered by tests they are not able to determine if a change would cause any regressions.
- Macha 4y agoI've noticed the opposite in a Java codebase I work in. Tests where the test is assertEquals(toJson(someObject), giantJsonBlobFromADifferentFile). Of course the test runner has no idea about formatting strings that happen to be json, so I end up having to copy these out into an editor, formatting them and eyeballing the difference, or for even larger ones having to save them out to files and diff them. And of course most of the fields in the mock aren't relevant to the class under test, so I'd trade them out for 5-6 targeted asserts for the relevant fields happily. The problem is, since it's a legacy codebase, there's many fields which are only tested incidentally by this behaviour, by tests that actually aren't intending to test that functionality.
- TeMPOraL 4y agoI had similar case recently, in C++. I ended up spending a few hours writing a simple JSON differ - a bit of code that would parse two strings into a DOM object graph using a rapidjson, and then walk down them simultaneously - basically, I implemented operator== which, instead of terminating early, recorded every mismatch. Then, I packaged it into a Google Test matcher, and from now on, the problem you describe is gone. I write: EXPECT_THAT(someObject, IsEqAsJSON(someBlobFromADifferentFile)); and if it fails, I get output like this: Expected someObject to be structurally equivalent to someBlobFromADifferentFile; it is not; - #/object/key - missing in expected, found in actual - #/object/key2 - expected string, actual is integer - #/object/key3/array1 - array lengths differ; expected: 3, actual: 42 - #/object/key4/array1/0/key3 - expected "foo" [string], actual "bar" [string] Etc. It was a rather simple exercise, and the payoff is immense. I think it's really important for programmers to learn to help themselves. If there's something that annoys you repeatedly, you owe it to yourself and others to fix it.
- tsss 4y ago> I think it's really important for programmers to learn to help themselves. If there's something that annoys you repeatedly, you owe it to yourself and others to fix it. It's a cultural problem. _I_ can do that, but my colleagues will just continue to write minimum effort tests against huge json files or database dumps where you have no idea why something failed and why there are a bunch of assertions against undocumented magic numbers in the first place. It's like you're fighting against a hurricane with a leaf blower. A single person can only do so much. I end up looking bad in the daily standup because I take longer to work on my tickets but the code quality doesn't even improve in a measurable way.
- hedora 4y ago> I've never seen a case where it would be hard to tell which assertion failed. There are a set of unit testing frameworks that do everything they can to hide test output (junit), or vomit multiple screens of binary control code emoji soup to stdout (ginkgo), or just hide the actual stdout behind an authwall in a uuid named s3 object (code build). Sadly, the people with the strongest opinions about using a "proper" unit test framework with lots of third party tooling integrations flock to such systems, then stack them. I once saw a dozen-person team's productivity drop to zero for a quarter because junit broke backwards compatibility. Instead of porting ~ 100,000 legacy (spaghetti) tests, I suggested forking + recompiling the old version for the new jdk. This was apparently heresey.
- ch4s3 4y agoJunit is especially bad about this. I often wonder how many of these maxims are from people using substandard Java tools and confusing their workarounds with deeper insights.
- turblety 4y agoYup, it's why I built `just-tap` [1] which trys to minimise as much magic that a lot of these frameworks try to "help" you with. 1. https://github.com/markwylde/just-tap https://github.com/markwylde/just-tap
- autarch 4y agoTAP is better than some things, but it has some serious issues that I wrote about on my blog a while back - https://blog.urth.org/2017/01/21/tap-is-great-except-when-its-not/ https://blog.urth.org/2017/01/21/tap-is-great-except-when-it... Basically it's nearly impossible to fully parse it correctly.
- hedora 4y agoIs test2 a flag for TAP? If you have to pick one or the other, then you're breaking the common flow (human debugging code before pushing) so that management can have better reports. The right solution would be to add a environment variable or CLI parameter that told tap to produce machine readable output, preferably with a separate tool that could convert the machine readable junk to whatever TAP currently writes to stdout/stderr.
- charrondev 4y agoThe majority of the testing I’ve written have been jest tests and PHPUnit tests, and note PHPUnit is my favourite. It’s easy to built up custom assertions, and all of the in built assertions have the ability to provide an additional failure message during a failure. Assertions throw an exception and the test runner catches them along with any exceptions thrown by the code in test, marks the test as a failure, and reports the given error message and a full stack trace.
- xhxivuvy 4y agoIt's not about it being hard to tell which assertion failed. It's about being hard to tell what the cause of the failure was. When every test calls ->run_base_tests() before running it's own assertion sometimes things fail before you get to the root cause assertion. The other problem of stacking assertions is that you'll see the first failure only. There may be more failures that give you a better picture of what's happening. Having each assertion fail separately gives you a clearer picture of what's going wrong. Fwiw, the book doesn't suggest what the reader is saying. It says what I've said above more or less. You don't need to always stick to the rule but it generally does improve things to the point I now roll my eyes when I come across tests with stacked assertions and lots of test harness code that runs it's own assertions, I just know I'm in for a fun time.
- shagie 4y agoWith Junit, you could use assertAll https://junit.org/junit5/docs/5.0.0-M2/api/org/junit/jupiter/api/Assertions.html#assertAll-org.junit.jupiter.api.Executable...- https://junit.org/junit5/docs/5.0.0-M2/api/org/junit/jupiter... Aside from that, you could do things like int failures = 0; failures += someAssertion ? 0 : 1; failures += anotherAssertion ? 0 : 1; failures += yetAnotherAssertion ? 0 : 1; assertZero(failures); With the appropriate logging of each assertion in there. Consider the situation of "I've got an object and I want to make sure it comes out in JSON correctly" The "one assertion" way of doing it is to assertEqual the entire json blob to some predefined string. The test fails, you know it broke, but you don't know where. The multiple assertions approach would tell you where in there it broke and the test fails. The point is much more one of "test one thing in a test" but testing one thing can have multiple assertions or components to it. You don't need to have testFirstNameCorrect() and testLastNameCorrect() and so on. You can do testJSONCorrect() and test one thing that has multiple parts to verify its correctness. This becomes easier when you've got the frameworks that support it such as the assertAll("message", () -> assertSomething, () -> assertSomethingElse(), ...) https://stackoverflow.com/q/40796756 https://stackoverflow.com/q/40796756
- catlifeonmars 4y agoFor the JSON example, some testing libraries (jest) will output a unified diff on assertion failure.
- kazinator 4y agoThink about the personality of someone who is so dissatisfied with the lack of verbosity in his test suites, that he needs a side project of writing a book about unit testing. Of course they will advocate testing one assertion per function, and make up nonsense to justify their recommendation. Secretly, they would have the reader write 32 functions to separately test every bit of a uint32 calculation, only refraining from that advice due to the nagging suspicion that it might be loudly ridiculed.
- Phrodo_00 4y agoI think the advantage of having an assertion per test is that it makes sure that all of your assertions are executed. In a lot of test frameworks (that use exceptions for assertions for example) the first assertion fail will stop the test. That doesn't mean you have to duplicate code, you can deal with it in other ways. In Junit I like to use @TestFctory [1] where I'll write most of the test in the Factory and then each assertion will be a Test the factory creates, and since they're lambdas they have access to the TestFactoy closure. [1] https://junit.org/junit5/docs/current/user-guide/#writing-tests-dynamic-tests https://junit.org/junit5/docs/current/user-guide/#writing-te...
- edgyquant 4y agoThis is a feature. I want it to fail fast so I can see the first issue not crawl the stack for error logs.
- un_montagnard 4y agoI personally prefer seeing everything that I broke, not just the first one
- Izkata 4y agoI vaguely remember a test framework I saw a decade+ ago that had both "assert*" to fail immediately and something else ("expect*" maybe?) to check and continue.