9 ms·
Evil Unit Testing (2019)
- dandigangi 6y agoIs shunt a thing? Never heard of that one. :embarrassed:
- srtjstjsj 6y agoMost people call that patch mocking. But it's good to have a name for shunt because otherwise people mistakenly think using a pure mock is a good idea. (It isn't. Use a fake instead -- real code but with a fast in-memory data store instead of a real database server)
- FroshKiller 6y agoThere's no reason to be embarrassed. You might be a very learned individual of deep and wide experience, but the truth is still that you have not heard of most things. None of us has heard of most things.
- dandigangi 6y agoI appreciate that. Thank you.
- srtjstjsj 6y ago> There are functional tests that exercise these classes working together. Fixing a bug in one class breaks many functional tests .. Huh? If fixing a bug breaks a functional test, then either the test was correct and you changed your expected system behavior and the test saved your bacon, or the test was a change detector test not a functional test. Saying "remove functional years" is just blinding yourself. A unit test will never tell you if the class you are changing has any relationship to the actual bug reported in production. It seems like OP has never worked on a large system, or doesn't know about fakes (not mentioned in the article) to run end-to-end functional tests quickly in memory.
- spawarotti 6y agoCame here to say exactly that. The author is advocating for mockist style of unit testing, which is quite terrible for many, many reasons. What you are describing is classic unit testing, and this is the way to go. Good article and book about it: - https://enterprisecraftsmanship.com/posts/unit-testing-dependencies https://enterprisecraftsmanship.com/posts/unit-testing-depen... - Unit Testing Principles, Practices, and Patterns
- pydry 6y agoMain reason it it tests unrealistically, I think. The other issues (doesnt catch bugs, hard to maintain) tend to flow from that.
- gwbas1c 6y ago> The author is advocating for mockist style of unit testing IMO: "mockist" style of unit tests are best for corner cases that can only be tested in isolation. For example, error conditions that are important to verify, but so difficult or impossible to test that the only way to test them is to test a single class in isolation.
- asdfasgasdgasdg 6y agoYep, I've been a professional programmer for about twelve years now working in all kinds of large scale systems at a certain internet company. I can say definitively that I've been saved by functional and end to end tests far more often than I have been saved by unit tests. Not that you shouldn't do both, but err on the side of writing tests that will find bugs in the entire system.
- Sodman 6y ago> err on the side of writing tests that will find bugs in the entire system While I agree these tests are going to be closer to your real production system, they also usually take longer to write, and take way longer to run. I can run thousands of unit tests and still have them be an order of magnitude quicker than a full e2e test which requires spinning up a k8s cluster, for example. I agree that both have their place, but I'd err on the side of writing tests that will run quicker where possible so that my build pipeline doesn't take forever (and eventually cost a fortune in compute power).
- thrownaway954 6y agomy god.... i haven't been to that site in more than a decade. talk about nostalgia.
- booleandilemma 6y agoIt reminds me of my college days. Glad they’ve kept the moose :)
- stuff4ben 6y agoSame! I think I learned how Struts 1 worked by reading on that site. It really opened me up to design patterns and unit testing.
- Kiro 6y agoI only write black box tests that call the outer most entry point (often a request to an endpoint) and then check if the database contains the expected results or if I get the correct response.
- gwbas1c 6y agoUnit tests (true unit tests) are good for corner cases that are very hard, or impossible, to test in such situations. I once lead a file system driver project. Most of the tests used the real file system. (They tested the driver and some glue code as a "unit.") But, there were some error conditions (corner cases) that we just couldn't test via a real file system, so those were true unit tests that ran a particular piece of code in isolation. > I only write black box tests that call the outer most entry point (often a request to an endpoint) and then check if the database contains the expected results or if I get the correct response. This is the approach I used when I wrote some personal blog software a few months ago. (It was a learning project.) I was extremely happy with the results, but in this case, it was useful because this was mostly a server-side rendered web application: https://github.com/GWBasic/z3/tree/master/test https://github.com/GWBasic/z3/tree/master/test
- oxfordmale 6y agoUnit testing, in the way the author describes, is a great way to test your mocks, however, make code harder to refactor and often add little value. Functional tests, done right, tend to be much more useful as they can match business functionality: if I ask for the weather in New York, I get returned a temperature. That code will call many classes and will not be a unit test, but is more useful than mocking the getTemperature method.
- dmm 6y agoFrom my experience the downside of functional testing is that it's slower and most importantly when a functional test fails it not obvious what is broken. The counterpoint I've heard is, "But we run the tests on every commit! If they fail it must be the latest change." That's a good point and obviously running the tests on commit is a good practice but I still think unit tests have value because: 1) Even if I know the lines of code that caused a test failure I'm still not sure what invariant I've violated. A unit test failing called "EnsureOutputIsEven" immediately tells you what's wrong in a way a functional test "ValidatePdfEmbeddedVideoGeneration" doesn't. 2) Unit tests help isolate problems from miscompilation, platform bugs, and undefined behavior in a way that a functional tests can't. A unit test tells me, "There's something wrong with this specific function" and I can look at the source for undefined behavior and the generated code for problems. A functional test just says, "Something is wrong somewhere! good luck!"
- UK-Al05 6y agoYou can still assert one small thing even with larger tests. Whats the relationship between single field of input and single field of output through multiple objects? Thats the invariant you test. It's also relationship the business is actually interested in.
- exclipy 6y agoOk so you're saying once in a blue moon when a test fails, having small tests will save you an hour. And that's worth the extra time you spend every day writing and rewriting them?
- 6y ago
- commandlinefan 6y agoAlthough I agree with his definitions, I meet (and work with) many people who vehemently disagree - especially on things like "unit tests should never use a database". I wish there was some authority who could establish useful working definitions once and for all.
- tchaffee 6y agoBy definition it's not a unit test if it uses a database. Unit test means to test an individual method or function, and specifically without testing anything else. > I wish there was some authority It's an interesting idea since we do have authorities for things like language specifications. In this case though I do think the definitions of different types of tests are pretty well established in our industry. [1]. You can find the same definitions at that source in literally thousands of other places around the web, and I would be very surprised to find someone with experience claiming that unit tests should hit the database. That's not to say tests should not use a database. But that's a different type of test. [1] https://www.atlassian.com/continuous-delivery/software-testing/types-of-software-testing https://www.atlassian.com/continuous-delivery/software-testi...
- pydry 6y agoIve seen people define unit test as something that could use a database. I've seen others vehemently disagree with this. "Unit" is not a particularly well defined thing. I think this lack of agreement is part of the problem coz it makes it impossible to have a conversation with these terms. For this reason I usually avoid talking about unit tests and try to use something a bit more specific (e.g. xUnit framework test) to highlight what I mean.
- tchaffee 6y ago> "Unit" is not a particularly well defined thing. Fair point. I think in this case I would stop focusing on definitions and focus on goals. Saying that one goal with our unit tests is for them to be lightening fast, so disk access including hitting the database is not allowed. Feel free to write a "unit test" that hits the database. But we are going to run that "unit test" along with our slower running tests that we call "functional tests".
- mumblemumble 6y agoI haven't experienced that problem. I've experienced the problem everyone else seems to have, too: Too many unit tests makes things brittle over time. One small change breaks a bunch of stuff in some mechanical way. Perhaps a required parameter got added to a function, or perhaps a class got too big and needed to be split up. Either way, you sometimes end up with a slew of tests that don't compile. Testing at a higher level should help, because the tests are more coupled to the behavior and less coupled to the implementation details. To the extent that you've achieved that, tests breaking due to a change shouldn't be spurious, because the breakage is due to a legitimate change in behavior that needs to be accounted for in the tests' expectations. Wild guess: The difference in experience here can be explained by the first three words of the article. A contractor who isn't sticking around on any one project for long isn't going to see consequences of their design decisions when they take many years to incubate. And a contractor who's seen this at "no less than six different companies" probably hasn't been a long-timer at many of them. Unit testing has only been de rigueur for 20 years, tops, so, even assuming those were the only six companies, I think it's safe to assume a maximum of 3 years per company. Subtract a year or two from the incubation time because it'll take a while to comprehensively revamp a test suite. A lot of Shakespeare's plays seem to end on a happier note, too, if you always leave the theatre right after the third or fourth act.
- rectang 6y agoThe best ROI I get when writing tests is from thorough unit testing of the most fundamental, most heavily used low-level components. That's because components like "StreamReader" or "Lock" or "Response" tend not to change much over time. In contrast, higher-level tests often break because the high-level components the test change frequently in response to business needs. I don't get this meme that unit tests are "brittle". Are the interfaces to your low-level components changing all the time? Perhaps they are not written with "loose coupling" in mind and are exposing needlessly fragile and complex APIs?
- mumblemumble 6y agoIt's frequently the domain model objects themselves changing that causes all the spurious breakage. This is not nearly as much a problem in dynamic languages, especially if you tend to do data-level programming instead of defining a lot of custom types for your domain modeling. And I suspect that this is exactly why classical TDD was first popularized among Java developers, while the London school first gained traction in the Ruby and Python communities.
- UK-Al05 6y agoThis must be a very old article, in the last 5 - 8 years the best ideas in unit testing has changed. Highly mockist code bases have ended up with their own problems when it comes to change and refactoring. Not read/writing files, dbs etc fine. Mocking out every object causes a lot problems. Refactoring a single method signature? Now the tests for the object, and each one its consumers(where it is mocked) break. Specify a public api(Testing is a good opportunity to design it), and test through that. Making everything public, and testing every method? You're going screw up your ability to change.
- JoeNr76 6y ago"The smallest amount of testable code. Often a single method/function," I hate (HAAAAATE) that definition of unit test. That's how some IDEs implemented unit tests, but it's not what was originally meant with the term 'unit' The unit tests the author means has no value outside testing while you are programming the code. For real value, unit tests shouldn't be bound to the implementation of your code. Unit tests should test a "unit of functionality", not a technical part of the code.
- nerdponx 6y agoThat's why they said "often". A better definition might be "the smallest amount of testable functionality".
- WolfOliver 6y agoactually the origin of the term unit test means a test can be executed as a unit, it does not has dependencies to other tests. It had nothing to do with testing code in isolation.
- ufmace 6y agoI think the types of testing that are useful are critically dependent on the language we're talking about. If you're working with a dynamic language like Ruby, Python, Javascript, then it's more valuable to have micro-tests on every little thing, because there's no compile step to find typos or type conversion errors. If you're working with something more static, like C#, Java, Go, Rust, etc, then it usually doesn't make sense to write tests like that. Better to stick to higher-level tests that ensure your classes are working together correctly. Except for the case of particularly tricky algorithms of course. I also say, don't be scared of hundreds of tests breaking. Very likely there is only one actual bug. Pick one failing test, run that one alone, fix whatever is broken, and then run the suite again. Usually you'll have fixed most of them.
- ravenstine 6y agoThey're also dependent on what it is that you're building. If you're writing an application, the value of unit tests is limited because the APIs internal to the application aren't being consumed by another developer but instead a user who's going to click/tap through an interface. So, in that case, it makes more sense to test the behavior expected to result from user interaction rather than test the return values of a given function, for instance, because it doesn't matter if a function does the right thing when the app is rendering a blank screen. You can test way more functionality at once by testing at the integration level or by writing "application" tests. Unit tests are best suited for two situations; either you've got a function where it's simply faster/easier to make sure it does the right thing by using a unit test, or you're writing an API that other developers will be consuming in their own applications.
- rectang 6y ago> If you're working with something more static, like C#, Java, Go, Rust, etc, then it usually doesn't make sense to write tests like that. I wouldn't trust myself to write code without unit testing it, nor would I trust you, nor would I trust anybody else. It is all too easy to make mistakes like off-by-one errors or other edge case misbehavior.
- TillE 6y ago
- roland35 6y agoI partially agree with the author, I think unit tests are great to ensure functions and methods are working correctly, and they have saved me a few times! However, here is my perspective from a hardware point of view: I think teams can be to quick to "mock" away real hardware from functional tests, and this can lead to hard to track bugs with communication. For example, on a project I worked on there was a bluetooth device and phone app. The team created a simulated device inside the app which mocked all communication. Things worked ok... until they didn't! What they should have done from the beginning is create the mocks on a "real" device and keep the Bluetooth communication in integration and functional tests. Bluetooth is a notorious pain in the neck but this is true for any external device or service. A mature project should have tests at multiple levels, from unit up to system level integration tests. The more integration you can provide developers the better, because then the surface area of untested hardware and software is smaller.
- flerchin 6y agoI prefer protected for methods I shunt like this over package private, but whatever. Shunt is a good word for this.
- noisy_boy 6y agoAtleast in context of typical CRUD APIs, a set of functional tests that run against an in-memory database like H2, are far more realistic and practical in catching issues while being very fast. My typical approach for such scenarios is to define, for every test: a) the base data, b) inputs to the service wrapper method (that is typically invoked via the handling method upon REST call (e.g. in case of a POST request, the body DTO) and c) the post-change expected state - all as JSON test assets. Then before every test, I can load the base data at the start of the tests, call the method with payload DTO, extract the final database state and compare that with expected result state (changes are rolled back at end of each test so they are isolated). I don't need to care about what the nested services/methods are doing as long as the expected state matches. This approach has helped me catch issues time and again and I would pick this over unit tests for typical CRUD APIs. One scenario where I would prefer unit tests is computation focused methods that would benefit from thorough testing using a wide variety of input values.
- barrkel 6y agoThe best testing strategy is situational, contextual. Categorical statements are rarely universal. Is the abstraction stack deep? Integration tests cover a lot more, while unit tests won't give you good confidence that the composition works. Unit tests are useful close to the control flow leaves, and where data flow is bottle-necked. Unit tests are often busywork in the middle of the stack where code up, down, left and right needs to be mocked or stubbed, and your test is just a restatement of the implementation logic written inside out. Hopefully there's enough of a modular architecture that whole submodules can be replaced with a simple API for higher level testing. Is the abstraction stack shallow and the data model simple? The difference between an integration test and a unit test is small, and if you replace the expensive bits (database, RPC generally) with mocks, fakes, stubs, then they're almost the same. Is the data model complex? Unit test setup is going to be expensive for every combination, while integration tests can leverage the code itself to generate the data model. Unit testing will likely only check one dimension of data complexity, but complex data models have multiple dimensions and for good coverage, you need cross-product tests which are expensive to run even as unit tests because of combinatorial explosion. I don't think there's a cheap simple answer. You need a mix across the spectrum and try and not run tests when you don't need to (smarter CI). Is the design changing a lot? Unit tests will be very brittle and you'll spend a lot of time rewriting them, as abstraction layers merge and split both horizontally and vertically. Integration tests are much more valuable. Having a lot of unit tests can inhibit refactoring, and refactoring debt will ironically create more tests to guard against regression from the bugs that come from complexity. Do you need to know that customers are going to be able to perform their basic routines right after a release? Unit tests won't give you this confidence; you need the highest level integration test driven from the UI, a smoke test. I don't think telling people to concentrate on one thing vs another is useful without context.
- x87678r 6y ago> Usually, for every pound of production code, I would like to see about two pounds of unit tests and two ounces of functional tests Twice as much code in unit tests than prod code? Maybe for a critical library. For most applications I'd think that is crazy.
- outworlder 6y ago> Twice as much code in unit tests than prod code Easily. Sometimes more, depending on the complexity. I had a configuration file generator with 5x as many lines of code being testing code. The consequences of getting even one field wrong would be severe, if they made to production undetected. There was some hairy legacy logic that I had to emulate. The thing is, you need to test all your corner cases. These are more important in unit tests. If the code is simple then you will be able to get away with much less. Every branch you add increases the complexity a lot.
- ensiferum 6y agoI've found the same. Unit tests easily have 3-5x the code that is being tested. But most of the code is very very straightforward with little complexity. I.e call this method with N different arguments covering edge cases border values illegal values and legal values. Example https://github.com/ensisoft/newsflash-plus/blob/master/engine/unit_test/unit_test_session.cpp https://github.com/ensisoft/newsflash-plus/blob/master/engin...
- bobwall 6y agoI agree that unit tests become the end all be all for many developers. For web development, I have gotten into a good pattern, where we have some reasonable unit tests. Then on the api, we have Postman tests that run on CI using the Neuman npm. Then for e2e tests, we use GhostInspector, which can also run in your ci pipeline. We do two sets of tests in Ghost, a set that are pretty happy path, and then some really complex ones that run between different sites etc. The first set run during CI, and if they break, then the build is broken. The others, we don't cause the build to break, as they are a bit more brittle.
- tel 6y agoMost discussion and arbitration about testing comes down to the anemic way programmers talk about interfaces. We say all the right stuff, but then skimp on the details. "Program to the interface, not the implementation!" Oh, but, what then is the interface? An interface should be understood as the minimal set of things one may expect to be true about some collection of related abstract types. - The methods for creating/introducing and destroying/eliminating values at these abstract types - The expectations around identity (memory model as relevant) - Universal properties that must hold (for *all* situations like this, the following will hold) Test specification is a component of naming your interfaces. In that sense, it's wild that test specifications don't live in the same place as interfaces most of the time. Note that test specifications are not necessarily executable. I can say "for all integers n and m, n x m = m x n" as a test specification, knowing that this is probably not feasible to actually test exhaustively. A test implementation may be partial, limited. The point of this is that it translates the question of "what test should I write?" into "how do I design interfaces, the expectations I allow of users?". The latter is a dramatically better thing to be spending your time thinking about. In my opinion, it subsumes something like 80% of the discussions people have around testing best practices.
- gnusty_gnurc 6y agoAgree with this absolutely. I'm dealing with this a lot lately. Namely having to work within a spaghetti of procedural code...told to use bizarre global scope variables disguised as static classes (Java). Wading through it all and making proper tests is so much more difficult than it needs to be. The messiness of even determining what's going on seems to reflect the inability to think about interfaces and construct objects accordingly. This is simple stuff. Java's got tons of standards, and none of them are used. This is OOP...if you've got objects, use them!
- bcrosby95 6y agoThis originally meant relying upon the public declaration of functions and methods rather than implementation details. Over time this has been warped to mean the "interface" keyword in OO languages. I prefer to code to the non-keyword-interface. Turning everything into a keyword-interface has zero value add and causes a proliferation of shitty keyword-interfaces that are not actually points of extension and polymorphism, which is what keyword-interfaces should be. So I would actually say they add negative value to a codebase.
- nunez 6y agoI never knew a partial mock was called a shunt.
- mristin 6y agoI find the combination between unit tests, component tests and contracts the best. Contracts allow you to test with tons of data you wouldn't have ever generated manually when enabled in component and e2e tests. The contracts obliviate often the need to write extensive unit tests. As they are part of the interface, we can avoid thus to re-write many unit tests when the interfaces change during a refactoring.
- bluGill 6y agoI disagree - I always test a real database, never mock it. This is because in my embedded domain we often ship with an empty database, so it is no big deal to add just the tables I need with whatever data I want, and then test it. Sqlite makes this easy for us. Likewise filesystems on modern operating systems are fast (there is a cache): I cannot measure the time difference I save mocking the filesystem calls out, so it is easier and safer to just work with files I create in the tests. It might some effort to ensure my tests put files in places where they won't collide with other tests, but this isn't hard once you have the idea in place. Since our plan was put into the code early on most people don't even consider bypassing it for some hard coded locatoin and so it is easy to use the plan on test files located in a different location. Note that the above is specific to my domain. I understand that some of you deal with one database so there is no abstraction that lets you substitute a test database you create on the fly, something not worth fixing since you still cannot afford a license for every developer and CI machine. I'm sure there is a similar argument against using a file system. The point: before saying don't use something expensive in tests do a benchmark: prove it actually is expensive. You may be lucky like me and discover that databases and filesystem access in tests are not expensive!
- dpc_pw 6y agoThe reason to mock out IO is not that much about the performance, but isolation. It's annoying to have to deal with setting up database instances, users, permissions and what not, just to run an otherwise standalone test. Also annoying when two test suites running in parallel conflict (though it can be avoided). If you use sqlite, it's much easier since it's all just working on standalone file. > I disagree - I always test a real database, never mock it. I always put an interface between the IO part and rest of the code, so I can either mock it or not. Ideally I would have a separate tests making sure that the real-IO implementation of the interface works, and not use in my normal test that are exercising logic. But it is indeed a lot of additional work, so I'm cutting the corners and let the test run against a real db implementation, only wrapping it in things like injecting faults etc.
- adamc 6y agoPeople do this because good unit testing often requires a better design, which is hard -- and, for existing code, may have negative ROI.
- kens 6y agoThe article is currently tagged with (2019), but it seemed a bit dated so I checked and it was written in 2012. https://web.archive.org/web/20120405045819/http://www.javaranch.com/unit-testing.jsp https://web.archive.org/web/20120405045819/http://www.javara... (In general I don't like HN's concern with tagging posts from previous years, but in this case I think it is warranted. It also raises the question of why so many people are upvoting an old article on unit testing.)
- trias 6y agonot just warranted, the (2019) is very misleading.
- dpc_pw 6y agoI indeed see people calling pretty much any tests, except e2e test unit-tests. However as per which test types to use: the right answer is (as with everything) "it depends". When doing testing the overarching problem is that the testing was well thought. Different types of testing will behave differently at different scales and different problems. Instead developers oftentimes thing in terms of WHAT, not WHY. One can't just do the same thing for completely different projects and expect the same results. Most of the time best testing is opportunistic - one that does what's easy and fast to achieve. In some apps functional test will test mostly separate and shallow code paths, which will not lead to a lot of coupling. With some languages a good type-system is already a 90% of unit test suite. A big part of testing is also designing your production code to be testable. (TDD is good here because it forces you to think about it upfront). With a right architecture even e2e test can be great.
- deleted 6y ago[deleted]
- djohnston 6y agoUnit testing is a waste of time IMO, only functional tests are worth the effort. My entire experience with unit tests these days is fixing something because the implementation changed and the behaviour didn't. They're a waste of time.