39 ms·
Unit Testing Is Overrated
- Marciac 6y agoI believe unit tests have a fundamental flaw: you're trying to solve a code problem by using code. Yes it can work out in the end. It also blows like code blows: waste of time, over-engineering, legacy code, too-big codebases, bugs, etc.
- mikl 6y agoBy whom? Compared to what?
- alecbenzer 6y agoCompared to integration testing and end-to-end testing.
- ChrisMarshallNY 6y agoYou may find this article interesting: https://medium.com/chrismarshallny/testing-harness-vs-unit-498766c499aa https://medium.com/chrismarshallny/testing-harness-vs-unit-4... I think unit testing definitely has its place. I use it a lot; but I have learned to moderate my reliance on unit testing. I tend to prefer test harnesses and manual (or automation-assisted) testing. I've sometimes written my own unit-testing frameworks, because the "canned" variants didn't give me what I needed. The term "unit testing" is quite old. It seems to mean something different, these days, from what it used to mean. As far as I'm concerned "not testing my software" is out of the question. For most of my projects, the testing code is vastly greater than the actual product code.
- mytailorisrich 6y agoUnit tests are very important and very cheap. They safe a huge amount of grief later on. As for everything, the pros and cons should be weighted to reach a practical and effective approach. For example there is rarely a need to test every single function. Overall, I don't find this piece very insightful.
- sys_64738 6y ago> For example there is rarely a need to test every single function. And that's where the bugs end of being. People suck at writing code so it needs to be reviewed by peers and thoroughly tested. If you said that in an interview for code you'd written then I'd point to the door.
- mytailorisrich 6y agoIt's always a question of balance and return of investment. It's good to aim for 100% coverage, but it's a diminishing return exercise and it may not be particularly valuable to reach it, so in practice it usually remains an aim. > If you said that in an interview for code you'd written then I'd point to the door. That's an extremely arrogant thing to say. Any experienced dev will understand my point, so...
- sys_64738 6y agoMost programmers are arrogant and that's why they write terrible code full of bugs. I posted elsewhere that if your code isn't tested then it's broken. It was sitting at negative four when I last looked. That comment should be in the hundreds. If folk don't understand that then they need to look into their software development practices.
- asadkn 6y agoTypical 100% coverage metric vanity with usually no real world benefits. > If you said that in an interview for code you'd written then I'd point to the door. And they would be better off not having to work at a place with an intolerable environment of the kind.
- sys_64738 6y agoYou aim for maximizing code paths in the minimum test cases. Test cases written correctly can be quickly reused with different input values. Design to test. Prove your code through automated unit testing. Skipping proper test development leads to terrible tech debt when you try to maintain the code. Error paths are generally the place that doesn't get most coverage and those are where you crash a lot of programs. This stuff is elementary.
- friedman23 6y agoI initially reacted very negatively to this article but decided to entertain the idea anyway and I have to say I'm convinced of the idea that they are overrated. Now I'm not saying that they are unnecessary and i definitely believe they are needed but i do think they are overrated. I've worked on many buggy systems that had very good unit test coverage. It was only with sufficient integration testing that were able to prevent constant regressions.
- mytailorisrich 6y agoUnit tests are not integration tests. Of course if your integration tests are lacking you'll have problems but this does not tell you anything about the value of unit tests, just that integration tests are obviously valuable.
- catdog 6y agoUnit tests have their place but in my experience there are also a lot of places in most codebases where they don't deliver enough value compared to the effort put into both writing and maintaining them (they can be a huge PITA while refactoring). I agree that in many cases integration tests often are so much more useful in catching regressions.
- mytailorisrich 6y ago> they can be a huge PITA while refactoring I see that as an useful feature: This shows you the cost of refactoring and the fact that this cost includes re-testing everything.
- dmos62 6y agoI see unit tests as executable documentation. So it's nice when code you're getting familiar with has good tests. That said, I don't write them myself, except when the code is complicated and difficult to predict, but then I'm more likely to refactor it.
- disgruntledphd2 6y agoThe author agrees with you, in that he makes the same recommendation around testing the functionality rather than the functions.
- david_draco 6y agoKnowing that some component provable works correctly is underrated though. I would favor - 33% unit tests (of well-defined units, such as functions that compute some subtle math logic, or read/write marshalling) - 33% integration tests (requiring multiple components to work correctly to achieve the result) - 33% business tests (automatically steering the entire application like a user would and testing the result like a user would see it)
- david_draco 6y agoAnd the remaining 50%: making it easy for people to give you useful feedback, such as error messages giving a web link to a user-friendly bug tracker. Ideally the link already fills in the stacktrace and system info.
- kemiller2002 6y agoI always argue for this and get told, “that’s not for the user. It’s adds screen bloat to the UI.” I never understand why people say that. How about we make it easier for us to help people. I mean at the core that is kind of our job.
- chha 6y agoThis! Not just for the systems development side of things, but just as much for normal usage. I tend to do work for fairly small organizations (<500 ppl), but with layer upon layer of bureaucracy and management, not to mention cultural and geographic differences. The main goal of the systems I work on is to provide technical documentation of complex industrial processes. If things break, it can be pricey and/or dangerous. Having good information is a must. However, if a user sees that something is off or just plain missing in the sometimes 30+ year old documentation, the easiest way to deal with it is to make a note of it and adapt to it for his or her work. Reporting the problem back in order to get it fixed is....difficult. There probably is a process for it, you probably don't have an account where you can log the time spent on it. Having a quick and low-threshold way to report problems would be of enormous value in the long run.
- unbendable 6y agoI read it and I already knew that I had this discussion a bazilion times, just like every comment in HN probably won't be anything new to me. Some topics are just made to be discussed forever
- sys_64738 6y agoIf it's not tested then it's broken.
- pachico 6y agoI can imagine you can argue in favour of any statement, given your custom, ad-hoc anecdotical use case. However, in general terms, unit testing (together with a very clear domain design) are the most solid pilars of your software.
- exdsq 6y agoWhat’s with the recent HN push against unit tests? Yes you need other tests too, but they serve a purpose. You can’t build on bad foundations! And the search space of integration tests is larger so it’s much harder to have good coverage of non-happy paths
- schwartzworld 6y agoyou can have good experiences with unit tests or bad ones depending on the health of your codebase. Hard to test code is a smell that many people interpret as being a problem with the concept of unit testing. Most of my experience is with React, and the majority of react devs I've seen don't unit test their code, because they don't write enough pure testable functions. As a result the community has leaned heavily on React Testing Library which is AMAZING for integration testing. Instead of checking that your function returns the right value, people will mount the component and then check to see that the right value is displayed in the rendered DOM. This obviously works, but writing the logic as a pure function and unit testing that function with a lot of different inputs gives you much more confidence.
- zaphar 6y agoI think it's a critical mass of people experiencing the "unit tests pass but the code is broken" problem. When unit tests are used to test glue code you end up testing your mocks and nothing else. Mocks are often done using a framework which encourages one off implementations per unit test. This introduces a maintenance burden where all the mocks have to be kept in sync. Unit tests are very useful but we somehow landed ourselves in a place where we have lots of line coverage in our unit tests but little confidence that the code actually works. There are at least 3 different applications at work where they unit tests are green while the code is broken or red when it's not. The cause is almost always the mocks. They either presume too much, making then fragile or they are flat out incorrect. Despite having a lot of test coverage there is little confidence for the developer that their change is correct. In a sense the writers of the tests were "doing it wrong". The burst of articles on unit tests and their failure modes are a reaction to the prevalence of this in our industry.
- 6y ago
- m12k 6y agoI disagree with the notion that making your code testable in isolation serves no other purpose than to write unit tests. It very specifically forces you to think about how and why each piece of code is coupled with other code, and generally requires you to make this coupling as loose as possible, to make testing in isolation possible. Loosely coupled code is also easier to reason about and easier to refactor. So testing doesn't just provide you with the value of tests, it also nudges you toward a saner architecture.
- alecbenzer 6y ago> Loosely coupled code is also easier to reason about and easier to refactor. Can you expand more on this? I think this is where the author would disagree. E.g., how is the code easier to reason about or refactor having introduced a location service interface that has only one one implementation?
- projektfu 6y agoSuppose you have a unit that makes a couple of api calls to set itself up, then performs a computation, then does some sort of storage. Thinking about how you would test this might lead you to an inversion of control approach, and you might isolate side effects. The storage provider might get passed in, both to make it easy to mock, and to reduce coupling and surprise.
- alecbenzer 6y agoI meant expand on how those things make code easier to understand, not why unit testing causes you to adopt them.
- projektfu 6y agoMainly it reduces surprise. If you call an interface with implicit dependencies, you won’t know if why it’s breaking without debugging and making sure it sets up its dependencies properly. If you call a testable interface with explicit dependencies, you can mock out those dependencies to debug parts individually.
- blackoil 6y agoFunctional Code with no side effect should be unit tested. Integration Code which glues various components together should have integration tests. If you feel like you need unit tests but have to create too many mocks, you have merged functional and integration code, separate them out.
- candiddevmike 6y agoUnit tests are insurance for later refactoring and library upgrades. This let's you avoid premature abstractions as you can easily swap out lines of code and verify you didn't break the app. This is especially important when you aren't the person doing the future refactoring.
- barrkel 6y agoTests are insurance for refactoring which doesn't change the interface that is being tested. Refactoring usually changes interfaces. Things are factored differently. The clue is in the name. The higher up the stack your test is, the more insurance it gives you for refactoring. The lower downs the stack it is, the more likely it is to be thrown away or heavily rewritten after refactoring.
- throw_m239339 6y ago> Refactoring usually changes interfaces. No, refactoring shouldn't change public interfaces. The very definition of refactoring is rewriting code without changing interfaces. > Things are factored differently. internally > In computer programming and software design, code refactoring is the process of restructuring existing computer code—changing the factoring—without changing its external behavior. https://en.wikipedia.org/wiki/Code_refactoring https://en.wikipedia.org/wiki/Code_refactoring You got the definition of refactoring wrong, please get it right, it's important. If you are breaking a public API, you are not refactoring anything. Any piece of code meant to be private shouldn't be unit tested at all, only the behavior of a public interface. Now internally you might call a third party lib, but that third party lib is then a separate "unit" itself. I don't like the term "unit" because it's yet another word that is easily misunderstood and lost its original meaning with time. unit testing should really mean "public interface behavioural integrity testing" or something like that.
- deleted 6y ago[deleted]
- UK-Al05 6y ago
- jacquesm 6y agoUnit tests are not a goal, they are a tool. Striving for 100% test coverage is nonsense, not testing your software at all levels is bad. Middle ground and moderation are where it's at, not a black vs white choice. Just like every other tool you should understand it's strengths and weaknesses and you should apply it properly, not dogmatically or it will bite you.
- alecco 6y ago> Striving for 100% test coverage is nonsense Tell that to SQLite guy.
- nottorp 6y agoSomething like sqlite can actually be unit tested to 95% or so. If you stretch the definition of 'unit test' a bit (don't mock i/o). Try that with a networked application that takes user input though...
- laumars 6y agoMocking networked services is something very much worth doing because you can then check you’ve set timeouts correctly, handling incomplete or junk responses gracefully etc. Those are the kind of hidden problems that can bite you on production deployments.
- nottorp 6y agoYou can mock networks all you want and you'll still cover only what you can think of. It will always break on the user's internet, because it's too diverse to predict. Doesn't mean you can't have some networking unit tests, just that you shouldn't believe in them too much. Edit: you said services. You thinking of server? I'm thinking of clients.
- tremon 6y ago"mocking networked services" is exactly what you would do when testing clients. And it doesn't have to be a static mock. It's not too hard to inject a fuzzer in your mock service response, although that's probably left to a separate testing routine, and not part of your unit test setup. But if you have no mock for your network service, you can't fuzz it either.
- zulgan 6y agothis is our take on it: https://eng.rekki.com/unit-testing-at-rekki/t.txt https://eng.rekki.com/unit-testing-at-rekki/t.txt TLDR: * test your core, make sure your core is strong * don't test your http api * don't mock * don't test writing and reading from the database * don't complicate your code to make it testable * ... unless you deem fit
- mav3rick 6y agoIf you don't do 2 you may as well not test at all. You may do everything else and still fail your end APIs. Your users don't care about your core or mocks. They want end APIs to work The rest just seem arbitrary. Can you make a change and be confident it works ? The last point sums up everything. "Use your best judgement"
- kentosi 6y ago"don't mock" How are you meant to write a unit test for a class without mocking out external dependencies? Wouldn't that make it an integration test?
- bencoder 6y agoNot the OP but I count integration testing as testing integrations /between/ systems - wherever your code depends on something outside your codebase. You can mock this for unit testing. Don't mock things that are already in your codebase. Use the actual object. Common argument against this I've heard is "But then when something goes wrong it is harder to figure out where the problem is" - I have never actually experienced this myself, but I have, very often, experienced being reluctant to do any refactoring because I'd have to rip up all the unit tests because they are testing only implementation details
- APhoenixRises 6y agoYou definitely have some strong opinions about testing. I'd suggest being more specific so your guidance is more clear. "people who test what they think is appropriate" is really subjective and makes it seem like there's two camps: people who "get" the author's perspective, and those who don't. I've tried the "do" and "don't do" approach, but that's not enough information usually. Teach by examples. Lots of examples. Calling things BS isn't going to change someone's mind. I'd also re-enforce that these practices are for your codebase. Every project is different and has different challenges. There's a reasonable counter-example for each item in your TLDR, and "unless you deem fit" doesn't feel like it gives developers leeway given how strongly you state your opinions.
- bencoder 6y ago> There is no formal definition of what a unit is or how small it should be, but it’s mostly accepted that it corresponds to an individual function of a module (or method of an object). And there-in lies the problem. Remove the idea that the unit is a single method/function. I subscribe to the idea that a unit is a unit of functionality. Nothing to do with the code implementation. Only mock where you're reaching out outside of your codebase (filesystem, network, operating system (time, for ex.)) You can still do unit tests for individual functions when you need to work on a complicated algorithm, but those functions should have no dependencies or side effects - pass in all the data you need
- nordsieck 6y agoOne interesting thing that is easy to notice about all of the examples of in the article is that they are absolutely infested with objects. I don't have anything against objects, per se, but I think they tend to make unit testing much more difficult to accomplish. The closer your code resembles pure functions, the easier it is to do dependency injection and unit testing.
- disgruntledphd2 6y agoYup, this is very true. As long as you return something deterministic, unit testing is easy. It's where you need to handle mutable state with objects that things get trickier. Unfortunately, these are exactly the places where you most need tests. I'm a big fan of constantly returning things rather than holding state in objects, for specifically this reason.
- HelloNurse 6y agoA basic issue of encapsulation: objects shouldn't rely on the mutable state of something else, but only on their own.
- zarathustreal 6y agoAgree. To add to this, many unit tests that you might have to do become obsolete with a strongly typed functional language. At that point you’re basically only integration testing the API boundaries / external interfaces.
- pjmlp 6y agoThere isn't pure functions the moment you touch any kind of IO. Plus the same problem arises with modules instead of objects, which traditionally are even harder to customize.
- nordsieck 6y ago> There isn't pure functions the moment you touch any kind of IO. You can get pretty far with good abstractions and dependency injection. Go's io::Reader and io::Writer interfaces are a great example of this. The resulting functions aren't pure in a technical sense, but they're pretty easy to unit test none the less. > Plus the same problem arises with modules instead of objects, which traditionally are even harder to customize. Maybe you could elaborate. I really don't understand what you mean here. From what I understand, modules just scope names, they don't maintain state. I don't see how they have the same problems as objects.
- yomly 6y agoMy own take on this is that it is hard to crystallise why or what a "unit" is. My cop out is it comes down to experience - once you've done this enough and solved enough problems and worked on enough teams, you'll get an intuition on this as it is more art than science. I find it to be some mixture of importance and complexity, while always balancing against the single responsibility principle. A simple `average` function might be trivial but if it's important to your business logic you probably want to test that separately, even if it is nested within a "unit". I find that following some iteration of "functional core imperative shell" helps here as it helps keep your core business logic to being data transformations and transformations on data are easy to test and their concerns are easy to reason about. This then helps me reason about what is "implementation" and what is a contract/interface which should be tested robustly. I guess really the art of writing just enough unit tests is to identify the seams and boundaries of your abstractions in your codebase, and potentially accepting that business seams in your codebase may be different from the seams of your actual software domain -- the latter being possibly more granular.
- drbojingle 6y agoUnit Testing, yes, it can be a waste of time. Depends on what you're doing. Unit Testing fails when you sink more and more time into trying to make a test for something because 'duh, unit test everything'. Fact is some code changes a lot and some code doesn't change much. Some code is also hard to unit test and some isn't. The intersection between code that doesn't change much and code that's hard to unit test should NOT be unit tested, especially if another form of testing works better. There's just no need to sweat over a test that wont run enough to justify the work that went in to writing it. The issue here is developers don't have a sense for economics. Diminished returns, marginal utility, and opportunity cost should be studied by ALL.
- pfdietz 6y agoUnit testing reveals latent defects, defects that can't be triggered by the system as it currently exists. Each such defect is a problem waiting to pop out in the future, as the system changes and the latent defects become exposed.
- alecbenzer 6y agoYes but discovering those hypothetical defects is usually less important than discovering the actual, currently present defects.
- goodoldneon 6y agoWe initially only had integration tests, because many people think they're better. I get it: itests use the real plumbing, so they're more representative of your runtime. But they're slooooow -- especially the tests that involve the DB (which is most of our itests). So we started adding unit tests. Utesting code that wasn't written for utests is painful: you often need to choose between refactoring or just patching the hell out of it. The latter is highly undesirable, since it leads to verbose tests, failures when you move a module, and the inability to do blackbox testing. But utests encourage our new code to be clean and readable. We've found that functional programming is much easier to test than object-oriented, and is easier for engineers to grok. We just sprinkle a little dependency injection and the whole thing works nicely. Itests have their place, but utests lead to faster feedback and more readable code.
- zarathustreal 6y agoWeird that you started using a functional approach, noticed that it’s easier to unit test, and drew the conclusion that unit testing is what led to more readable code. Consider that functional code is the source of readability. Also we don’t typically call it “dependency injection” in the functional world
- goodoldneon 6y agoYou're absolutely right: functional code is the source of the readability. But writing unit tests incentivizes engineers to keep things functional. What's a better term than "dependency injection"? What should I call an argument whose default is always used in production code, but is there to make passing a mock easy? I'm not trying to be snide -- I'm genuinely curious.
- cnity 6y agoOT: Good Old Neon is a fantastic piece of writing.
- bigmanwalter 6y agoI always just called it a "default argument"
- chimprich 6y agoI tend to mentally divide code into roughly two types: "computational" and "plumbing". Computational code handles your business logic. This is usually in the minority in a typical codebase. What it does is quite well defined and usually benefits a lot from unit tests ("is this doing what we intended"). Happily, it changes less often than plumbing code, so unit tests tend to stay valuable and need little modification. Plumbing code is everything else, and mainly involves moving information from place to place. This includes database access, moving data between components, conveying information from the end user, and so on. Unit tests here are next to useless because a) you'd have to mock everything out b) this type of code seems to change frequently and c) it has a less clearly defined behaviour. What you really want to test with plumbing code is "does it work", which is handled by integration and system tests.
- davidcorbin 6y agoThis. Making the distinction between the two is huge. Save time and money testing the right pieces of a codebase.
- knubie 6y agoI've seen this concept called by many names including CQS[0] and Functional Core, Imperative Shell[1]. I'm just leaving this comment here for those that are interested in reading more. [0] https://en.wikipedia.org/wiki/Command%E2%80%93query_separation https://en.wikipedia.org/wiki/Command%E2%80%93query_separati... [1] https://www.destroyallsoftware.com/screencasts/catalog/functional-core-imperative-shell https://www.destroyallsoftware.com/screencasts/catalog/funct...
- pydry 6y agoFunctional / imperative doesn't exactly map on to these two concepts. "Computational" is often imperative and integration code isnt always imperative (a lot of react code would fit in this box, for instance).
- eropple 6y ago"Computational" code that isn't by the vernacular definition of "functional"--and you can write functional code regardless of programming language--is something of a red flag to me. Operate only on your inputs. Return all of your outputs. No side effects.
- moomin 6y agoThe example is a straw man. He refers to a doubling in code size, but that's just because he's got hardly any code. In a real class, you still have one method (or two! or three!) on the interface but you've probably got 300 lines in the implementation. The only thing being "duplicated" is the external interface. Might as well retitle this "Why I don't like the Interface Segregation Principle".
- squirrel 6y agoThe units here are poorly designed - for example, I'd expect the LocationProvider to be responsible for choosing how to get the location, not the caller - and this makes them hard to test. The solution is fixing the design, not throwing out unit tests entirely. The standard reference on this is http://www.growing-object-oriented-software.com/ http://www.growing-object-oriented-software.com/ .
- edpichler 6y agoThis title is wrong. It's not the tool's or technique's fault. Unit Testing is not overrated. It can have been overrated by some of us sometimes or most of the time, depending on who you are.
- DrBazza 6y agoA couple of ways of looking at unit tests: * a 'journal' - at the date this test was written, this is how we expect the system to behave * an ELI5 - if I'm trying to use this method, why am I passing all these complex objects? Unit tests declare expected behaviour, and should make the developer think about their methods. For example, why pass complex objects to just to print a string or a count or similar? And why pass those objects? could a method be generalised to take a lambda, or an interface instead? Could the method be pure? And so on. Unit testing isn't overrated, just a bit misunderstood.
- DeonPenny 6y agoTry reformatting a large code base without unit test. You'd pull out your hair trying to rename a variable.
- pjmlp 6y agoThankfully using strongly typed languages and IDEs keeps my hair in place.
- slim 6y agounit tests are the equivalent of repl for languages that are not designed. you need unit tests to close the feedback loop. you can't simply write code for days without having any feedback. some code simply does not have an obvious immediate visible feedback on the ui
- cheez 6y agoI use unit tests for complex units of code and depend on frameworks to do simple things correctly. Like I don't need to write unit tests for inserting things into a database. Unless there are complications around it.
- conatus 6y agoIt's a perennial but: unit tests are more about how you think and rigourously approach a problem than preventing regression. I'd say if you wrote unit tests and then deleted them, your code would be better for it. Indeed, I often don't commit some of the tests I've written in order to write the code! I'll ship some subset of them. It's a notebook. They serve as a form of living documentation for the code and help increase velocity in a build under the right conditions. For example, you do need to know a certain function does what you think it does because the rest of the system isn't even in place yet. You might have to approach this from the outside, via integration, but the speed of doing this is quite slow. Versus a unit test. This is not to mention refactoring! The piece seems to be more about the value of mocking and how far to go with isolation. Which is a slightly tangential issue. I agree that in particular styles of object orientated programming this becomes absurd. This article, which is linked provides a more convincing case, on the grounds that unit testing foregrounds the system as software as opposed to the software as a useful and functional thing in the world, meeting user needs. However, it omits the being about to _react_ to user needs is actually a central party of agility. Unit testing allows maximum reactivity to changing requirements without regressions in the code and makes this code navigable. Changing customer needs means your carefully built functional tests are going to be just as useless and rotted. This has been the case with large test suites of functional tests - say in something like Cucumber - I've seen. Better test at a lower level, which moves slightly less rapidly. https://www.sacrideo.us/the-fallacy-of-unit-testing/ https://www.sacrideo.us/the-fallacy-of-unit-testing/
- roca 6y agoOne argument made for unit testing which this post doesn't address is that it's easier to understand and debug unit test failures. To which I say: yes it is, but quality debugging tools (which don't exist in many domains, and aren't used nearly as much as they could be where they do exist) can mitigate this issue. I'm talking about tools like https://rr-project.org https://rr-project.org and (self-promoting!) https://pernos.co https://pernos.co.
- korijn 6y agoopinions on blog posts are overrated
- deleted 6y ago[deleted]
- DougBTX 6y agoPerhaps the problem isn't mocking dependencies, but trying to hide the fact that GetSolarTimesAsync needs two pieces of data to work: a date and a location. But the original signature is just this: public async Task<SolarTimes> GetSolarTimesAsync(DateTimeOffset date) That introduces a lot of complexity: * The SolarCalculator needs to be able to work out its own location, so it needs a LocationProvider * SolarCalculator needs to be IDiposable since it owns a LocationProvider * The SolarCalculator will need more methods if it ever needs to calculate the times in a different location * If fetching the location is slow, but the application needs to calculate times for multiple dates (eg to build up a table of times), then the SolarCalculator will need an method that takes in an array of dates to be efficient But all that could be solved by making the function take all of the arguments it needs to return its value: public SolarTimes GetSolarTimes(DateTimeOffset date, Location location) No location provider needed, no IDiposable, just one efficient stand-alone method. Unit testing this is now just: var calculator = new SolarCalculator(); var actual = calculator.GetSolarTimes(new Date(...), new Location(...)); var expected = new SolarTimes(...); actual.Should().BeEquivalentTo(expected); ...so, perhaps the issue isn't that unit testing is a bad idea, but that code which is hard to use in a unit test might also be hard to use in a wider application? And perhaps the fix is to make the code easier to use?
- chrisandchris 6y agoI agree with you and that‘s most times also my observation: Unit tests tend to show how decoupled and re-usable your code is. Is a function gets hard to test with a unit test, this points usually towards an architectural issue.
- fernandotakai 6y agoif i got a dollar every time i started testing a piece of code, realized that it was waaaay to complex, refactored the code so the tests were easier to write/understand... i would have a lot of dollars. 100% of the time, it was the right idea and the code became a lot better.
- romwell 6y ago
- makkesk8 6y agoEngineering a good code base is very hard and is unique to every project and the mentality especially in the .net community of "loosely coupled code is good" usually ends up over engineering and adds more complexity than necessary. I've seen far too many cases where added complexity has killed the project's time budget because things just take so much longer to do. You need to maintain a healthy balance and actually evaluate if writing this particular system within the project in a way where we can "swap it out" later on is actually a use case. Every project is unique and "unit testing all the things" might not be the solution for your project. Where I work we shifted our mentality when we highlighted this problem to only unit test bugs and write less decoupled code and in bigger services that are more mission critical we do integration testing instead. This works very well for us but as highlighted earlier, every project is unique and you should experiment what works for your particular team because what works for others might not work for you.
- KingOfCoders 6y ago"however many find it encumbering and superficial." Author doesn't want to write unit tests. There are two factions of people on the internet, those that write unit tests and those that don't like to. Nothing has changed in the last 15 years. Whenever I write many unit tests I feel happier and more confident. If I don't and catch up writing unit tests, I always find bugs in my code. Never haven't I found bugs when increasing test coverage from a low start.
- solraph 6y agoI'm a massive fan of unit testing, but I mostly agree with the observations. However, I (mostly) disagree with the conclusion. The problems with unit testing I've seen to come from the following anti-patterns in various combinations. 1) The use of unit tests as the exclusive automated test type. ie; No functional tests, integration, etc. 2) Test doubles for most or every dependency, even purely functional dependencies like math libraries. 3) Not using the appropriate kind of test double for the test at hand. (Dummies vs Fakes, vs Spies, vs Stubs, vs Mocks) 4) The overuse of mocking libraries. Mocking libraries have their place, but in opinion, are used approximately a hundred, perhaps even a thousand times more often than they should be. I use them to create test doubles in exactly three scenarios: 1) A dependency that does not have an interface, usually a third party library. This usually happens in one place only, and is used for writing the wrapper code test. 2) A dependency that has an incredibly large interface and/or dependency graph where building a set of stubs or spies is simply not worth the effort. 3) I want to test weird edge cases that's not available any other way, such as theoretically unreachable code. These should not be the majority of your unit tests!
- mberning 6y agoThat’s all well and good until you need to upgrade some massive project to a new JDK or your Rails app from one major version to another. Then you will be wishing you had significant coverage.
- NCG_Mike 6y agoI'd suggest that someone who thinks that UTs are over-rated are over-rating their ability. I'd also suggest that those people are the exact people that should have extra testing done on their code.
- maps7 6y agoI read most of the article and I am not convinced. I'll stick with the unit tests. In my experience they improve code quality, maintainability and reduce bugs.
- cjfd 6y agoNo, it is not overrated. The author is just using a not-so-handy definition of 'unit'. When a 'unit' is a single method/class/function 'unit' testing is not in general a very wise thing to do. In most cases it is better to unit test a set of related classes/methods/functions as a whole because the communication between these things also needs to be tested. In that case it is quite impossible to overrate the importance of unit testing. I also don't think the definition the author uses is the original definition of 'unit'......
- throw_m239339 6y agoMore like unit testing in some languages is a freaking chore because it involves complex unit test frameworks. Unit testing should be as easy as writing a few lines of code, just to ensure that an interface behaves as expected, but it's often not the case because well many languages make it incredibly difficult, for various reasons, or that you end up writing walls of code (thus behavior ironically) with mocks, stubs, fakes and what not, just to test a single method... "Testing experience" should absolutely be a core concern for any modern language. More generally the notion of "developer experience" should be a core design aspect in any new language and not be delegated to third party tools. What is the language with the best testing experience?
- notacoward 6y agoUnit tests have a purpose, which is mostly to protect the programmer against future mistakes. Integration and system tests protect the user against current mistakes. I've been on projects that focused almost exclusively on unit tests and on projects that focused almost exclusively on integration tests. The latter were far better at shipping actually working code, because most of the interesting problems occur at the boundaries between components. Testing each piece with layer after layer of mocks won't address those problems. Yay, module A always produces a correct number in pounds under all conditions. Yay, module B always does the right thing given a number in kilograms. Let's put them together and assume they work! Real life examples are seldom this obvious, but they're not far off. Also note that the prevalence of these integration bugs increases as the code becomes more properly modular and especially as it becomes distributed. I firmly believe that integration tests with fault injection are better than unit tests with mocks for validating the current code. That doesn't mean one shouldn't write unit tests, but one should limit the time/effort spent refactoring or creating mocks for the sole purpose of supporting them. Otherwise, the time saved by fixing real problems more efficiently - a real benefit, I wouldn't deny - is outweighed by the time lost chasing phantoms.
- inertiatic 6y agoWhat, no. That's exactly the opposite. Unit tests protect you against current mistakes. They're tied to the exact implementation. "Right now my function X should call Y on it's dependency Z before it calls A on it's dependency B. I know that my method should do this, because this is how I designed it now. Let me write a test and expect exactly that." Integration and unit tests will tell you whether in the future your code will still work when you refactor. "Okay, we rewrote the whole class containing the function. Does running my thing still end up writing ABC into that output file?" Otherwise I agree with you mostly.
- notacoward 6y ago> They're tied to the exact implementation. If unit tests are tied to an exact implementation, they''ll fail on correct behavior and that's definitely wrong. It shouldn't matter whether X calls Z:Y or B:A first, whether it calls them at all, whether it calls them multiple times, whether it calls them differently. All that matters is that it gets the correct answer and/or has the same final effect. Unit tests should be based on a module's contract, not its implementation. This is in fact exactly what's wrong with most unit tests, that they over-specify what code (and all of its transitive dependencies) must do to pass, while by their nature leaving real problems at module interfaces out of scope.
- shawnps 6y agoEarly in my career I saw a large legacy project that was riddled with bugs turned around after a senior developer insisted on having unit tests. No one else believed in the value of unit testing, so he added them on his own in his free time. Occasionally another developer would push up some code that broke the senior developer's tests, and he gradually got the upper hand because he now had proof that his tests were finding real problems. Everyone started writing unit tests, and the code broke less. Developers became more confident in deploying, and eventually most PRs looked roughly the same: 10-20 line diff on the top, unit tests on the bottom. If there were no tests, the reviewer asked for tests. It became a fun and safe project to work on, rather than something we all feared might break at any moment. I've since started insisting on having them as well, especially when I'm using dynamically typed languages. A lot of the tests I write in Python for example are already covered in a language like Go just by having the type system.
- xtracto 6y agoI programmed the first 10 years of my life in compiled statically typed languages (C, C++, Java, etc), then I needed to start programming in Ruby for production environments and initially I felt "naked"; I felt so insecure when building something and not having it compiling successfully. That's when I really got into Unit Tests, bugs as stupid as "vlue" instead of "value" typos can plague your codebase in languages like javascript, python, ruby, etc; and testing is the only way to find them (other than... in production errors).
- ak39 6y agoMost systems in production already have key use cases that stand in for unit and system testing. Find those and use them after every new feature or change to the system. I support an existing reporting system with several hundred tables and thousands of interdependent stored procedures. This system has a fairly often-used "global report" which touches nearly 90% of the tables. We run this user report after all changes to the database with a few swaps in parameters (history vs current etc). If this report and a few others "pass" (results are the same as previous reports), we know that the changes are safe. When the reports show differences, we confirm that the changes were intended. If not, something broke.
- renke1 6y agoReminds me of snapshot testings.
- bumby 6y agoAssuming 100% coverage is a well-intentioned goal but pragmatically unattainable, what are some ways to prioritize what gets unit tested? E.g., unit test every identified hazard in a hazard analysis
- royosherove 6y agoUnit tests have a lot of value, in an overall testing strategy: https://pipelinedriven.org/article/a-pipeline-friendly-layered-testing-strategy-amp-recipe-for-dev-and-qa https://pipelinedriven.org/article/a-pipeline-friendly-layer...
- gregdoesit 6y agoThe article emits a key point, when talking about any practice: the context, in which unit testing is performed. The size of the team, the type of company, the technology, and the impact of product defects. For a startup with a small team and few customers building an MVP? Unit testing is overrated. For a company with 50 engineers in 10 teams building a product, that moved $500,000/day in revenue? Unit testing could or could not be overrated. For a company with 1,000 engineers working in the same repo, shipping a product that moves $50M in revenue per day? Unit testing is most likely underrated - and essential. You cannot ignore how the organization works, and the cost of a defect that a unit test could have caught. I happen to work at the third type of organization, and while unit tests might not be the most efficient type of safety net, it is a very big one. We have other types of testing layers on top of unit: integration and E2E tests as well. Also, one more fallacy in the article: "If we look back, it’s clear that high-level testing was tough in 2000, it probably still was in 2009, but it’s 2020 outside and we are, in fact, living in the future. Advancements in technology and software design have made it a much less significant issue than it once was." This is not true everywhere. High-level / E2E testing on native mobile applications in 2020 is just as bad as it was on the web in 2009.
- 2rsf 6y ago> Unit testing is most likely underrated - and essential. You are right, but it still doesn't mean aiming for high coverage. In the big company case you'll want to cover the interfaces and dependencies and less of your team's code. I know that part of this will fall under "integration" but definitions are sneaky.
- spacekitcat 6y agoI'm a big fan of unit tests and from my perspective, this article misunderstand what unit tests are for. Unit tests verify that the central assumptions you've made about some module's behaviour hold water. Unit tests aren't supposed to "exercise user behaviour" or "verify business logic", it verifies the theoretical behaviour of a carefully isolated module under specific conditions. A well crafted suite of tests makes it easier to reason about your application's behaviour.
- theshrike79 6y agoThis is the correct way, but in my experience less-traveled programmers start striving for that elusive 100% coverage and end up writing unit tests for completely obvious stuff like constructors (which do nothing else than setting values) or setters/getters.
- spacekitcat 6y agoDevelopers always have to estimate the cost/benefit of how they're spending their time and overly strict coverage targets get in the way of that. Test coverage targets make very little sense for a statically typed language like C#, a fools errand almost. In dynamically typed languages, it's hard not to almost hit 100% if you're doing some honest TDD. Just for example.
- GiorgioG 6y agoCode is a liability. Unit Tests are code obviously and are no less prone to contain bugs than the code under test. And of course they require maintenance just like any other part of the codebase. It feels like the industry has blindly pushed for unit testing everything and 80% or more code coverage as the gold standard. I’ve given up arguing about the cost/benefit of unit tests at work. I feel that the software the teams I’ve worked on over the past couple of decades still produce about as many bugs as before unit testing came along. I’m not building pace makers or aviation software, mostly LOB applications. Unit tests provide a false sense of security (especially to management.) Yes sometimes they help catch refactoring bugs, but at what cost?
- exdsq 6y agoI’m a software test engineer and spend quite a bit of time performing code reviews. One thing I always ask for is proof that the code works - this requires a combination of unit, integration, and e2e tests. Unit tests are great to show that some code works in isolation, and then a few integration tests can cover the functionality. You should at a minimum cover each path through a function with boundary cases, which would be far too much effort to do via integration tests.
- APhoenixRises 6y agoWhat's missing from the title of these articles is "to me" or "in the context of my project". The approach to testing a CRUD/line of business app is tested is different than the approach for testing something like EC2, which is also different than how a library/package is tested. Without context, it's really easy to argue exceptions or counter-examples to any opinion.
- billman 6y agoAny good idea in software can and will be abused to the point where it seems like a bad idea. It's all about context people.
- claudiusd 6y agoI can't believe I'm wasting my time on another testing debate. Speaking as a formerly young and arrogant programmer (now I'm simply an arrogant programmer), there's a certain progression I went through upon joining the workforce that I think is common among young, arrogant programmers: 1. Tests waste time. I know how to write code that works. Why would I compromise the design of my program for tests? Here, let me explain to you all the reasons why testing is stupid. 2. Get burned by not having tests. I've built a really complex system that breaks every time I try to update it. I can't bring on help because anyone who doesn't know this code intimately is 10x more likely to break it. I limp to the end of this project and practically burn out. 3. Go overboard on testing. It's the best thing since sliced bread. I'm never going to get burned again. My code works all the time now. TDD has changed my life. Here, let me explain to you all the reasons why you need to test religiously. 4. Programming is pedantic and no fun anymore. Simple toy projects and prototypes take forever now because I spend half of my time writing tests. Maybe I'll go into management? 5. You know what? There are some times when testing is good and some times where testing is more effort than it's worth. There's no hard-set rule for all projects and situations. I'll test where and when it makes the most sense and set expectations appropriately so I don't get burned like I did in the past.
- heyoo 6y agoYour points are fine, but I do not see how they apply to the blog post. Overall, the blog post says, unit tests take a long time to write compared to the value they bring - instead (or also) focus on more valuable automated integration tests / e2e tests because it is much easier than it was 10-20 years ago.
- claudiusd 6y agoMy point is that OP is in step 1 of 5. It's not to say there aren't any good thoughts there, but the overall diatribe comes from a place of inexperience so take their advice with a grain of salt.
- heyoo 6y agoI don't think OP is step 1. OP is not arguing against testing, although the title could lead one into thinking that. OP is arguing for better, more reasonable testing.
- ddevault 6y agoTests are most useful when they provide fast access to a code path for repeated testing during development, when the code path would otherwise be far removed from the normal course of user interaction with the software. This closes the debugging loop while you're developing new features, when it's applicable. The second time, less often, when tests are useful is when making large changes with broad implications, and quickly verifying that everything still works and that you haven't overlooked some subsystem. The least frequently useful application of tests is regression tests. 99 out of 100 regressions only happen once. If you add a regression test, it won't happen again (unless you overlooked something), but it was unlikely to happen again anyway. Generally I write the first category of tests when I feel that it would be useful to solving the problem I'm working on. Then, when I finish the code, I commit the test, because why not? It's written. It'll never fail again, but hey. This creates a reasonably managable collection of tests which, more often than not, test the more complex (and therefore more fragile) parts of the codebase, and are a decent representative sample of all of the subsystems. This provides sufficient test coverage to support the second case. The third case is so rare that it can be addressed on a case-by-case basis. The most stable software is software which doesn't change. Keep your scope small and your complexity low, and don't be afraid to mark a finish line. This is more effective than exhaustive automated testing.
- pelasaco 6y agowow, I keep reading those "test is overrated" (second article, this week), and I don't see how would I survive, maintaining a 10K LOC rails application, without unit testing my models, services, finders, etc.. I guess Unit Testing is Overrated, Long live to Unit Testing!
- mlthoughts2018 6y agoThis is comically bad. Way too long and myopic but the points aren’t even useful points. > “ unit tests are only useful to verify pure business logic inside of a given function.” Yep, that’s one of the most important things you need to verify. You should also verify integration test success, and then the unit test allows you to immediately observe where is the failure: is it pure business logic failure when all external factors were mocked? Or is it integration failure? Or is your dependency flaky & untrustworthy? Without unit tests you can’t (a) develop business logic in isolation from the external resources it will integrate with or (b) easily isolate what is a business logic error vs what is an integration error. It’s just so dumb to use language like saying unit tests “are only useful” for this. That’s a hugely valuable thing to be useful for! > “ no practical purpose other than making unit testing possible.“ This is incredibly bad circular reasoning. You must first already agree that unit tests confer no value, only then is this considered a point by the author. But if unit tests do add value (and they really do) then refactoring to facilitate unit tests also adds value! The point about testing a hidden implementation is mixed. On one hand the example used here is just a bad example. On the other hand, testing a hidden implementation can be a very good thing because it assists the act of development in the first place. The test helps the person writing the business logic to write, by factoring into a test that's proof of correctness. Maybe it’s debatable that it should be removed like scaffolding when they are done if it’s a hidden implementation, but that’s really a local decision for a team to make. Sometimes it’s good leave those tests of hidden implementations because they add extra protection for changes that can have unintended consequences. > “unit tests are expensive” yawn.
- yeahgoodok 6y agoI've been saying this for years and suffice it to say it has been a career-limiting move for me. If any new developers are reading this be warned that these are considered dangerous ideas. Know your audience before sharing controversial opinions.
- tasubotadas 6y agoI feel that this is relevant https://dev.tasubo.com/2020/06/underappreciated-superpower-a-look-into-useful-software-tests.html https://dev.tasubo.com/2020/06/underappreciated-superpower-a...
- x87678r 6y agoSoftware is not all the same. Some people are writing Auto-Pilot software, others are writing some useful tool for a few people to speed up some data cleaning for a few months. There are no rules that you must do this for software when we're all doing different things.
- js8 6y agoI agree with the article a lot. First of all, I also do not condone the idea that developers shouldn't worry about other tests than unit tests. I think the developers should be responsible for producing working code, end to end, and having automation and QA is just a nice bonus, additional verification. Anyway, regarding unit testing. Lot of the objections against unit testing becomes clearer once we start talking about it in a formal way, best in functional terms. Let's have a function y=f(x) that we want to test. In unit testing, we generate some examples (x1,y1),.. that we run this function with and compare the output. If we have two functions, f and g, where z = g(f(x)), even if we unit tested each of them separately, we still can fail because what we didn't test was if the domain of g is indeed a subset of range of f. In fact, that cannot be unit tested, since unit tests only verify logic, not the domain and range of the functions. That's the first objection to unit tests, there are holes in integration. This is especially insidious because if two different people wrote the two functions, they can each have different assumption on the domain and range, yet they won't detect it by unit testing, because they both wrote the tests that only operate under their respective assumption. (BTW, this also shows that the code coverage metrics are meaningless, unless you can coverage all the code executed down to libraries, because you can always leak the coverage through the datatype and vice versa.) Second objection to unit tests is how do you generate the test cases, in particular, the output? Well, the unit to be tested has to be reasonably small. This means more work (more mocking) and more holes in the integration assumptions, as above. Personally, I believe property based testing is superior in all respects and should replace unit testing. Property based testing forces you to write down assumptions that you have written your code with, and it scales better because you decouple generation of test cases from the assumptions themselves. So formally in property based testing, we would create a generator of input cases for x, and also a property - a function that verifies what the y looks like (possibly even given x). In fact, this approach completely subsumes unit testing, because for each of the test cases (x1,y1),.. as above we can just write a property that checks if the output of f is y1 given the x1, and so on. However, property based testing is stronger. We could also produce a property that would state that input to g has to be in its domain. Then we could easily detect the above problem with composing f and g, just by verifying the properties for our generated test cases. So it resolves the first objection. That's the beauty of the properties, they really test the data, not the logic. I believe that what we really need to test as developers is the assumptions that we put on the data structures we work with, rather than logic that the functions do. If you want to verify the logic, read the code you have written again. The second objection to unit testing is also somewhat resolved, because we don't have to produce complete test output, we only verify some of it's properties. Also, separating generation of inputs from properties let's you naturally reuse outputs from the other parts of the program for testing. You only to get the input generation right, the other things will sort of test themselves. So for instance to test g, we don't need to create an extra input generator, we can just live with the outputs of f. In essence, property based testing just makes tests themselves to compose better. It always bothers me a lot when writing unit test, how little bang I get for the buck. I need to come up with all these test examples, and they usually only cover a small piece of code. What often happens is that I actually know (in my head) the testing generator and properties, it's just instead of writing them up so that computer would understand them as well, I just write a few examples. It feels totally wrong. Ideally, I would love to see framework that would let some of the verification properties (from property based testing) to live in the code as additional runtime assertions. I think that would be much more practical approach than having a lot of unit tests, and it would tie nicely with defensive programming. Finally, I would like to point out to my earlier, more general comment I made about testing: https://news.ycombinator.com/item?id=23470259 https://news.ycombinator.com/item?id=23470259
- keithnz 6y agoThese kinds of articles criticising unit testing always seem to work out more to do with the way people go about composing their software and the design. Interface explosion and unnecessary abstractions as presented in the first part is often the nightmare people get into early in the their software development journey. But that's a problem with design. Possibly unit testing lead you down that path, or perhaps that path was shown to you by some advocate of unit testing. In the article even the final code seems too messy for my tastes, also could be simpler and more modular. Which then makes the testing more straight forward and you'd pretty much the same thing in the end.
- Dowwie 6y agoHow much wasted effort is expended by testing policies? Consider, for instance, managing development by setting an arbitrary target % coverage for unit tests. As team refactors code, unit tests will break, and the test coverage requirement will compel someone to revisit old tests and refactor them. Refactor more code and the same tests are breaking again! Must refactor the tests -- yet again -- to reflect the latest changes. Test coverage policies are very costly.. On the other hand, unit testing does present some benefits. Consider a function that cannot be easily unit tested. This tells you that it's probably too complex and would benefit by refactoring into more manageable parts. Also, business logic unit tests are beneficial.
- gchamonlive 6y agoLittle late to the party, but I wanted to cross-reference another post from earlier this week: https://news.ycombinator.com/item?id=23749676 https://news.ycombinator.com/item?id=23749676 (Code Only Says What it Does) I believe unit and integration tests, apart from checking whether code does what you think it says, it serves both as a sort of executable documentation and, most importantly, it highlights development intent. If you TDD your code iteratively, reflecting not only the unit in your code, but the intent in your tests, you get a much more healthy testing base. Code coverage is not really a good metric. Intent coverage is more interesting, but a whole lot more subjective and elusive. All in all, tests should be written not with your own self in mind, but with whoever might come later to maintain your code.
- owaislone 6y agoDepends on what kind of software you are building. If you are a 5 man team trying to move very fast and deliver new features, I think rigorous unit testing can take enough time that it won't be worth the effort but if you are working on something with a couple dozen other developers where no one person might be able to fit the whole system in their head, I think unit testing is invaluable. A few days ago I saw another HN post that discounted testing as well stating that you already need to know what to test in order to verify the behavior but that is not even the purpose of tests in large projects maintained by large teams. Tests serve two purposes in such projects. They act as internal documentation for contributors that is far easier to keep up to data as it complains loudly when it fails and they detect breaking that would otherwise go completely unnoticed and end up shipping. This is invaluable for large projects and software that is shipped to be installed like libraries and installable binaries. If you are writing a service that you deploy in your own environment to serve some API requests then sure, too much tests will have diminishing returns.
- alexandercrohde 6y agohttps://blog.alexrohde.com/archives/178 https://blog.alexrohde.com/archives/178
- chrischattin 6y agoYes! Thank you! I've been saying this for years. The whole point of testing is make sure you aren't breaking something when you add a feature/refactor/delete old code. Its purpose is to speed up development. Excessive unit testing just creates a brittle test suite, and adds more work without much benefit. It slows you down. As a Rails dude focused on startups, iterating rapidly and what the user sees is what you care about. Therefore, I focus on integration tests that run the whole stack. That lets me mess with the implementation code without re-writing the test suite. At the same time, it provides regression protection and a good place to start troubleshooting. Plus, Rails already has tests for the "plumbing". Tests should serve the developer, and speed up the iterative process, not add work to the project b/c of a dogmatic adherence to TDD or unit test all the things design. Just my (unpopular) opinion.
- alexandercrohde 6y agoOne thing I've learned, is that unit testing seems much much more painful than it has to be when you are using too many classes. When you are writing code as pure functions (i.e. stateless), it's actually much less painful. In the provided example, I would never write a class to curl a website and parse json.
- smitty1e 6y agoWord of the day: autotelic "(of an activity or a creative work) having an end or purpose in itself" A slightly improved word choice over "autoerotic" when the design choices are getting too self-indulgent.
- KingOfCoders 6y agoUnit tests are a tool to solve a problem. If you don't have the problem, don't use the tool. If you can do major refactorings without impact on productiviy, don't have a QA department doing manual tests, junior developers can deploy to production with confidence on their first day and customers are happy about the quality of your product, don't solve a problem that isn't there.
- throw_this_one 6y agoI think Unit Testing is good for when you have big teams and changes that can go across knowledge boundaries... so the tests can keep people in check from making changes that mess up other parts of the codebase inadvertently.
- travisgriggs 6y agoGiggle. How things evolve. "Unit Testing" came out of the Smalltalk work around 1997. I was there. It was Kent Becks brain child. The first NUnit framework was SUnit. What was a unit? Kent was never absolute about this, I always felt because as a consultant he wanted to peddle the theory far and wide. But the early examples all had a pretty strong trend towards "a unit is an object." Nowaday "what is an object" is pretty loose; I just finished some Dart tutorials where an object is a way of "organizing our code into smaller reusable pieces" which ironically, I did in Fortran77 with common blocks and well factored files. But in the Smalltalk world, which was "objects all the way down", an object was small amounts of imperative behavior bound to data, where computational results were achieved via an approximation of the way cellular biology solves problems: lots of little glumps of data that achieve a larger result by sending messages to each other. This process of turning behavior and algorithms into things, or reification, was sometimes easy and sometimes hard. An object for Point, obvious. An object for SortCollationPolicy, less so. What Unit Tests did was help programmers design good objects. Beck said this in eXtreme Programming eXplained. He said that traditional QA departments would laugh themselves silly at what unit tests did. But that the value was that it drove good design. And that in a collaborative (pair programming) environment, it helped communicate the design intents around objects to fellow developers. I did the Smalltalk koolaid fest for 20 years. I found Unit Testing to be immensely effective. It made my designs more cellular again and again. When my designs were solid, I had less bugs. As a mechanical engineer, I still see similarities between unit tests and geometric dimensioning and tolerancing, a practice in the mechanical world that also swam against the current of conventional testing practices and left some shaking their head. In todays world where OO design is more of a "small unit of organization" I'm not surprised that unit testing also seems meh.
- agentultra 6y agoI think what a lot of people tend to miss with this discussion is that TDD is Test-Driven development. It doesn't have to be unit tests. The point is that you think first about what a valid specification is and you write a test for it. There's an anecdote from Djikstra that I'll paraphrase: Djikstra was working on a problem where two programs running on a shared memory computer were not allowed to enter their critical sections at the same time. He tasked his graduate students with finding the algorithm that would guarantee this. A student would provide a program. Djkistra would review it and find that it contained errors. The student would take the feedback and produce a longer, more complicated program. Tired and unable to find more time to review the increasingly complicated programs, Djikstra tasked his students to submit with their program a proof of its correctness. Djikstra then need only verify that he understood the proof and that the program implemented it faithfully. Shifting the burden of proof from the reviewer to the author made Djikstra's work much easier and the programs more robust. I'm not suggesting we need to start writing proofs. We're practical industry programmers who aren't working with such high-assurance software most of the time. However unit tests, weak as they are, are at least a form of proof. Proof by example. For trivial code where a few examples would suffice to convince you of its correctness I would say they are quite useful. Overrated though? I don't think so. They're also useful as a design tool. The OP gives an example of testing business logic that makes a bunch of HTTP API calls. The author claims unit tests are of little value here. Well how would you test that? Me, I would defunctionalize the calls the execute the HTTP requests. I'd provide an interpreter that the user can run their program in. The production code could use an algebra that makes the HTTP requests. The test code could use an algebra that stores requests made and returns canned responses. It might seem like "extra code," but it gives us the ability to decouple our logic from how its executed. Not only can we test this without having to mock our HTTP library (which is, itself quite well tested) but this decoupling opens new avenues for managing our program. We can imagine affixing to our algebras some logging actions. We could write a development version of our interpreter algebra that logs out everything and a production version which masks secrets and sensitive information from the logs. I've met programmers who can "just write the code," and they manage well for themselves. However I've also worked with such programmers who don't. The former are quite rare. And working with either group is difficult to say the least. If I am reviewing a piece of code I need to check whether your thinking is sound: did you consider the essential properties, edge cases, and did you spend time proving that you've thought about them? I don't want to read 600 lines of code and try to understand it... it's too much. But a proof, even an incomplete hand-waving one, I can understand. That being said... sorry for the wall of text. Unit tests aren't the be-all-and-end-all of testing. They're a beginning. And they can be a useful tool when you're starting out. Look to property based tests. Think about the tests before you write the code. And run the tests frequently and often. And refactor, refactor, refactor.
- ulisesrmzroche 6y agoI like the writing but he’s making shitty unit tests to make a point. The criticism needs to be applied to well made unit tests. I do agree with the general idea that integration and e2e tests are more important
- palotasb 6y agoWell thought-out article with good points and deductions! I have a few counterpoints and I think the example is not the best way to argue against unit testing as there are better ways to implement the same features and write better unit tests without bloating the code. Looking at the SolarCalculator class, I would go another way first and refactor it like so: public class SolarCalculator { public static SolarTimes GetSolarTimes(Location location, DateTimeOffset date) { /* ... */ } } 1. Made the method static (make it a free function in other languages) 2. Take an explicit Location parameter 3. Return a SolarTimes object directly, not async, not a Task<solarTimes>, and remove Async from the name 4. Drop the now unnecessary class LocationProvider member This becomes more easily unit testable without any excess Arranges steps at the beginning. public class SolarCalculatorTests { [Fact] public Task GetSolarTimes_ForKyiv_ReturnsCorrectSolarTimes() { // Arrange var location = new Location(50.45, 30.52); var date = new DateTimeOffset(2019, 11, 04, 00, 00, 00, TimeSpan.FromHours(+2)); var expectedSolarTimes = new SolarTimes( new TimeSpan(06, 55, 00), new TimeSpan(16, 29, 00) ); // Act var solarTimes = solarCalculator.GetSolarTimes(location, date); // Assert solarTimes.Should().BeEquivalentTo(expectedSolarTimes); } } The GetSolarTimes function is now purely computational and has no plumbing at all (stealing terms from chimprich). I think the original author would also agree that unit testing SolarTimes GetSolarTimes(Location location, DateTimeOffset date) has none of the problems that unit testing async Task<SolarTimes> GetSolarTimesAsync(DateTimeOffset date) had. (Added benefit: the interface is more flexible, it can be reused in more use cases without modification but that is not the point.) I find that such a refactoring often solves all of the problem entirely. It might seem like we just swapped the problem under the rug and forced the calling code to add the complexity (and the tests, interfaces, mocks etc.) that we discarded, but in practice this is often not the case. // Uses original async implicit-location interface // (assumes an existing solarCalculator instance) var solarTimes = await solarCalculator.GetSolarTimesAsync(date); // Uses proposed non-async explicit-location interface // (assumes an existing locationProvider instance) var solarTimes = SolarCalculator.GetSolarTimes(await locationProvider.GetLocationAsync(), date); The reason we can often get away with this in practice is that the complexity increase in the caller is small. We did not add additional state to the caller, we did not push more testing/mocking complexity to the caller. The assumed locationProvider instance in the caller replaces the solarCalculator instance in the caller. If testing/mocking locationProvider is required, testing/mocking solarCalculator ought to have been tested too. We require the caller to test/mock something else, not something new. If the original async Task<SolarTimes> GetSolarTimesAsync(DateTimeOffset date) interface is required nonetheless, it can be implemented as a pure "plumbing" function. As such I would agree that unit testing it would provide less value than integration testing. A simple pattern that can be applied here instead of an ILocationProvider interface and all the baggage that comes with it is using a Func<Task<Location>> or lambda instead. This allows both testing the instance with custom location providers and unhindered usage of the SolarCalculator class without always needing to inject a dependency. public class SolarCalculator { private readonly Func<Task<Location>> _locationProvider; // default constructor for normal usage public SolarCalculator() { internalReaLLocationProvider = LocationProvider(); _locationProvider = async () => internalReaLLocationProvider.GetLocationAsync(); } // constructor for custom locations and testing public SolarCalculator(Func<Task<Location>> locationProvider) { _locationProvider = locationProvider; } // Gets solar times for current location and specified date public async Task<SolarTimes> GetSolarTimesAsync(DateTimeOffset date) { return GetSolarTimes(_locationProvider(), date); } public static SolarTimes GetSolarTimes(Location location, DateTimeOffset date) { /* ... */ } } (Sorry, I haven't implemented the IDisposable pattern for internalReaLLocationProvider and I might have misplaced an async keyword or two because C# is not my most recent language.) To support the "pyramid-driven" paradigm I argue that the most complicated part of this feature is the solar time calculation and it would well deserve a large test suite containing many test cases like GetSolarTimes_ForKyiv_ReturnsCorrectSolarTimes above (edge cases, diverse locations, etc.). Conversely, the higher-level functions don't need this level of testing. Since they contain no complex logic, I usually assume that if they work for one input, they will work for any other. Testing the async, automatic-location version with the simple Kyiv-based input is enough, there is no need to test it with midnight sun and all the same edge cases as the base function. The point I'm making is that unit testing is not as overrated as the original example suggests. The code can be modularized better (not by making everything an interface), with a well-unit-testable "computational" part and a part which is mostly "plumbing". I agree with those saying that the second part benefits more from integration testing than unit testing, and I can agree with keeping them "as highly integrated as possible, while keeping their speed and complexity reasonable". But I insist that unit testing the 1st part and writing it in a way that it is unit testable is important.
- zoomablemind 6y ago> The primary goal is not to write tests which are as isolated as possible, but rather to gain confidence that the code works according to its functional requirements. Very long article that leads to a trivial conclusion that unit testing is not a substitute to higher level testing, and carries the overhead effort. It seems to me the author is addressing cases where unit test coverage is blindly used as dev project metric. PMs are often not familiar with code internals, but they need some assurances that project is on-track. Thus trying to collect insights from whatever output provided by automated tools. Unit tests can have 100% coverage, but still testing the wrong thing. A common misuse is testing to the actual code, instead of testing the expected behavior. Unit tests is a developer's tool, not PMs metric. On the other hand Acceptance tests should be the common ground, which may be the Integration tests or a whole separate suite altogether tailored to user requirements. When coding a function, one needs to test preconditions and assumptions prior to following the main logic. Unit tests serve the very same purpose by enforcing assumptions on Unit level at the granularity that makes sense. Noone needs to test trivial getters and setters, but one needs to ensure that objects remain in valid/known states. Thus Unit tests untie developer's hands to recraft the "unit" without fear that the pearls of correctness would get lost. Paradoxycally, Unit testing is a productivity tool, just as diagraming, or ... sketching prior to coding. Some developers can maintain a perfect mental picture of their code without any need for such tools. Some devs see design and implementations right away. If I were to inherit their codebase, I'd rather see their assumptions validated, so I don't need to do the coverage by eyeballing the code.
- kazinator 6y agoThorough unit tests are required for code that is open to unforeseeable new uses. The existing application may not provide enough coverage. Unit tests can exercise all the requirements of a module that are not yet being used now so that in the future, someone can do new things with that module without having to fix it first (thereby risking the existing application). The argument that it's more productive to test modules through the application only applies to private, dedicated modules that will only ever be invoked in support of the use cases arising in the application. Basically, it boils down to whether or not the piece of code is an independent product (where "product" could refer to something with "customers" internal to the organization). In engineering, all building blocks that are separate products to be integrated into other products are rigorously tested on their own, whether they are integrated circuits, or steel cables or whatever.
- vannevar 6y agoIt's worth noting that what the author describes as functional testing was, in fact, what unit testing was originally intended to be. "Unit" referred to a unit of functionality, not a unit of code. But that was an ambiguous concept, and it was simpler and easier to just write tests for each unit of code (typically a method), and so that's the approach that has prevailed.
- bpyne 6y agoExcellent, thought-provoking piece that comes at a good time for me. I've been looking at Python's unittest for a specific system I've been developing over the course of 12 years at work. It became clear recently that we need more automated tests. However, as I started to design a test suite in unittest, the scope of it became overwhelming. Looking at it as a suite of functional tests makes it more reasonable to write.
- spion 6y agoLike all things taught and learned by cargo-culting, unit tests have long lost the original intent. No, a unit is not a function, or a method, or a class, or a file. A unit is a clearly separated, non-trivial software component with a minimal and stable interface to the rest of the software system. A unit is a good if it requires little to no mocking to test and a very small number of messages to test. It should also do something non-trivial - think small library size, not class size. The unit should probably have a README which explains the small interface and the small number of necessary dependencies. The unit tests should largely treat the unit as a black box and be based on its promises in the README.
- monksy 6y ago> focusing on unit tests is, in most cases, a complete waste of time. Someone is pretty inexperienced by making that claim. Unit tests help to isolate the expectations and verification to very small levels. > While these changes may seem as an improvement to some, it’s important to point out that the interfaces we’ve defined serve no practical purpose other than making unit testing possible. No it simplified the responsibility of the class. It also simplified your tests as well. Now you don't have to have tests that test many different scenarios. > Note that although ILocationProvider exposes two different methods, from the contract perspective we have no way of knowing which one actually gets called. Tests don't verify how you use or call other classes/methods. You can do verification if you want via mocks. > Unit tests have a limited purpose Yes, as does integration tests, functional tests, system tests. Etc. You shouldn't be trying to do unit level testing via an integration test. > For example, does it make sense to unit test a method that calculates solar times using a long and complicated mathematical algorithm? Most likely, yes. It does if you want to verify that the functionality is setting up the request correctly. > Does it make sense to unit test a method that sends a request to a REST API to get geographical coordinates? Most likely, not. That's not a unit test. That's an integration test(if you use a mock) or a functional test if you want to end a live end point. > Unit tests lead to more complicated design It highlights that the original design was complicated or the methods had side effects (which you should avoid). In his own example he seperated the resources from the functionality. > Unit tests are expensive No they're not. Mocks do not belong in a unit test. That's for integration tests. If you have them there you have issues. They're cheap to write and quick to run. > Unit tests rely on implementation details It's all about how you write you code, if you're trying to lump everything together, that's what you get. > Unit tests don’t exercise user behavior Correct, unit tests don't. Functional/feature or above do. I stopped reading after this. It feels like the guy is just trying to argue that he doesn't like testing.
- mdoms 6y agoI weep for the state of our industry.
- wanderr 6y agoI've worked at really small startups (<20 engineers) and pretty large companies (>1k engineers). At small companies we have absolutely been able to get away with 0 unit tests while maintaining agility - being able to do major reactors quickly even when working in dynamic languages, while maintaining a high level of quality, even when operating at reasonably large scale. The key is clear, well written code and strong ownership from senior engineers who have a deep understanding of the code they own. On the other hand, at large companies, extensive unit testing has been invaluable. Code bases are older, ownership changes hands frequently, new engineers join all the time, old ones move on to other projects, dozens of teams are calling each others code, refactoring is done by people who had no hand in writing the code in the first place, dependencies are higher and harder to track down completely, and it's not realistic to cover all important functionality with integration tests. Engineers often must rely on unit tests to prevent others from breaking their code, and to ensure that they are not breaking someone else's. Yes, unit tests can be highly problematic and costly to maintain, and they do add friction and time to initial development, but in these scenarios the benefits outweigh the costs considerably.
- woah 6y agoThe article isn't about this, but in my experience, a lot of unit testing is driven by loosely typed languages, and tests for things that would be a compile time error in something stricter. Rust in particular has a way of making many unit tests redundant.
- mnm1 6y agoThis was obvious to me as soon as I wrote my first unit tests. I was all about uncle Bob's philosophy till I realized it doesn't work and discourages creativity and problem solving. Now I remove most old unit tests as they are a liability. They break while the code they supposedly test works perfectly. Maintaining them is a waste of time. I do have some unit tests for specific algorithms written in functional languages. They can be helpful but are no substitute for proper testing that actually catches bugs, whatever way that's achieved. In software, people love fashion. Unit tests were and are to some extent fashion. Like microservices and other ideas that only apply to the top 0.1% of companies, unit testing is generally useless in apps. There is some use when used with specific algorithms where the inputs and outputs are always constant and known. Libraries, programming languages, etc. But for apps, they are mainly a liability.
- je42 6y agoHow about this, instead labelling something as overrated, start with a more positive and differentiating point of view: - when to use unit-tests ? - when are other tests more useful ? - which bugs can unit-tests find ? - in what situations is changing code to make it testable some-times beneficial ? - when is it a complete waste of time to make code unit-testable ? Things to consider when answering these questions above: - what is the customer impact if there is bug ? - what is the developer impact if there is bug ? - what is the developer impact writing tests ? - how quickly can we fix it and release a new version ? - how quickly can we fix the hotfix in case we messed up ? - how can we find out if there is a bug ? [...] I think it is great that he starts thinking about the value of unit-tests and testing in general in his particular context. But i think it helps keeping an open mind knowing that some approaches work better than others depending on the context.
- kelvin0 6y agoTesting should be an exercise in risk assessment. If module A is very expensive to fix and has a high probability of failing in production, well of course you want to have it rock solid and should be thoroughly tested. You could build a priority index, with something like : priority for testing module A = cost of fixing A x probability A fails The other problem is getting management onboard and demonstrating a ROI justifying the time spent building these tests and using them. I personally failed at that and still am trying to figure out how to get them to understand the benefits. I've progressed, but it's one heck of an uphill battle for me.
- sub7 6y agoFunctional testing is underrated.
- robertlagrant 6y agoCode that's written a certain way just to make it unit testable is less common in something like Python, where mocking is super powerful.
- DominikD 6y agoI used to think that unit tests are a waste of time. But they find tons of bugs in my code so I simply can't ignore the reality by posting some random untestable code and claim that it proves that tests are (most of the time - let's cover our bases folks!) a waste of time.
- boothby 6y agoA plug for an old-is-new-again feature I hacked together in an evening... https://github.com/boothby/dissert https://github.com/boothby/dissert I like assertions. They're a really good alternative to unit tests because they can be used in a real environment without the need for maintenance-intensive mocking etc. But assertions have a significant performance cost, especially when they involve consistency checks on large datastructures. So, a pattern that I've found useful is heavyweight asserts that can be enabled / disabled through external means, in conjunction with high-coverage integration testing. This is really easy in some languages (c/c++, for example, using the preprocessor) and can be more fragile in languages like python (hence the plug above -- which isn't the best way, but is a way, to achieve this 'best of both worlds' testing).
- kccqzy 6y ago> Unit tests, as evident by the name, revolve around the concept of a “unit”, which denotes a very small isolated part of a larger system. There is no formal definition of what a unit is or how small it should be, but it’s mostly accepted that it corresponds to an individual function of a module (or method of an object). Heh I feel like this is the crux of the issue. Because there's no standardized definition of what a unit is, people sometimes tend to choose the wrong unit to test. I personally believe it is usually wrong to test a single function or method in a class. I tend to test the behavior of a whole class at the same time. Testing each individual method is too white-box, and makes your testing code too coupled to the implementation. Basically, don't test internal details (unless the internals are very complicated), just test the externally visible behavior, which is usually presented as a whole class. I also don't agree with always mocking out dependencies. If your "dependency" is just an instance of a different class, then just use the real deal. Sure, now your definition of a unit now encompasses not just your code, but your dependency's code, but if your dependency's code affects the externally visible behavior of your code, it remains your responsibility if your dependency changes things and breaks your code. That's what abstraction means: you present an interface, your client doesn't need to care about how it's implemented, and what dependency your code needs.
- nojvek 6y agoI only believe that theres only one kind of testing. Black box testing. And that is broken into two types. Integration and unit. Tests that validate the interface of the big black box and assert that it does the right thing (integration tests). I.e assert the behavior customers depend on. That black box is made up of many smaller black boxes that talk to each other. So you test the boundaries of each of those smaller black boxes. (Unit tests) The ROI for testing the big black box is much higher. But when something fails it’s hard to know what exactly caused it and how to fix it. If you have a decent number of tests for the smaller black boxes then you know what box needs to be fixed. But all good tests are black box tests. I.e they don’t test the internals but the interface of using that box. A well designed system is made up of small boxes that do one thing very well and work with other boxes. Once you’ve tested them you can forget about them and work on higher abstractions. They will continue doing what they promised.
- bob1029 6y agoI only use unit tests as a temporary critic when developing brand new abstractions from scratch. With complex implementations, it is incredibly easy to forget one out of hundreds of constraints and lose track of important capabilities as you go. That said, the moment the abstraction is in a clean state and consistently passing all tests and has been integrated well with existing logic, the unit tests are deprecated as far as I am concerned. I wont ever explicitly delete them, but I recognize that the unit tests are potentially just as flawed as what they are testing (the same developer wrote them after all), and coming back into that abstraction after 6+ months elapses means I'd probably just have to rewrite tests from scratch as a mental exercise and in order to restore my own sanity. I can recall at least one occasion where I wasted 2 days chasing down a failing unit test only to find out the test itself was flawed - in the worst way, random pass/fail based on a race condition between multiple threads that were part of the testing code. I think that's the biggest danger with unit testing. Other developers assuming the tests you wrote are foolproof and sending them on pointless errands.
- jb3689 6y agoSeeing Java examples after this does not surprise me. Testing is hard in Java and it forces you to change your design. Maybe better put - testing stateful objects is hard I do a lot of dev in Ruby and testing there is super easy and powerful. Say what you will about monkey patching, open classes, and reflection, but it does make it very easy to write great testing libraries. I'd argue that testing libraries in Ruby lead to better design (e.g. using more methods, making things single purpose, thinking about the interface first can lead to better naming etc) That said, I don't test as much as I used to. Unit testing is really good for testing algorithms (e.g. transform this complex JSON; rebalance load). Outside of that, some light end-to-end testing will catch most other things, and you can use staging and gradual rollouts to derisk bigger changes
- jasonlhy 6y agoI think the value of unit test is to test the logic in software abstraction instead of relying on external dependencies, but I believe many unit test is just for the sake of unit test
- danans 6y ago> Does it make sense to unit test a method that sends a request to a REST API to get geographical coordinates? Most likely, not. Isn't a "unit test" which sends a request to a REST API almost by definition an "integration test", or even a "live server smoke/staging test", since it is testing the integration between separate live systems (the requesting code and the server). If anything, what should the unit tested if anything is the code that generates the request, which should be separate from the code that sends the request to the server. But only if that code is non-trivial.
- burlzad 6y agoMost of the problems defined in the article tend to go away with an interpreted language such as JS. In JS unit testing is super cheap and super easy, with very common patterns.