17 ms·

Unit Testing is Overrated (2020)

• overstood 5y ago
  This is more of a criticism of mock heavy testing than anything else. The author should read Beck’s original TDD book and experiment with testing without mocks.
• toolslive 5y ago
  To me, the value of unit tests really depends on the power of the type system. If you're in working with Haskell, Ocaml, ... , unit test have almost no value. However, if you're using python, php, javascript,... unit tests will help you to avoid most silly mistakes.
  • rectang 5y ago
    You're only talking about the subset of tests which the type system can supplant. Logic bugs are outside of the type system's domain, and the generalization "unit tests have almost no value" does not apply to unit tests which validate logic.
• bluehatbrit 5y ago
  Previous HN discussion from 2020: https://news.ycombinator.com/item?id=23778878 https://news.ycombinator.com/item?id=23778878
  • dang 5y ago
    Thanks! Macroexpanded: Unit Testing Is Overrated - https://news.ycombinator.com/item?id=23778878 https://news.ycombinator.com/item?id=23778878 - July 2020 (387 comments)
• injidup 5y ago
  The first example is such a straw man. The LocationProvider class has no business being a class. It should just be a function. Problems then go away.
  • ensiferum 5y ago
    And the sunset calculator doesn't need a location provider. All it needs are location coordinates which can be given to the calculate function. In fact the whole thing then becomes a standalone function which takes Location and other relevant parameters and returns the computer sunrise/sunset values. Pure function and super easy to unit test. If one needs to do a lot of "mockups" for your unit tests then maybe one needs to consider the API and class design. Removing needless coupling helps testability by removing the need to use mocks in the first place.
    • manbash 5y ago
      In design terms, I would claim that Sunset Calculator is a "component" (an integration of units), with a "required interface" (i.e. abstracted) that provides the coordinates. A role of one unit inside this component is to use the provided coordinates and only perform the calculation.
• dhzhzjsbevs 5y ago
  Hot take. Unit testing is not overrated, if you feel this way you likely just suck at optimising your suite for high RoI tests or chase some stupid metric like code coverage getting mad when you find out you wasted a bunch of time writing worthless tests. https://youtu.be/z9quxZsLcfo https://youtu.be/z9quxZsLcfo
  • adhesive_wombat 5y ago
    There's nothing that cements the value of units tests more, for me, than surfacing bugs in new code almost immediately that would otherwise need debugging "in situ" in the application. Figuring out where your bug is is so much harder when you have to do it though the lens of other code, whereas in your unit test, you can just see "oh, I'm returning foo.X, not foo.Y". It also makes sure you actually can construct your object at all without dragging in a dependency on the entire system. Code without tests tends to accrete things that can only be set up in a very long and complex process. This is both hard to reason about (because your system can only be thought about as an ensemble) and fragile when the system changes: you now have to unpick the dependencies to allow your "SendEmail" function to not somehow depend on an interface to some SPI hardware! But there's certainly value in not spending hours testing obvious code to death: a getter is almost certainly fine, and even if it isn't, the first "real" test that uses it will fail if you're returning the wrong thing. But if, down the line, you do find a bug in it, then something was probably not that obvious!
    • jseban 5y ago
      Nothing cements the uselessness of unit tests more, for me, than not surfacing bugs in new code and having wasted all that time preemptively debugging code that works.
      • adhesive_wombat 5y ago
        Then don't write tests for that code. Writing tests for trivial stuff is pointless. Write tests for code that could actually be wrong, or where you need to prove to yourself or others that the invariant is maintained. Critically, where it's important that the invariant stays true even after refactoring. It's also valuable to have some testing framework available, just so that it's then easy to write tests when they're needed (which comes back around to "make sure that your objects can ever be constructed" thing). Not all unit tests have to be preemptive, even if the presence of the ability to quickly write them is.
• furstenheim 5y ago
  I really like Spotify strategy https://engineering.atspotify.com/2018/01/testing-of-microservices/ https://engineering.atspotify.com/2018/01/testing-of-microse...
  • randomtask 5y ago
    So, they suggest that you spin up the dependent services as part of the test runner and run tests for your service against those? I've worked on a codebase that did this extensively. Over time it lead to a test suite that takes a long time to run, slowing down the release process considerably, and inevitably requiring further engineering effort to make the tests run in a reasonable amount of time. I can't say I would recommend it as an approach based on my experience with it, but YMMV.
    • rmetzler 5y ago
      You don’t use persistent databases to run these tests but put them on a tmpfs or in memory, right?
      • dustymcp 5y ago
        Yes correct then you can do Them in parallel
      • azeirah 5y ago
        Can you (or someone else) tell me more about how to run database tests efficiently? I'm working on an app that has 200+ tables and requires a lot of data to be in the database before anything even works at all. Of course good tests interacting with the database need a fresh database every time so even a test suite of about 10 tests can take 10-20 minutes already.
    • pydry 5y ago
      >Over time it lead to a test suite that takes a long time to run, slowing down the release process considerably Slower, but the test suite is far more effective at catching bugs. These tests do take more up front engineering but since they arent pale imitations of the code they also dont need to be totally rewritten every time the code is refactored. Higher capex, lower opex.
• JoeNr76 5y ago
  I stopped reading right here: "Considering the factors mentioned above, we can reason that unit tests are only useful to verify pure business logic inside of a given function." That just isn't true and it makes the rest of the blogpost also not true. A unit test should test "a unit of functionality" not just a method or a class. Your unit tests also shouldn't be coupled to the implementation of your unit of functionality. If you are making classes or methods public because you want to unit test them, you're doing it wrong. The exception is maybe those tests you are writing while you're doing the coding. But you don't have to keep them around as they are.
  • Chris2048 5y ago
    > stopped reading > it makes the rest of the blogpost also not true what do you know of the rest of the blogpost if you stopped reading?
    • JoeNr76 5y ago
      Wild speculation with a sprinkle of logic (garbage assumptions lead to garbage conclusions) HOWEVER, I have skimmed through it by now and the last paragraph is actually quite good. It takes a while to get there and the chosen examples aren't great, though.
  • Ygg2 5y ago
    > "Considering the factors mentioned above, we can reason that unit tests are only useful to verify pure business logic inside of a given function." > That just isn't true and it makes the rest of the blogpost also not true. It's not? Writing unit test is all about minimizing side effects and interaction with other systems.
    • fivea 5y ago
      > It's not? Writing unit test is all about minimizing side effects and interaction with other systems. Not really. Writing unit tests is all about verifying behavior from combinations of inputs, and side effects are inputs as well. You can write unit tests that inject delays and timeouts and retries and throw exceptions under specific circumstances.
• glenjamin 5y ago
  Some time ago I stopped debating the definition of "unit" in testing, and instead started focusing on whether my tests were fast, reliable, and provided a useful signal about the health of the system when they failed. I've been much happier since then.
  • 2OEH8eoCRo0 5y ago
    This is what Working Effectively with Legacy Code says: Here are qualities of good unit tests: 1. They run fast. 2. They help us localize problems. A unit test that takes 1/10th of a second to run is a slow unit test.
    • AstralStorm 5y ago
      Nice joke, it takes longer to upload firmware with the unit test suite than that. The test that is too short is most likely not being thorough enough, or the functionality it tests is so simplistic the test likely brings no value.
      • 2OEH8eoCRo0 5y ago
        What about integration or end to end tests?
    • glenjamin 5y ago
      That's surprising to me, because the author of that book recently re-advertised his long post about a hard-line definition of a "unit" test https://www.artima.com/weblogs/viewpost.jsp?thread=126923 https://www.artima.com/weblogs/viewpost.jsp?thread=126923 I think the properties you list are good ones, but if you drop the word unit from your post the advice is just as good and you can't get mired in ontological discussions
  • stepbeek 5y ago
    This is the sticking point I've found. The definition of "unit" is different depending on who I talk to.
• _glass 5y ago
  Unit Tests are still underrated in my experience. It pains me when I see developers testing their functionality end to end, manually. Discover a formatting error in their input, then do this again. When I was working in the office I could see that easily 90% of a lot of developer's time is spent like this. Then I showed that a unit test can discover this, people are still in denial, that it was just for this small error. But software development is a lot about nitty gritty details. And unit testing fixes this.
  • capableweb 5y ago
    Many developers seem to lack a focus on automation in the most basic sense of the word as well. While consulting, I see lots of backend developers constantly making a change, waiting for the server to reload with the new changes, switch to Postman/cURL/whatever, fire off their request and see if it's accurate. Rather than just having the request handler as a function that is under test with assertions, and have the test re-run on each change. So much time during a developers day is spent on just repetition, and they don't seem to care.
    • brtkdotse 5y ago
      Plenty of time to check Reddit while waiting for the server to reload.
  • guzik 5y ago
    Once I wrote my third game in Unity, with unit tests coverage of about 80%. It was least buggy game in my entire career. It was also the hardest time as I had to start thinking more about creating testable components rather than messy spaghetti code I used to write before.
    • bobertlo 5y ago
      I’ve found that while I did spend a lot of time at first figuring out how to write testable code, I now am so much better at writing in this way that I am much more productive since everything has great test coverage. Saves me going back to modules I wrote 2 years ago to add features.
• Starlevel001 5y ago
  A lot of the software I use is extensively unit tested. This software is still buggy and broken. Ergo, unit testing is not working.
  • synergy20 5y ago
    one example: sqlite3 has more testing code than its own code, so it can swap the whole design at will once a while, it is working there. unit testing is essential to me.
    • warmwaffles 5y ago
      fuzz testing be damned.
• jcranberry 5y ago
  >Tests, whose scope lies higher on Mike Cohn’s pyramid are either written as part of a wider suite (often by completely different people) or even disregarded entirely. My experience is the exact opposite. Actually my experiences are the opposite of many things in this article.
• keewee7 5y ago
  Unit tests are useful because it improves software design by encouraging you to develop modular software units that can be tested independently. However you need integration tests and end-to-end tests to improve software correctness.
• hatch_q 5y ago
  Maybe, in type-safe language like C#. But in a dynamic language, we rely on unit tests to catch obvious typos and regressions.
  • codr7 5y ago
    I'm wondering if that's part of the problem, the fact that unit testing started out in Smalltalk. It does make a lot more sense in a dynamic language, because any line that's not touched is a potential typo.
• aaaarc 5y ago
  Tests are an extension of documentation, in my opinion. I go on Github and I read the readme then I go to the test folder and try to see some valid usage examples. This is what writing useful tests mean, from a client perspective I just want to understand what's valid and what's invalid. Also useful to make sure we fixed specific scenario bugs. I've seen a lot of people doing things wrong but we can summarise in two cases: - "We've got 100% coverage, it must mean we've got no bugs, right?" - We've got tests covering happy path, everything else is tested manually. It takes forever and it's not shared anywhere, but "trust me" I've tested this.
• chrisdbanks 5y ago
  I don't like: - manual testing - fixing bugs found in production under extreme time pressure - people finding embarrassing bugs in my code - spending lots of time explaining what code should be doing to other developers Unit testing massively reduces all of these. Therefore I assume people that don't like unit tests either enjoy doing these things or are masochistic.
• viach 5y ago
  > Consider mocking only as a last resort Consider trying to write tests in a non-trivial Spring Boot app without mocks
• WolfOliver 5y ago
  A similar discussion from 17 days ago: "The Big TDD Misunderstanding" -> https://news.ycombinator.com/item?id=30734241 https://news.ycombinator.com/item?id=30734241
• mav88 5y ago
  For a contrary view, see: https://quii.gitbook.io/learn-go-with-tests/meta/why https://quii.gitbook.io/learn-go-with-tests/meta/why The book is Go specific but there's a lot of wisdom in this particular chapter that applies to any language.
• theknocker 5y ago
• lucasyvas 5y ago
  As a predominantly fullstack web developer, I will only ever voluntarily write unit tests and end to end tests, but avoid in-code integration tests in most circumstances. This is frankly the easiest and best bang for buck. Unit tests force you to make your code more flexible, they're fast to write, fast to run. Maintenance depends on how overboard you go with your verifications - try to verify the most important parts. End to end tests require no elaborate scaffolding of internals and allow you to test it as a real user interacts with it. Generally fast to make, slow to run, and maintenance depends on how disciplined you are with stable identifiers/interaction points. As for in-code integration tests - I love the idea of them, but they're absolutely miserable 9 times out of 10 due to extremely convoluted processes to "bring up" parts of your application. If you use DI it shouldn't be as bad, but almost nobody does and it becomes a total clusterfuck not worth the maintenance burden. As an idealist, I want all 3. But most codebases frankly cannot support all 3, so the most value will come from whatever is easiest. I've found that's most often unit and e2e.
  • tmstieff 5y ago
    The article actually argues the opposite. Developers should move their focus to integration / "real-world" tests. The major summary bullet point being: "Aim for the highest level of integration while maintaining reasonable speed and cost" My experience mirrors the author's. In any "real" business application, the unit tests end up mocking so many dependencies that changes become a chore, in many cases causing colleagues to skip certain obvious refactors because the thought of updating 300 unit tests is out of the question. I've found much better success testing at the integration level. And to be clear this means writing a tests inside the same project that run against a database. They should run as part of your build, both locally and in CI. The holy grail is probably writing all your business logic inside pure functions, and then unit testing those, while integration testing the outer layers for happy and error paths. But good luck trying to get your coworkers to think in pure functions.
    • rectang 5y ago
      > in many cases causing colleagues to skip certain obvious refactors because the thought of updating 300 unit tests is out of the question. Good! They shouldn't do the refactor. Because "obvious" refactors often introduce bugs (e.g. copy/paste errors), and if developers can't be bothered to write tests to catch them, they're going to screw over the other team members and users who will be forced to deal with their bugs in production. > The holy grail is probably writing all your business logic inside pure functions, and then unit testing those, while integration testing the outer layers for happy and error paths. So settle for half a loaf. Write all the easy unit tests first. The coverage will be very incomplete, but something is better than nothing. Write all the easy integration tests next. Never write the hard tests if you can help it.
• Veuxdo 5y ago
  Testing in general is overrated :)
  • rybosworld 5y ago
    This is how we get Windows Vista
• gautamdivgi 5y ago
  Unit tests are essential. I've developed some devops code around self-healing where the degrading signals were not reproducible. We knew they occurred because we lost a non-trivial number of nodes a day. A weird set of bugs between Linux, Kubernetes & Docker. The problem was we could recognize the signals and take action. The entire daemon I created to trap and execute on these was built and deployed on unit tests. In fact, I had more lines of unit test code than actual functional code because I had to mock the hell out of what the actual system would look like. Another situation - where my wife used to work in medical devices. Feature development velocity was a problem because there was a queue on the very limited set of devices (CT scanners, PET scanners, etc.) that the team could use for testing. Debugging was very hard. Fixing bugs was hard because to debug you needed a device. With unit testing and mocks they made a lot of contention on the devices go away. Write your unit tests. It will help you and the people coming in after you to maintain the code.
• synergy20 5y ago
  unit testing is essential, TDD might be a stretch in practice. instead of having separate test/ directory with multiple unit test files, I put unit test code right in the source code(at the end of the file), so when I modify the source I can easily update the unit test in the same place, and they never get out of sync. It worked for me very well. They can be turned on/off easily too. Sometimes it's easier to understand the code by reading the unit tests than reading the whole lengthy code, and it's helpful when they're in the same place. For me, unit test code is part of the real code, not something 'additional'.
• tough 5y ago
  I was building an API and started adding some tests, then added coverage and It got me to think about the complex-ness of the codebase, how many branches / statements I had per file to test etc, I think it helps greatly to reason about modular and good software. But it takes time too. YMMV
• deleted 5y ago
  [deleted]
• swalsh 5y ago
  Unit testing is a tool, you don't need a 100% test coverage. But sometimes it's the right tool for the job. Sometimes it's not. If I have a piece of code... usually an algorithm that handles a handful of different use cases, and making a change might unexpectedly cause an issue to another use case, i'll unit test the heck out of it. If I have some code where it has a simple input, and output but it takes some time to test it, i'll write a unit test to make development faster. But i'm not going to add an interface to every concrete class in my project, and design literally every component so I can mock it. Writing software is a business, i can sprinkle TDD around for a high ROI. If I use it EVERYWHERE the ROI is very little if not negative.
  • mring33621 5y ago
    Very much agree with "sprinkle TDD around for a high ROI" This requires both developer expertise and trust from stakeholders. But, VeryBigHugeBankCo requires me to have 70% unit-test coverage for 'new' code, as measured by Sonar or whatever, or else I am simply not able to deploy my change to PROD. So here I am refactoring a DAO, using strategy pattern and functional interfaces, so that I can cover my DB interactions adequately for that 70% metric. No, I can't just use H2 DB, as the SQL is Oracle-specific. Yes, there is very little actual value in the new refactor, except to check the box. This is how we do...
    • mring33621 5y ago
      Oh and don't get me started about how, especially when using mocking frameworks, the test code is sometimes wayyyy harder to understand than the actual code under test. NOT A WIN. I have told mgmt that test code is equally likely to have bugs as production code, but, again, this is just box-checking, not actual attention to detail.
      • thow-58d4e8b 5y ago
        It gets worse than that. Often, code is pushed into an absurdly wrong level of the development stack, just because it makes testing it easier. For example, something should be a database trigger, but in-memory testing database gets in the way. Other times it strikes all the way to architectural topics - message queues replaced by REST calls, because the former is difficult to mock
• awill88 5y ago
  Used to feel this way and largely agree when there are less contributors on the project or the tests are so trivial like things doing what they obviously do. However, unit tests can describe behavior and written succinctly can be beneficial for other people’s eyeballs. I try not to get into discussions about “clean” or “elegant” code for the most part, as most people prefer the smell of their own farts.
• Veuxdo 5y ago
  My problems with unit testing: - There's no consensus on what a "unit" is. - There's no consensus on how much coverage is needed. - There's no consensus on what should be mocked. - There's no consensus on if private methods should be tested independently, even though they obviously shouldn't. - It violates "don't repeat yourself". - The developers who write the tests usually write the code. Most bugs occur when developers overlook a use case. Do the math.
• rybosworld 5y ago
  The biggest issue in my experience is trying to target metrics. E.g. some % of coverage. If you encourage developers to write tests with the mindset that "these should make my life easier", you'll get better and more maintainable tests. If you encourage developers to write tests "just because" you will get very useless tests.
• novocantico 5y ago
  One argument against 100% test coverage that I've never heard articulated in my decade career as a professional software engineer is that we need to learn to just trust trustworthy software engineers. Yes, there will sometimes be mistakes because we're human. But I'm convinced 100% test coverage is a form of extremism. Once an engineer has proven their ability, and as long as they continue to prove their ability, then a test suite becomes a new kind of tool: not one to prove to management or stakeholders that the code works, but one to prove to the senior engineers that a certain spot of the code works as they intended it.
  • avl999 5y ago
    Unit tests are never a tool to prove stuff to managers/customers/stakeholders. They are a tool to prove to yourself and your teammates that your code works and have some reasonable amount of confidence that it will continue to work when your teammate (who might be fresh college grad hire with no experience) changes it. Testing has nothing to do with engineer ability or experience. Regardless of that you can always make a mistake. And even if you are a rockstar Sr dev who never makes a mistake, in the real world any piece of code is gonna get worked on by people of all skill levels and experience so the idea that "a senior engineer wrote it so we can just trust it" doesn't work even if one accepts the assumption that you can just trust the code from an experienced dev. I am against 100% coverage requirements as well (except for perhaps dynamically typed languages) but I would still expect a reasonable amount of coverage and that coverage include good tests.
    • novocantico 5y ago
      > Unit tests are never a tool to prove stuff to managers/customers/stakeholders. They are in certain agile circles. Ones I used to work in.
  • jseban 5y ago
    > we need to learn to just trust trustworthy software engineers Absolutely agree, and when we have identified trustworthy software engineers, we also need to give them authority. This doesn't work however, because it puts the middle manager on the spot to make this judgement call, and it also goes against the grand master plan of software engineering which is to turn it into outsourceable factory work.
• agentultra 5y ago
  > Unit tests lead to more complicated design You will here the exact opposite from proponents of test-driven design. The most complicated code I've seen is integration-test heavy code. It turns into a big ball of mud very quickly as new features added simply add more lines of code to already long procedures. Over a hundred lines of database calls, HTTP calls, variable bindings, conditions and branches. Code like this doesn't compose well and is a refactoring nightmare. Where writing testable code can lead to complicated designs is where developers are not thinking about abstractions and are, instead, introducing indirection. This is a common solution in many object-oriented languages with inheritance and virtual methods. There's a whole essay and reams of textbooks on what makes a good abstraction in a precise way but most developers think abstraction means indirection. However, if developers are concerned about writing abstractions it often reduces indirection and leads to easier-to-test code. This is because abstractions introduce precise semantic layers and code under test then shouldn't be concerning itself with underlying layers. A test-driven approach can lead to better designs but I think the claim that it will guarantee them is a bit over-sold. > Unit tests are expensive It appears the author means in the form of developer time. In terms of execution time if the tests are slow there's an underlying problem that should be addressed. In general is it more expensive to ship errors to production or to take the extra time to validate your assumptions, invariant properties, etc? > Unit tests rely on implementation details If you write them that way, they will. If you spend most of your effort mocking out dependencies in your code it means you're missing opportunities for abstraction. Users of your code must care how the underlying dependency is implemented and there fore there is no abstraction. If folks are forging ahead with mocking out dependencies like this across the code base you have bad abstractions and tight coupling in your code. You could step back from this and treat it like a bug and fix it... but this isn't a problem of the practice itself if people are ignoring the signs. The authors solution to their straw man is... more end-to-end, behaviour driven tests. While valuable these are demonstrably more time consuming activities: both in execution time and in developer time. I think a good peppering of unit tests is healthy; it aids in refactoring different layers of the application and gives decent indicators if your refactoring changes behaviours in a rapid-feedback loop. However I think unit tests are... still not great. There's not a lot of empirical evidence to support them in reducing error rates in code. And I think that tracks with their use in verifying systems: at best they're an example of a single input producing a single expected output... but the domain and range of most units is quite large and even a hard-core unit-testing team will not cover most of the problem space. In essence they only prove the existence of errors, not their absence. For that we need better tools like property-based tests and verification techniques like refinement typing, separation logic, etc as suits the problem space.
• adregan 5y ago
  Does anyone else use unit tests as a form of interactive development? For languages like typescript and kotlin, I’ve found them useful in situations where I don’t have a repl available. Custom react hooks are much easier (and faster) to develop in unit tests without having to load in a whole application. Exploring data models and their functions before implementing views is easier in unit tests. I find unit tests to be a really productive playground and I think it’s great to have a place to later jump in and extend functionality in a safe space.
  • ramses0 5y ago
    Tests ensure you always have two consumers for an API (the API "in-place" as used by the functional portion of your system, as well as "how the tests use it"). This ensures new uses of the API have a path/model to follow and prevents you from having to "instantiate the world" just to call your particular function. I also think of tests as "scaffolding" when building a skyscraper. They make it easier to access certain "things" before they're fully complete and are a cost that makes building the actually building less risky and quicker. Why does no-one question the value of paying for putting up and tearing down a scaffold when building or repairing a building, but can't see the value of unit testing?
• langager 5y ago
  I really enjoyed this article and spoke to a lot of the lived experience I've had in over abstracted codebases that had tests incredibly coupled to implementation. I think this title could maybe be rewritten as "Unit Testing that Requires Mocks is Overrated". Unit testing something that has no (or very simple) dependencies is great. For example: - some kind of encryption that takes in a string as a key - serialization that expects the same output as the input passed in - a transform function that takes in one type and expects a different type out As soon as you rely on a DB, file system, etc... you're probably better off with an e2e test. At the end of the day, it comes down to data contracts. This could be the functions a package exposes or the GQL/REST/gRPC/whatever API. That is the most important thing to not break the behavior of. Write good tests that target the external facing data contracts and treat implementation like a black box and you'll be empowered to do the important things that tests should enable like refactoring, reworking abstractions that may have made sense at one point, but no longer do, and let your codebase evolve as you learn. Tests that are a barrier to reworking internal abstractions are not good tests.
• wikidani 5y ago
  Oh god this resonates so much with what I'm going through right now. I'm on a team that rotates members pretty much every 6 months and have been put in charge of writing unit tests for a series of repositories that I have never seen in my life. I must also add that I had 0 experience with unit testing beyond knowing what the concept is when asked to start testing. Now, I wouldn't be too bothered because it was a sort of learning experience but what was supposed to be a two week task at most has been dragging for a couple of months because they keep adding APIs and other backends which further complicates the issue and, of course the code isn't really unit testable so I have to modify code that I haven't seen before made by people who aren't on the team anymore just to make tests work All of this to achieve a very arbitrary 80% coverage that's required by business on a few REST APIs that's not even ours!! And don't get me wrong I get the importance of testing but the enphasis on unit testing these days seems ridiculous
  • magicalhippo 5y ago
    > I'm on a team that rotates members pretty much every 6 months What kind of software do you make? Where I am, I was still very much learning after 6 months and it wasn't until a year or so before I became really effective.
    • wikidani 5y ago
      It's mostly line of business software and yeah, it took me a year before I got to know the ins and outs of the system so you can absolutely imagine the clusterfuck it is when user acceptance and deployments start coming in
  • rectang 5y ago
    > I'm on a team that rotates members pretty much every 6 months and have been put in charge of writing unit tests for a series of repositories that I have never seen in my life. I must also add that I had 0 experience with unit testing beyond knowing what the concept is when asked to start testing. I don't think the concept of "unit testing" is the main problem here.
    • 5y ago
• yoran 5y ago
  In my current work, which a GraphQL backend API, I've identified only two use cases for unit tests: - complex pure functions. For instance, I have a street address "cleaner". The input is an unclean street address (weird capitalisation and so forth) and ouputs a clean address. Unit tests are perfect here because they're fast, and it's an isolated piece of logic that has no side-effects (no writing to DB or network request). - wrappers around third-party libraries. Yes, in theory you're supposed to trust that a library will respect its API unless major version changes etc. In practice, it's not the case. And I'm also lazy and don't want to read all the release notes of all the libraries when doing updates. By wrapping a utility function from a third-party library and unit testing that function, I can have real confidence that the library still works in the way I'm expecting it to work. And it fails loudly when it no longer does. Outside of these, I only have end-to-end tests that test the whole stack: send a GraphQL query and check that the resulting JSON is what I expect.
  • ramses0 5y ago
    > - wrappers around third-party libraries. That's an "acceptance test" or "fit for purpose" test in case you're looking for industry terms. Closely related is "law of demeter" (the law of one dot), as it buffers you from changes in dependent API's.
• ChrisMarshallNY 5y ago
  I agree. Here's my screed on the topic: https://littlegreenviper.com/miscellany/testing-harness-vs-unit/ https://littlegreenviper.com/miscellany/testing-harness-vs-u...
• me_me_mu_mu 5y ago
  Team I'm on obsesses with 100% unit test coverage which means that it takes 3-5x as long to ship a feature while engineers write tests that are: 1) Almost always better to be part of integration test 2) Offer no real ROI because the framework usually handles most of these issues gracefully to begin with, so its just a few hundred lines of cute code trying to mock out shit (or better yet getting multiple people in a zoom and wasting their time trying to test 4 lines of code - WHICH THE FKIN LIBRARY/FRAMEWORK ALREADY HANDLES) 3) Complete garbage. They write the code first and then add tests. That's fine if these were integration tests, but they run their little commands to print out a report that says 92% test coverage. Oh no No nO nO NO No meme-1.gif.jpeg. We have incidents regularly compared to my previous team, which was granted all Senior/Staff engineers and we didn't spend our time chasing dumbass metrics. If you want 100% unit test coverage, just do TDD and not this half-ass yolo cowboy approach. Or just write simple, single responsibility code so it can be tested easily. But nah, you know how it is. These people are annoying as hell to work with, and unfortunately I work at a bigger company so this kind of BS is normal as people try to chase promotions instead of actually shipping quality products. I personally wouldn't hire people like this, nor keep them around this long.
  • jrockway 5y ago
    100% test coverage is really the starting point for testing. Once all the lines of code you've typed in actually run before they're delivered to the customer, then you can start adding the real test scenarios!
    • hinkley 5y ago
      And how, pray tell, do you discover all of the scenarios that need to be covered? If only there were some sort of coverage tool that could tell you...
  • ravenstine 5y ago
    That's the thing that pisses me off about how testing and the periphery around testing (like coverage) are commonly treated. They're seen as being virtuous in and of themselves and no one wants to measure whether they are counterproductive. Even if one took that initiative, good luck getting anyone to care. Sure, I suppose one could say that any tests is better than no tests, but that also dismisses whether testing can be done better. Code coverage is the worst. I've worked with codebases with various degrees of code coverage, and I've never seen a correlation between coverage and the number of bugs encountered. There may be an asymptotic nature to how much coverage is needed, but I would bet the threshold is low in a general sense. Once you've reached even 25% coverage, it's really dubious whether more coverage than that is actually telling you anything. Of course, we could measure success by number of bugs and company revenue, but no, we can't do that; those could make people look bad.
• calderwoodra 5y ago
  Dogmatic unit testing is overrated. For backend testing, I've had amazing success following a few simple principles: 1. Abstract and mock code you don't control that require network calls. 2. Use frameworks that support database abstraction out of the box 3. Test end to end functionality. Every test should basically be getting your DB into a specific state, then hitting an API endpoint and validating the output and/or state change. It's not that hard, you can get decent and worthwhile code coverage for not much work, and even if you only write a few tests with low coverage, you can easily add regression tests later if you notice a bug.
• clueless123 5y ago
  To me, Unit Tests are live documentation for your code. You definitely do not need 100% coverage, but if you can exercise a good percentage of your code, you are making your future "you" live easier. Just consider that in many cases code lifespan is 20% dev , 80% maintenance.
• goto11 5y ago
  > In this article I will be relying on the “standard” definitions, where unit testing targets smallest separable parts of code Maybe that is the standard understanding, but I don't think it was the original meaning, and I don't think it is a helpful definition. Testing only "smallest separable parts" is what leads to an over-reliance on mocking and brittle tests. Unit test should test a "unit of behavior", not a unit of code. That is, a unit test should test a single assertion about the behavior of a component. The test should not care how the behavior is implemented, i.e. it should be possible to refactor or fully rewrite the implementation without changing the test, which means the test should not care if one or many sub-component are involved (as long as no external systems are involved).
• oxff 5y ago
  Unit Testing is software construction's "we have to do something to validate, this is something, we have to do it".
• mvaliente2001 5y ago
  My interpretation of UT has evolved after I began using Clean Architecture. My original approach was to test functions and methods, which had as a consequence the calcification of the code: changes were painful because they broke a lot of test. Their granularity level was too low. Now, my units of test are use cases. They naturally offer great coverture, test and document business logic, and easily allow refactoring. OP criticizes dependency injection, and comment that its unique purpose is making UT easier. I strongly disagree. I've been able to easily replace web frameworks and OORDB libraries using DI, and even develop business logic before selecting infrastructure technology. In that sense, the fact that UT are easier to write is an indicator that the system is easily adaptable. A good UT allows me to reduce integration testing to the minimum. What my IT do is verify that two systems can exchange messages, but as soon as I get a request or response representation stripped from any I/O reference, it's UT time for me.
• ivanhoe 5y ago
  To me it seems that author's quarrel is not with the unit testing per se, but with the idea of the 100% unit test coverage. And I agree with that, coverage is as useless metric as the number of lines of code, its sole purpose is to make managers happy. Tests themselves are not overrated, just like with everything else in engineering one has to use some common sense when choosing what to test, and what not, it's as simple as that - but because this can't be easily bottled and prescribed as a rule of thumb, managers/leads prefer to cover their asses by insisting on 100% coverage...
• fleventynine 5y ago
  Some observations after using many different testing philosophies on many different teams in the past 20 years: 1. Unit tests for code with no side effects (code that takes well defined input and produces well-defined output) are easy to write, easy to understand, and I've never regretted writing them. 2. I've had good experiences transforming as much error-prone logic as possible into code with no side effects. 3. Integration tests that execute in an environment that is similar to production have proven to be incredibly valuable, especially when refactoring. When possible, I prefer to make technology-stack choices that make such an environment quick to setup and teardown such that these tests can run in seconds. 4. Many bugs occur because team A makes bad assumptions about how team B's system behaves, and they encode these assumptions into their (passing) test cases. 5. Unit tests with lots of mocks should be avoided; their cost-to-benefit ratio is terrible. Sometimes the best solution is to delete these tests. Relying on mocks for a few error-case tests that are hard to reproduce on the real system is ok. 6. If they cannot be avoided, fake implementations should be written by the same team that writes the real implementation. They understand how it actually behaves much better than their users, and are in a good place to reuse critical logic from the real implementation in the fake to make it more realistic. 7. If my project includes firmware running on custom hardware, building an emulator for the hardware that can run on standard computing infrastructure is valuable for writing test cases against. 8. More tests or coverage is not always better. We have a finite amount of time to improve our systems, and it is our duty as engineers to ensure that the benefit we get from adding a specific test-case is higher than using that time to make some other improvement to the project.
• cuddlybacon 5y ago
  My gut reaction to the title was to disagree. Looking at the article, the thing that stands out strongest is this graph: https://tyrrrz.me/static/b606743d039cfb56391c5793d0dda8f2/bb630/Test-conversion-efficiency.png https://tyrrrz.me/static/b606743d039cfb56391c5793d0dda8f2/bb... It shows test cost vs test isolation. I agree with the assumed correlation. My experience though is the real correlation is above the assumed correlation, not below it. My guess is that the author's organization moves part of the cost somewhere else where the author doesn't have to see it. At the start of my career, I was one of those people who had this cost dumped on them so that the devs could ignore it: I was an on-team QA engineer.
• kevwil 5y ago
  "Oooo...yeahhhh, ummm...I'm gonna have to go ahead and sort of disagree with you there." - Office Space UT is not the panacea and won't do your laundry for you, but well-tested code is better than sloppily- or un-tested code. Always will be, IMHO. UT is the core of that, for languages where a unit of functionality can be executed directly. Integration tests are also critical, but IMHO it's foolish to do one or the other. "But dependencies ..." yeah that's been solved many times over with mock tools and such, and refactoring to reduce complexity. If your code is too complex to test properly, it's too complex to release. Don't be lazy, do it right.
  • ozim 5y ago
    But you are writing like a single person would be writing whole project. I might not be too lazy - but I have to collaborate with 5-10 people. Well-tested code is also some kind of utopia, no one have seen such code but everyone is claiming that once you get there it is all unicorns and roses. I can write perfect code if I do it alone for my toy project - when I have team of people that has to agree on stuff it is not going to happen.
• mysore 5y ago
  i really wish unit testing was easier for graphics / webgl / 3d development
• commandlinefan 5y ago
  > the interfaces we’ve defined serve no practical purpose other than making unit testing possible That's actually almost never true - abstracting the sorts of external dependencies (like databases and web services) that actually cause problems for unit tests make the code more manageable in terms of maintenance and troubleshooting. The problem is when people go mock-crazy and mock every class that the unit depends on. Rather, just mock the things that can actually make the unit test fail if they're unreachable for some reason independent of the actual code.
• monksy 5y ago
  I stopped reading half way through his claim. TL;DR The guy doesn't understand how to test code and he doesn't understand test isolation. He responds to requests to eat his vegtiables as "but I dont hafta" Unit tests are intended to test small mostly uncoupled parts of the code. (Usually scoped to single functions). Integration tests confirms the combination of functions, or the class in a context. Testing of user behavior is intended to be done in the functional tests, and only that. When you get to functional tests you should assume that most of the code is trustable underneath, as that the tests support that. The intention of the separation of the testing strategies means you don't have to do a ton of combination tests in a higher level (where the test is more expensive to run, write and maintain).
• johnthescott 5y ago
  amen.
• dullcrisp 5y ago
  The first question I’d ask if I were writing unit tests for their first example is why a solar calculator containing mathematical business logic should depend on a location provider rather than just accepting a location as a method parameter. Unit tests help you to design your classes based on use cases—if a simple class is hard to test you should think about whether it might have a simpler design. If you have a brute forced design and each of your classes have dozens of dependencies and your tests require pages of setup and are hard to reason about and are tightly coupled to the implementation then the tests might not be providing any benefit at all.
  • blt 5y ago
    Totally agree. Functions should never accept an object reference or another function as a parameter when the necessary information could be provided directly as data (where "data" means C structs without pointers, protobuf, JSON, ...). Even code that mutates its arguments is easy to test if it's all simple data.
• tippytippytango 5y ago
  I forgot who, but someone semi-famous said unit tests should be called "programmer tests". That is, unit tests are for the programmers, not for verification. They are used as a tool by the programmer to: Tighten up the development loop (manual e2e is often too slow) Prove to yourself that the code does what you want Communicate intent to other programmers Documentation to other programmers Help programmers debug regressions I really think this holds up because, really, who except for programmers that can read and comprehend unit tests are going to trust they verify a product works? The more distant a stakeholder is from a project the more holistic the testing needs to be in order for that stakeholder to trust it.
  • gnulinux 5y ago
    This is exactly right. I write unittests first and foremost to gain confidence this shit won't blow up on me on prod. I also take extra care of them because it's by far the best documentation of the code. To me these are good enough reasons to write extra potentially unnecessary tests.
• wilmoore 5y ago
  I remember being on a team where I'd frequently point out that the tests were not only flaky, but useless. I suggested that we delete the useless & redundant tests and either fix or delete the flaky ones. That suggestion was met with not only resistance, but almost hatred. I think some of them got way too attached to these tests and could no longer see the forest for the tress.
• Barrera 5y ago
  > Although the design above is perfectly valid in terms of OOP, neither of these classes are actually unit-testable. Because LocationProvider depends on its own instance of HttpClient and SolarCalculator in turn depends on LocationProvider, it’s impossible to isolate the business logic that may be contained within methods of these classes. The design of those classes is bad, and the perceived need to reach for mocks (or interfaces) is telling you that. The next step the author takes is to go for interfaces. That's not what I'd do. Instead, I'd decouple network from calculation with intermediate dumb data structures. The network side produces the data structure and the calculation side consumes it. The network side can further be broken down with a transformation of the API call result into the dumb data structure used for calculations. This idea comes from an old Bob Martin presentation [1]. The whole thing is worth watching to put it into context. This is the kind of thing that never seems to get discussed in these pieces praising integration tests over unit tests. For even better results, ditch the classes altogether and just use functions. It's quite surprising how badly designs come off the rails with the "everything is a class" model. [1] https://www.youtube.com/watch?v=WpkDN78P884 https://www.youtube.com/watch?v=WpkDN78P884
  • jesterman81 5y ago
    This!
  • hinkley 5y ago
    Which in turn comes from Bertrand Meyer, who probably stole it from somebody else in 1990. Code should act or decide, but not both.
• jimmont 5y ago
  the testing pyramid is worth knowing about; unit tests double code size and maintenance cost for the unit; I've found unit tests only fit long-lived code and coverage doesn't generally correlate with reliability of product performance; black box end to end testing at the interface level for the general scenarios does correlate with reliability, for those scenarios; writing tests that properly separate concerns in this way seems to make it easier to write and maintain tests, and the product
• tpoacher 5y ago
  I don't consider myself an expert on this topic, but I can't shake the feeling that this person almost defines unit tests in a way that builds them up to fail, but is not how people who use Unit Tests effectively would necessarily define or understand by the term unit tests. It's certainly not compatible with what Michael Feathers describes as unit tests in his "Working Effectively with Legacy Code" classic, a book I consider the bible on unit tests (even if a bit dated by now). If anything, many of the things the article describes as drawbacks of "unit tests" are exactly the things that Michael Feathers seems to describe Unit Tests are "not" if they are to be effective as unit tests. Obviously I'm being awfully vague here, but, yeah. That was the feeling I got from the article. It's just another case of "My particular definition of X, which is not necessarily what others understand by X, is bad. Use what I define as Y instead, which may be what some people used to mean by X in the first place, but let's call it Y from now on because X is bad."