5 ms·

Unit Tests Are Overrated: Rethinking Testing Strategies

42 points by kiyanwang 3y ago

scotty79 3y ago
> removing distinct tester roles and trying to hold developers accountable for the quality of the code they are writing That's the worst idea ever. Testing requires completely distinct set of skills different than development. There are 10x programmers but there are also 10x testers.
- baz00 3y ago
  Yeah there's some poor thinking by the original article. Developers must be accountable for their code quality yes but must be held accountable by an external force. I've been in too many organisations where empty, useless unit tests are generated to build coverage and say "look we are accountable" when they are not really. That might be an external QA / review team under a completely different management tree to the development team. Realistically I've never seen an org where it's possible to scale it up and self-accountability being viable. There are too many defective personalities to manage this.
  - pipo234 3y ago
    He's not promoting that removing dedicated testers is a good idea. Full(er) quote: > Several strategies are getting deployed — shift left (e.g., more unit tests), shift right (e.g., test in production), transitioning from manual to automated testing and even removing distinct tester roles and trying to hold developers accountable for the quality of the code they are writing. > > These testing strategies are met with varying degrees of success and can even differ from team to team based on skill level, maturity, and culture. By all means, have developers write their own unit tests. To show their intent, document use cases, etc. And do not leave that to a junior colleague or a brain dead AI, optimizing for high McCabe complexity code coverage or whatever metric. AND (indeed) have QA done by someone with different skills. Development requires a focused mindset, in the zone for long stretches, int the box. Test requires distraction, exploring, out of the box — "hey, that's weird" or "I wonder what will happen if I do this".
    - astrobe_ 3y ago
      Trying to avoid shipping bugs is a defense-in-depth affair, because fundamentally tests can only show the existence of a bug, but not that a program is bug-free. It starts right in the editor, where syntax highlighting can help with spotting mistakes such as writing 010 instead of 10 (0 prefix for octal is a terrible idea). Pretty sure that bug-free code is one of the prides of programmers worth their salt; but there's no helping, when you forget about something while coding, it is very likely that you also forget to test it. That's why we need a separate tester.
SideburnsOfDoom 3y ago
This is true only if you adhere to the common definition of "unit test" where a unit test must test a single class or method on a class, often with use of lots of mocks. Common but not necessarily best. Let me pose a question: if refactoring is "changing code while tests stay green", then how do you refactor when any change to public methods always breaks tests? When "extract class" means a change to test class structure as well? Testing at this close-coupled level far from mandatory. IMHO, the use of the word "unit" is deliberately vague: if it was always a "class test" or "method test" then it would be called "method test" and "class test" not "unit". "unit" means "you choose". it means "a chunk of useful functionality", in problem domain or business terms. It means "can I test this in isolation from e.g. file system, http services and other concurrent tests", as per Michael Feathers's guidelines (1) I'm not suggesting that you should never unit test at a close level of class and method; I am suggesting that you might benefit from a majority of unit tests being not at this close level. 1) https://www.artima.com/weblogs/viewpost.jsp?thread=126923 https://www.artima.com/weblogs/viewpost.jsp?thread=126923
- nottorp 3y ago
  Well, Agile also used to make sense before it was called Agile and turned into a set of rituals. Same for unit tests and any other practical matters that have turned into religion. Plus there's the little matter that you tend to think that your application is tested if it has automated tests. Then your colleague in sales who clicks like he's having an epileptic episode comes over and your app crashes 3 different times [1]. [1] True story. I used to ask him to click 2 min on whatever I was working on when he passed by my desk. [2] PalmOS had the gremlins tool. To my knowledge no other platform has it.
  - SideburnsOfDoom 3y ago
    > Well, Agile also used to make sense before it was called Agile and turned into a set of rituals. If you want to make sense of "Agile" read "The Agile manifesto" (1), it's not long even with the 2nd page (2). but yes, the modern criticisms of modern "agile" are pretty much the same as the problems that caused that agile movement in the first place. And are starkly at odds with the Agile Manifesto, e.g. "Individuals and interactions over processes and tools" and "Responding to change over following a plan" vs "Our tool, Jira, enables micromanagement of human resources over a long-term plan" 1) https://agilemanifesto.org/ https://agilemanifesto.org/ 2) https://agilemanifesto.org/principles.html https://agilemanifesto.org/principles.html
vosper 3y ago
> For as much as companies have focused on DevOps and continuous delivery, the sad truth is that the trend of releasing software has remained relatively flat, with only 10% of teams releasing at least daily. Sad truth? Only? How about an alternative interpretation: 90% of teams don’t need or want to release at least daily. And so they don’t. Continuous delivery is something that 90% of software teams don’t need and shouldn’t waste time on. It’s been overhyped.
- ckdot2 3y ago
  Even if you don't need or want to deploy daily, your software should be continous deliverable.
- ojkelly 3y ago
  Releasing code to prod is the end of the feedback loop, until then you have possibly working software. The long you wait until release, the less likely it will work as you expect. You don’t need to release to users, that can happen later. Every team I’ve been on that could release daily never wanted to go back to anything less. The tooling is still too hard for mass adoption by teams, but the benefits are real.
- throw__away7391 3y ago
  You’re both conflating two issues. A solid fully automated CI/CD is table stakes for any professional project. Releasing on some particular cadence has little to do with this aside from the fact that it isn’t possible without such automation. In many contexts I’d consider security and audit ability far more important than release pace, and even on personal projects you want the deployment to be reproducible and explicitly defined.
- benwad 3y ago
  I disagree about continuous delivery. Small, regular releases mean less risk in each release. Basically every company I've worked for has used CI/CD, although 'release daily' seems arbitrary (you release when you merge - that's the whole idea. If there's nothing merged today, we have nothing to release).
hkon 3y ago
Most of the code I have written in the last years has been request response centered. For these I find integration tests to be the best testing strategy. I'll add the occasional unit test for logic heavy operations.
- kypro 3y ago
  I think the real value in unit tests isn't validating your logic is accurate (although sometimes it is for logic heavy methods as you say), but in validating that subsequent changes (often made by other devs) don't break prior logic. In complex codebases it's easy to miss things when making changes and unit tests are great ways to catch issues in really simple bits of logic that have just later been overlooked. Integration tests are typically so heavy and multivariable they can only be used to validate the most common scenarios. They're great sanity checks, but provide terrible coverage whenever there exists nuanced logic.
fefe23 3y ago
This is the first and only article by this guy on that web site. He sells browser automation. I tried finding non-platitude insights but couldn't. His conclusion is: > By embracing a more holistic testing approach, teams can deliver better software faster. As with so much in life, the key is using the right tool for the job, not just the tool that is most convenient at the moment. When it comes to testing, smart and strategic beats brute force every time. Note how this statement conveys zero information and could be said about literally anything.
- ildon 3y ago
  Sounds a lot like chatgpt
  - jasonjmcghee 3y ago
    I think it similarly lacks substance, but doesn't have the ChatGPT tone / cadence / vocab imo.
- quickthrower2 3y ago
  Sounds like someone selling a system to a non technical pointy hair
- h0l0cube 3y ago
  If you're looking for an article with substance and a similar bent (functional specifications driving decisions on test granularity), I'd try this one: https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste...
  - benterix 3y ago
    This one is much more substantial and the author has some excellent points like the futility of aiming at 100% coverage.
    - h0l0cube 3y ago
baz00 3y ago
No they're not. Unit tests are only overrated when you're hacking some shit together rather than doing forward engineering. No one does proper forward engineering these days which is why it doesn't seem applicable. If you have thought about the problem first you know the constraints and they match 1:1 with the test cases, then you build it, then it works. You can't test or understand side effects for everything at the boundary of a system from experience. The biggest sin of the whole IT sector at the moment is lack of quality and this thinking drives it further towards hell. Do less, do it slower but do a better job please.
- kriro 3y ago
  This POV is debateable (but I sympathize strongly with your sentiment). If quick but shitty allows me to capture a lot of value then quick and shitty might be the way to go. Yes there are major downsides but especially in a "scaleup" setting they are often dwarfed by the upsides of shitty but fast. As an engineer, I hate it because I want to build perfect and long term all the time but in many cases you can't really argue with the monetary benefits of knowingly delivering underwhelming things that get the job done well enough for now (even knoing deep in your heart that many of the issues might never be fixed). A good example is security. Sometimes it's righ to ship stuff with security issues that you know exist and might even be 10+ million dollar issues if discovered quickly if for example fixing thoses issues might interfere with adoption because they make user onboarding slightly more annoying.
  - baz00 3y ago
    Let's flip this around. As a customer, if time to market is your primary driver, I don't want your product because you're choosing to put me at risk.
    - aidos 3y ago
      That’s up to the customer. If the software is delivering good return on investment, they really might not care at all.
- louthy 3y ago
pipo234 3y ago
> TDD itself is also overrated, as writing testable code is much more important than the actual tests. Indeed, that is exactly the point of unit test and code coverage: proving that your code is well organized and testable through its API. > This is why getting artificial intelligence projects to create basic tests isn’t the impressive accomplishment people seem to think it is. Also correct. Additionally, the point of unit tests is to document, to show the code's use case. It's generally not possible to device intent (why) from implementation (how), but if the AI were magically able to deduce that that's a sign that the code is either trivial, or plagiarized (or both). > By embracing a more holistic testing approach, teams can deliver better software faster. Sure, if velocity is your objective. Another approach is to stop and think about whether you're testing in internal (unit test) or external (integration, acceptance test) quality. As a developer, I use unit tests primarily to verify complicated business logic to prevent the embarrassment of obvious flaws in my code further downstream.
- haspok 3y ago
  > > TDD itself is also overrated, as writing testable code is much more important than the actual tests. > Indeed, that is exactly the point of unit test and code coverage: proving that your code is well organized and testable through its API. You can have 100% unit test code coverage and your code can still be absolute sh*t and do absolutely nothing useful. Also, your code can work, with 100% unit test coverage, and have the tests test nothing useful. So reasoning about the quality of your code through the quantity of your unit tests is highly suspicious. And btw, TDD does not advocate code quality and well organized code - if anything, it lends itself to the opposite.
  - pipo234 3y ago
    > You can have 100% unit test code coverage and your code can still be absolute sh*t and do absolutely nothing useful. Totally true. But give me those unit tests, and I can tell that the code is sh*t. If I have trouble understanding the source, it could be because it's doing complicated stuff. It could be highly optimized. It could I'm too dumb. If I have trouble understanding the unit tests demonstrating what the code does, there is either something wrong with the tests or the implementation. Fix it, rinse repeat. I wasn't saying that unit tests or code coverage are a panacea or that TDD guarantees great code. But it's a tool. It's better than writing documentation, but it serves more or less the same purpose. Sometimes it's useful, sometimes not. And its complete absence is usually a bad smell. I think Michael Feather's book demonstrates how code improves by simply making it testable https://www.oreilly.com/library/view/working-effectively-with/0131177052/ https://www.oreilly.com/library/view/working-effectively-wit...
h0l0cube 3y ago
This is much better critique of the value of unit tests https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste...
Dudester230602 3y ago
Unit tests are a poor man's type system.
- solraph 3y ago
  Types are wonderful, and a good type system makes so many errors disappear / impossible, that I'd otherwise have to write tests to cover in a dynamic language. Types however, are not a substitute for unit tests. No type system is going to tell me if my income tax calculator covers all the edge cases.
  - haspok 3y ago
    I would argue that your income tax calculator logic should probably not be unit tested, it sounds complicated enough to involve multiple classes / units of code. What you really want to test is business logic, not code logic.
    - pasc1878 3y ago
      So the unit test is a business case. All tests should be on business logic - code logic might not fit the business case at all and so that should be a fail.
    - solraph 3y ago
      I was using the example of income tax as an entirely theoretical example of where types won't save you in response to the GP. That said, I don't follow what you mean by "to test is business logic, not code logic." To me, that is a distinction without difference. An income tax calculator in a (sane) tax system is essentially a set of pure of functions or classes. This is a perfect place for a unit test. I might have additional unit tests for sub components of the hairy bits, but it's still unit tests all the way.
- cjfd 3y ago
  You need both. The type system is not going to catch if you do Rectangle(width, height) where you should have done Rectangle(height, width).
choeger 3y ago
Sigh. Releasing more often is not an ideal situation. Yes, being able to release every single change on its own, fully tested, is a nice to have tool. But the real world works in office days, weeks, and quarters. And whatever amount of work you put into your test suite and automation, sometimes you cannot test some things. Sometimes you have to wait for a real day (or God forbid a real quarter) to pass. Yes, testing, especially unit and integration tests, are a sine-qua-non for quality work. But so are automatic proofs of absence of certain errors (vulgo type checks) and non-functional tests. But all this has been at least partially resolved. What we are still sorely lacking is simulation. Everyone who ever worked with a staging environment knows how limited this usually is. What I would like to work with before I retire is a tool that lets me standup a complete copy of the production environment, isolated from the real world of course, with all external services automatically mocked. Now that would help to improve the quality.
- smokel 3y ago
  I've worked on quite a few, admittedly large, projects where we simulate nearly all external interfaces. It is mostly the human that is most expensive to replace with a simulator, apart from simple end-to-end or stress tests. Is anyone working on a reinforcement learning agent for proper monkey testing? :)
- MoreQARespect 3y ago
  >And whatever amount of work you put into your test suite and automation, sometimes you cannot test some things. Sometimes you have to wait for a real day (or God forbid a real quarter) to pass. I honestly can't imagine any software product with test automation that would require more than a real day to pass, never mind a quarter. Not unless some person or bureaucratic process was deliberately slowing it down. >But all this has been at least partially resolved. What we are still sorely lacking is simulation. Everyone who ever worked with a staging environment knows how limited this usually is. What I would like to work with before I retire is a tool that lets me standup a complete copy of the production environment, isolated from the real world of course, with all external services automatically mocked. Now that would help to improve the quality. There are plenty of tools that do this. I run simulators to catch sent mails, pretend to be browsers, mock REST APIs, etc. This is the bread and butter of test automation. The actual scenario writing I find to be the easiest part. The real problem is that this stuff is usually done really badly by cheap, poorly skilled people who will never earn as much as a dev and who almost never do this part of the job properly. The actual solution is in my username.
davethedevguy 3y ago
Most criticisms of unit tests in articles like this are actually criticisms of bad unit tests. They also overlook a large part of the value of unit tests, which is fast feedback during development time. We could push all of our tests up to the browser level, but then I lose the almost instant feedback that unit tests give me. Sure, if your tests mock everything, and only test individual methods, then they probably are brittle and provide little value over more 'expensive' (e.g. slower) tests higher up the test pyramid. If the tests only interact with the public APIs (e.g. the 'ports' in a ports and adapters architecture), then they are less coupled to implementation, can represent real business scenarios (instead of 'input x gives result y'), and add more value.
- mattmanser 3y ago
  You're not describing unit tests, they're integration tests. You've fallen into the very trap of terminology the author mentions at the beginning of the article.
  - davethedevguy 3y ago
    I think I'm disagreeing with the definition implied by the article. To me, an integration test involves integrating with another system. Testing groups of functions or classes together is still a unit test, by Wikipedia's definition at least "unit testing is a software testing method by which individual units of source code—sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures—are tested to determine whether they are fit for use" [1] and by Michael Feather's definition "A test is not a unit test if: - It talks to the database - It communicates across the network - It touches the file system - It can't run at the same time as any of your other unit tests - You have to do special things to your environment (such as editing config files) to run it." [2] Indeed, it's the inevitable result of doing TDD; the refactor step is likely to break out smaller classes/functions already covered by the existing tests. If we write new, more granular tests every time we do the refactor step, then we end up with brittle tests. [1] https://en.wikipedia.org/wiki/Unit_testing https://en.wikipedia.org/wiki/Unit_testing [2] https://www.artima.com/weblogs/viewpost.jsp?thread=126923 https://www.artima.com/weblogs/viewpost.jsp?thread=126923 Edit: To clarify, I'm not talking about calling the public API over the network, I'm talking about the public methods that consumers (including your network layer) would call. That might be where the confusion lies here.
KronisLV 3y ago
I guess it depends on the software that you are writing and how it's structured. The testing pyramid seems to make some sense, as long as you realize that you need a multitude of different tests for checking various things. For example, when writing a library that dealt with JWT and called into some other libraries, I opted for a lot of unit tests, I think the code coverage was around 90-95%. Writing tests for almost all of the functionality helped me actually discover one or two bugs which I could immediately fix without having them be released. Also, tests acted as documentation there, showing various use cases for the library, in addition to helping me actually make the API more user friendly. Integration and UI tests are also useful to see how the solution fares in the real world, but once you have to automate a browser or deal with mocking the DB (if you can't spin up a container, migrate the schema and seed it with real data for your tests), the complexity skyrockets. You might still want to have them and plenty of them, but at the same time you'll deal with slow and flaky tests more often, especially if the tech you choose or are stuck with won't support your needs all that well (e.g. Selenium). It's even worse if the thing you're testing changes a lot. As long as you have a staging environment, you should be even well off for anything from load testing to even the occasional bit of manual/acceptance testing, or trying to reproduce issues there. With the help of logs/metrics you can reason about changes/bugs/reproduction scenarios more safely, without impact on prod. Even better with other supporting things, like feature flags. Test whatever you can: sometimes the limiting factor will be time, other times the complexity of the system (or how testable it is in the first place, e.g. no fixed element IDs in a web browser), other times you won't get the infrastructure, but it's still a nice pursuit.
xnx 3y ago
Some good guidelines just posted today: https://web.dev/ta-what-to-test/ https://web.dev/ta-what-to-test/

42 points by kiyanwang 3y ago

> removing distinct tester roles and trying to hold developers accountable for the quality of the code they are writing That's the worst idea ever. Testing requires completely distinct set of skills different than development. There are 10x programmers but there are also 10x testers.

baz00 3y ago
Yeah there's some poor thinking by the original article. Developers must be accountable for their code quality yes but must be held accountable by an external force. I've been in too many organisations where empty, useless unit tests are generated to build coverage and say "look we are accountable" when they are not really. That might be an external QA / review team under a completely different management tree to the development team. Realistically I've never seen an org where it's possible to scale it up and self-accountability being viable. There are too many defective personalities to manage this.
- pipo234 3y ago
  He's not promoting that removing dedicated testers is a good idea. Full(er) quote: > Several strategies are getting deployed — shift left (e.g., more unit tests), shift right (e.g., test in production), transitioning from manual to automated testing and even removing distinct tester roles and trying to hold developers accountable for the quality of the code they are writing. > > These testing strategies are met with varying degrees of success and can even differ from team to team based on skill level, maturity, and culture. By all means, have developers write their own unit tests. To show their intent, document use cases, etc. And do not leave that to a junior colleague or a brain dead AI, optimizing for high McCabe complexity code coverage or whatever metric. AND (indeed) have QA done by someone with different skills. Development requires a focused mindset, in the zone for long stretches, int the box. Test requires distraction, exploring, out of the box — "hey, that's weird" or "I wonder what will happen if I do this".
  - astrobe_ 3y ago
    Trying to avoid shipping bugs is a defense-in-depth affair, because fundamentally tests can only show the existence of a bug, but not that a program is bug-free. It starts right in the editor, where syntax highlighting can help with spotting mistakes such as writing 010 instead of 10 (0 prefix for octal is a terrible idea). Pretty sure that bug-free code is one of the prides of programmers worth their salt; but there's no helping, when you forget about something while coding, it is very likely that you also forget to test it. That's why we need a separate tester.

SideburnsOfDoom 3y ago

This is true only if you adhere to the common definition of "unit test" where a unit test must test a single class or method on a class, often with use of lots of mocks. Common but not necessarily best. Let me pose a question: if refactoring is "changing code while tests stay green", then how do you refactor when any change to public methods always breaks tests? When "extract class" means a change to test class structure as well? Testing at this close-coupled level far from mandatory. IMHO, the use of the word "unit" is deliberately vague: if it was always a "class test" or "method test" then it would be called "method test" and "class test" not "unit". "unit" means "you choose". it means "a chunk of useful functionality", in problem domain or business terms. It means "can I test this in isolation from e.g. file system, http services and other concurrent tests", as per Michael Feathers's guidelines (1) I'm not suggesting that you should never unit test at a close level of class and method; I am suggesting that you might benefit from a majority of unit tests being not at this close level. 1) https://www.artima.com/weblogs/viewpost.jsp?thread=126923 https://www.artima.com/weblogs/viewpost.jsp?thread=126923

nottorp 3y ago
Well, Agile also used to make sense before it was called Agile and turned into a set of rituals. Same for unit tests and any other practical matters that have turned into religion. Plus there's the little matter that you tend to think that your application is tested if it has automated tests. Then your colleague in sales who clicks like he's having an epileptic episode comes over and your app crashes 3 different times [1]. [1] True story. I used to ask him to click 2 min on whatever I was working on when he passed by my desk. [2] PalmOS had the gremlins tool. To my knowledge no other platform has it.
- SideburnsOfDoom 3y ago
  > Well, Agile also used to make sense before it was called Agile and turned into a set of rituals. If you want to make sense of "Agile" read "The Agile manifesto" (1), it's not long even with the 2nd page (2). but yes, the modern criticisms of modern "agile" are pretty much the same as the problems that caused that agile movement in the first place. And are starkly at odds with the Agile Manifesto, e.g. "Individuals and interactions over processes and tools" and "Responding to change over following a plan" vs "Our tool, Jira, enables micromanagement of human resources over a long-term plan" 1) https://agilemanifesto.org/ https://agilemanifesto.org/ 2) https://agilemanifesto.org/principles.html https://agilemanifesto.org/principles.html

vosper 3y ago

> For as much as companies have focused on DevOps and continuous delivery, the sad truth is that the trend of releasing software has remained relatively flat, with only 10% of teams releasing at least daily. Sad truth? Only? How about an alternative interpretation: 90% of teams don’t need or want to release at least daily. And so they don’t. Continuous delivery is something that 90% of software teams don’t need and shouldn’t waste time on. It’s been overhyped.

ckdot2 3y ago
Even if you don't need or want to deploy daily, your software should be continous deliverable.
ojkelly 3y ago
Releasing code to prod is the end of the feedback loop, until then you have possibly working software. The long you wait until release, the less likely it will work as you expect. You don’t need to release to users, that can happen later. Every team I’ve been on that could release daily never wanted to go back to anything less. The tooling is still too hard for mass adoption by teams, but the benefits are real.
throw__away7391 3y ago
You’re both conflating two issues. A solid fully automated CI/CD is table stakes for any professional project. Releasing on some particular cadence has little to do with this aside from the fact that it isn’t possible without such automation. In many contexts I’d consider security and audit ability far more important than release pace, and even on personal projects you want the deployment to be reproducible and explicitly defined.
benwad 3y ago
I disagree about continuous delivery. Small, regular releases mean less risk in each release. Basically every company I've worked for has used CI/CD, although 'release daily' seems arbitrary (you release when you merge - that's the whole idea. If there's nothing merged today, we have nothing to release).

hkon 3y ago

Most of the code I have written in the last years has been request response centered. For these I find integration tests to be the best testing strategy. I'll add the occasional unit test for logic heavy operations.

kypro 3y ago
I think the real value in unit tests isn't validating your logic is accurate (although sometimes it is for logic heavy methods as you say), but in validating that subsequent changes (often made by other devs) don't break prior logic. In complex codebases it's easy to miss things when making changes and unit tests are great ways to catch issues in really simple bits of logic that have just later been overlooked. Integration tests are typically so heavy and multivariable they can only be used to validate the most common scenarios. They're great sanity checks, but provide terrible coverage whenever there exists nuanced logic.

fefe23 3y ago

This is the first and only article by this guy on that web site. He sells browser automation. I tried finding non-platitude insights but couldn't. His conclusion is: > By embracing a more holistic testing approach, teams can deliver better software faster. As with so much in life, the key is using the right tool for the job, not just the tool that is most convenient at the moment. When it comes to testing, smart and strategic beats brute force every time. Note how this statement conveys zero information and could be said about literally anything.

ildon 3y ago
Sounds a lot like chatgpt
- jasonjmcghee 3y ago
  I think it similarly lacks substance, but doesn't have the ChatGPT tone / cadence / vocab imo.
quickthrower2 3y ago
Sounds like someone selling a system to a non technical pointy hair
h0l0cube 3y ago
If you're looking for an article with substance and a similar bent (functional specifications driving decisions on test granularity), I'd try this one: https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste...
- benterix 3y ago
  This one is much more substantial and the author has some excellent points like the futility of aiming at 100% coverage.
  - h0l0cube 3y ago

baz00 3y ago

No they're not. Unit tests are only overrated when you're hacking some shit together rather than doing forward engineering. No one does proper forward engineering these days which is why it doesn't seem applicable. If you have thought about the problem first you know the constraints and they match 1:1 with the test cases, then you build it, then it works. You can't test or understand side effects for everything at the boundary of a system from experience. The biggest sin of the whole IT sector at the moment is lack of quality and this thinking drives it further towards hell. Do less, do it slower but do a better job please.

kriro 3y ago
This POV is debateable (but I sympathize strongly with your sentiment). If quick but shitty allows me to capture a lot of value then quick and shitty might be the way to go. Yes there are major downsides but especially in a "scaleup" setting they are often dwarfed by the upsides of shitty but fast. As an engineer, I hate it because I want to build perfect and long term all the time but in many cases you can't really argue with the monetary benefits of knowingly delivering underwhelming things that get the job done well enough for now (even knoing deep in your heart that many of the issues might never be fixed). A good example is security. Sometimes it's righ to ship stuff with security issues that you know exist and might even be 10+ million dollar issues if discovered quickly if for example fixing thoses issues might interfere with adoption because they make user onboarding slightly more annoying.
- baz00 3y ago
  Let's flip this around. As a customer, if time to market is your primary driver, I don't want your product because you're choosing to put me at risk.
  - aidos 3y ago
    That’s up to the customer. If the software is delivering good return on investment, they really might not care at all.
louthy 3y ago

pipo234 3y ago

> TDD itself is also overrated, as writing testable code is much more important than the actual tests. Indeed, that is exactly the point of unit test and code coverage: proving that your code is well organized and testable through its API. > This is why getting artificial intelligence projects to create basic tests isn’t the impressive accomplishment people seem to think it is. Also correct. Additionally, the point of unit tests is to document, to show the code's use case. It's generally not possible to device intent (why) from implementation (how), but if the AI were magically able to deduce that that's a sign that the code is either trivial, or plagiarized (or both). > By embracing a more holistic testing approach, teams can deliver better software faster. Sure, if velocity is your objective. Another approach is to stop and think about whether you're testing in internal (unit test) or external (integration, acceptance test) quality. As a developer, I use unit tests primarily to verify complicated business logic to prevent the embarrassment of obvious flaws in my code further downstream.

haspok 3y ago
> > TDD itself is also overrated, as writing testable code is much more important than the actual tests. > Indeed, that is exactly the point of unit test and code coverage: proving that your code is well organized and testable through its API. You can have 100% unit test code coverage and your code can still be absolute sh*t and do absolutely nothing useful. Also, your code can work, with 100% unit test coverage, and have the tests test nothing useful. So reasoning about the quality of your code through the quantity of your unit tests is highly suspicious. And btw, TDD does not advocate code quality and well organized code - if anything, it lends itself to the opposite.
- pipo234 3y ago
  > You can have 100% unit test code coverage and your code can still be absolute sh*t and do absolutely nothing useful. Totally true. But give me those unit tests, and I can tell that the code is sh*t. If I have trouble understanding the source, it could be because it's doing complicated stuff. It could be highly optimized. It could I'm too dumb. If I have trouble understanding the unit tests demonstrating what the code does, there is either something wrong with the tests or the implementation. Fix it, rinse repeat. I wasn't saying that unit tests or code coverage are a panacea or that TDD guarantees great code. But it's a tool. It's better than writing documentation, but it serves more or less the same purpose. Sometimes it's useful, sometimes not. And its complete absence is usually a bad smell. I think Michael Feather's book demonstrates how code improves by simply making it testable https://www.oreilly.com/library/view/working-effectively-with/0131177052/ https://www.oreilly.com/library/view/working-effectively-wit...

h0l0cube 3y ago

This is much better critique of the value of unit tests https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste...

Dudester230602 3y ago

Unit tests are a poor man's type system.

solraph 3y ago
Types are wonderful, and a good type system makes so many errors disappear / impossible, that I'd otherwise have to write tests to cover in a dynamic language. Types however, are not a substitute for unit tests. No type system is going to tell me if my income tax calculator covers all the edge cases.
- haspok 3y ago
  I would argue that your income tax calculator logic should probably not be unit tested, it sounds complicated enough to involve multiple classes / units of code. What you really want to test is business logic, not code logic.
  - pasc1878 3y ago
    So the unit test is a business case. All tests should be on business logic - code logic might not fit the business case at all and so that should be a fail.
  - solraph 3y ago
    I was using the example of income tax as an entirely theoretical example of where types won't save you in response to the GP. That said, I don't follow what you mean by "to test is business logic, not code logic." To me, that is a distinction without difference. An income tax calculator in a (sane) tax system is essentially a set of pure of functions or classes. This is a perfect place for a unit test. I might have additional unit tests for sub components of the hairy bits, but it's still unit tests all the way.
cjfd 3y ago
You need both. The type system is not going to catch if you do Rectangle(width, height) where you should have done Rectangle(height, width).

choeger 3y ago

Sigh. Releasing more often is not an ideal situation. Yes, being able to release every single change on its own, fully tested, is a nice to have tool. But the real world works in office days, weeks, and quarters. And whatever amount of work you put into your test suite and automation, sometimes you cannot test some things. Sometimes you have to wait for a real day (or God forbid a real quarter) to pass. Yes, testing, especially unit and integration tests, are a sine-qua-non for quality work. But so are automatic proofs of absence of certain errors (vulgo type checks) and non-functional tests. But all this has been at least partially resolved. What we are still sorely lacking is simulation. Everyone who ever worked with a staging environment knows how limited this usually is. What I would like to work with before I retire is a tool that lets me standup a complete copy of the production environment, isolated from the real world of course, with all external services automatically mocked. Now that would help to improve the quality.

smokel 3y ago
I've worked on quite a few, admittedly large, projects where we simulate nearly all external interfaces. It is mostly the human that is most expensive to replace with a simulator, apart from simple end-to-end or stress tests. Is anyone working on a reinforcement learning agent for proper monkey testing? :)
MoreQARespect 3y ago
>And whatever amount of work you put into your test suite and automation, sometimes you cannot test some things. Sometimes you have to wait for a real day (or God forbid a real quarter) to pass. I honestly can't imagine any software product with test automation that would require more than a real day to pass, never mind a quarter. Not unless some person or bureaucratic process was deliberately slowing it down. >But all this has been at least partially resolved. What we are still sorely lacking is simulation. Everyone who ever worked with a staging environment knows how limited this usually is. What I would like to work with before I retire is a tool that lets me standup a complete copy of the production environment, isolated from the real world of course, with all external services automatically mocked. Now that would help to improve the quality. There are plenty of tools that do this. I run simulators to catch sent mails, pretend to be browsers, mock REST APIs, etc. This is the bread and butter of test automation. The actual scenario writing I find to be the easiest part. The real problem is that this stuff is usually done really badly by cheap, poorly skilled people who will never earn as much as a dev and who almost never do this part of the job properly. The actual solution is in my username.

davethedevguy 3y ago

Most criticisms of unit tests in articles like this are actually criticisms of bad unit tests. They also overlook a large part of the value of unit tests, which is fast feedback during development time. We could push all of our tests up to the browser level, but then I lose the almost instant feedback that unit tests give me. Sure, if your tests mock everything, and only test individual methods, then they probably are brittle and provide little value over more 'expensive' (e.g. slower) tests higher up the test pyramid. If the tests only interact with the public APIs (e.g. the 'ports' in a ports and adapters architecture), then they are less coupled to implementation, can represent real business scenarios (instead of 'input x gives result y'), and add more value.

mattmanser 3y ago
You're not describing unit tests, they're integration tests. You've fallen into the very trap of terminology the author mentions at the beginning of the article.
- davethedevguy 3y ago
  I think I'm disagreeing with the definition implied by the article. To me, an integration test involves integrating with another system. Testing groups of functions or classes together is still a unit test, by Wikipedia's definition at least "unit testing is a software testing method by which individual units of source code—sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures—are tested to determine whether they are fit for use" [1] and by Michael Feather's definition "A test is not a unit test if: - It talks to the database - It communicates across the network - It touches the file system - It can't run at the same time as any of your other unit tests - You have to do special things to your environment (such as editing config files) to run it." [2] Indeed, it's the inevitable result of doing TDD; the refactor step is likely to break out smaller classes/functions already covered by the existing tests. If we write new, more granular tests every time we do the refactor step, then we end up with brittle tests. [1] https://en.wikipedia.org/wiki/Unit_testing https://en.wikipedia.org/wiki/Unit_testing [2] https://www.artima.com/weblogs/viewpost.jsp?thread=126923 https://www.artima.com/weblogs/viewpost.jsp?thread=126923 Edit: To clarify, I'm not talking about calling the public API over the network, I'm talking about the public methods that consumers (including your network layer) would call. That might be where the confusion lies here.

KronisLV 3y ago

I guess it depends on the software that you are writing and how it's structured. The testing pyramid seems to make some sense, as long as you realize that you need a multitude of different tests for checking various things. For example, when writing a library that dealt with JWT and called into some other libraries, I opted for a lot of unit tests, I think the code coverage was around 90-95%. Writing tests for almost all of the functionality helped me actually discover one or two bugs which I could immediately fix without having them be released. Also, tests acted as documentation there, showing various use cases for the library, in addition to helping me actually make the API more user friendly. Integration and UI tests are also useful to see how the solution fares in the real world, but once you have to automate a browser or deal with mocking the DB (if you can't spin up a container, migrate the schema and seed it with real data for your tests), the complexity skyrockets. You might still want to have them and plenty of them, but at the same time you'll deal with slow and flaky tests more often, especially if the tech you choose or are stuck with won't support your needs all that well (e.g. Selenium). It's even worse if the thing you're testing changes a lot. As long as you have a staging environment, you should be even well off for anything from load testing to even the occasional bit of manual/acceptance testing, or trying to reproduce issues there. With the help of logs/metrics you can reason about changes/bugs/reproduction scenarios more safely, without impact on prod. Even better with other supporting things, like feature flags. Test whatever you can: sometimes the limiting factor will be time, other times the complexity of the system (or how testable it is in the first place, e.g. no fixed element IDs in a web browser), other times you won't get the infrastructure, but it's still a nice pursuit.

xnx 3y ago

Some good guidelines just posted today: https://web.dev/ta-what-to-test/ https://web.dev/ta-what-to-test/