19 ms·

The day I started believing in unit tests

• teeray 3y ago
  I still like one of the defining characteristics of Unit Tests (paraphrasing Michael Feathers from memory): they are fast and cheap to run. Sure, they might not perfectly simulate production like integration tests, but they also don’t take hours burning cash in cloud infrastructure while risking failure from unrelated races dealing with those dependencies. You can use Unit Tests to get to a place where you’re fairly confident that the integration tests will pass (making that whole expensive affair cheaper to run).
  • Cthulhu_ 3y ago
    That's exactly it; QA is a layered / tiered / pyramid shaped process, the more you catch lower down, the less reliance there is on the upper layers, and the faster the development iterations.
  • gbacon 3y ago
    From Working Effectively With Legacy Code by Feathers, p. 14[0]: Unit tests run fast. If they don’t run fast, they aren’t unit tests. Other kinds of tests often masquerade as unit tests. A test is not a unit test if: 1. It talks to a database. 2. It communicates across a network. 3. It touches the file system. 4. You have to do special things to your environment (such as editing configuration files) to run it. Tests that do these things aren’t bad. Often they are worth writing, and you generally will write them in unit test harnesses. However, it is important to be able to separate them from true unit tests so that you can keep a set of tests that you can run fast whenever you make changes. [0]: https://www.google.com/books/edition/Working_Effectively_with_Legacy_Code/fB6s_Z6g0gIC?hl=en&gbpv=1&pg=PA14 https://www.google.com/books/edition/Working_Effectively_wit...
    • brightball 3y ago
      That's a great rule of thumb.
    • atticora 3y ago
      My "unit tests" do hit the database and file system, and I have found and fixed many many problems during testing by doing so. I have found many other problems with those calls in production when I didn't do so. Yes, they make testing a lot slower. Our main app takes around 40 minutes to build which isn't good. I'd like it to be faster. But writing a bunch of separate integration tests to cover those functions would be a steep price. I can understand reasonable people choosing either approach.
      • berkes 3y ago
        > My "unit tests" do hit the database and file system, and I have found and fixed many many problems during testing by doing so. I have found many other problems with those calls in production when I didn't do so. No-one said that integration tests can't also be very valuable. From the little context I get that you write integration tests, and that is fine. They are useful, valuable! But they are not unit-tests. edit: on re-reading, I get the feeling that for you "integration tests" are a synonym for "end to end tests". But -at least in most literature- end-to-end tests are a kind of integration-test. But not all integration tests are end-to-end tests. In my software, I'll often have integration tests that swap out some adapter (e.g. the postgres-users-repository, for the memory-users-repository, or fake-users-repository. Or the test-payment for the stripe-payment) but that still test a lot of stuff stacked on top of each-other. Integration tests, just not integration tests that test the entire integration.
        • lokar 3y ago
          And integration tests can also be fast
          • alemanek 3y ago
            Test containers really help with this. Should still have the big system tests that run overnight but a set of integration tests using Test Containers to stand in for the infrastructure dependencies is awesome. My team has a ton of those and they run inside a reasonable time frame (5min or so) but we still allow for excluding those from test runs so you can run just the unit tests.
    • deleted 3y ago
      [deleted]
    • randomdata 3y ago
      In other words, unit tests – unless, perhaps, you count an empty test as being a unit test – do not exist. "Talking" to structured sets of data, i.e. a database, is fundamental to computing.
    • jcranmer 3y ago
      I'd actually quibble a lot on this definition--if you want to unit test code that needs to do any of the first three things, well, you have to do them to test the code. I would say that a test for a network protocol that works by spinning up an echo server on a random port and having the code connect to that echo server is still a unit test for that network protocol. In my definition, it would still be a unit test if it is fast and it talks to a database, a network server, or the file system, so long as such communication is done entirely in a "mock" fashion (it only does such communication as the test sets up, and it's done in such a fashion that tests can be run in parallel with no issues).
    • dllthomas 3y ago
      I've never liked the conflating of target size and test time constraints. I very much agree that there's benefit to considering the pace of feedback and where it falls in your workflow; immediate feedback is hugely valuable, but it can come from unit tests, other tests, or things which are not tests. Meanwhile, some tests of a single unit might have to take a long time. Exhaustive tests are rarely applicable, but when they are it's going to be for something small and it's likely to be slow to run. That should not be in your tightest loop, but it is probably clearer to evict it simply because it is slow, rather than because it is not a unit test for being slow.
    • closeparen 3y ago
      It is pretty excruciating (and IMO useless) to write true, DB-isolated unit tests for DB access layer code.
    • physicles 3y ago
      I haven't found the distinction between unit tests and non-unit tests to be that useful in practice. The important questions are: 1. Is it kinda slow? (The test suite for a single module should run in a few seconds; a large monorepo should finish in under a minute) 2. Is there network access involved? (Could the test randomly fail?) 3. Do I need to set up anything special, like a database? (How easy is it for a new developer to run it?) If the answer to any of those is Yes, then your test might fall in the liminal space between unit tests and integration tests -- they're unit-level tests, but they're more expensive to run. For example, data access layer tests that run against an actual database. On the other hand, even if a test touches the filesystem, then it's generally fast enough that you don't have to worry about it (and you did make the test self-cleaning, right?) -- calling that test "not a unit test" doesn't help you. Likewise, if the database you're touching is sqlite, then that still leaves you with No's to the three questions above.
    • _dain_ 3y ago
      >1. It talks to a database. >3. It touches the file system. These are BS. Maybe they made sense in the beforetimes when when we didn't have Docker containers or SSDs, but nowadays there's no reason you can't stand up a mini test database as part of your unit test suite. It's way simpler than mocking.
      • icedchai 3y ago
        100% this. A guy I work with just rebuilt our CI/CD pipeline and we're spinning up a database and all dependent services in containers. There are no mocks and it works great. In previous lives, I worked on tests that mocked everything. We spent more time creating and maintaining mocks than writing the actual tests.
        • nerdponx 3y ago
          I think those are still considered "integration" tests in the traditional way of thinking. The problem is that a lot of applications don't do much other than interact with external resources, to the point where isolated unit tests are rare or only cover non-critical code, while just about any substantive test is an "integration" test.
          • icedchai 3y ago
            Yes, you are correct. I never cared much for the traditional way of thinking about tests. I've had more arguments with people claiming this is an integration test, not a unit test, write some mocks, etc. Outside of very specific pieces of code, you generally get more value from integration tests.
    • munksbeer 3y ago
      I've dropped my archaic thinking on what constitutes a unit test or an integration test. I now seldom write what most people consider unit tests, in the "one class, one test" sense. Instead, I classify units of logical coherence and write unit tests for those. I write financial trading systems - not hft, but still latency sensitive. I will test an order through our pipeline as a unit of work. This will necessarily touch multiple classes. So as an example, a test will cover orders which are accepted and then filled, orders which are rejected, and so on. Many people would classify these as integration tests, and to be fair, I don't really care what you name them. To me these are much more valuable than the traditional "one class, one test" mechanism because it means I am free to refactor the internals of our pipeline as much as I want with very low impact on the test code. One of the whole points of test code, that I think has been lost, is that it should be there to give you confidence in the correctness of your application under change. Writing "one class, one test" is a bad way to achieve this.
  • BeetleB 3y ago
    > they are fast and cheap to run. But expensive to write. Especially if you want them to be fast and cheap to run.
  • AtlasBarfed 3y ago
    THey are fast and cheap WHEN YOU FIRST WRITE THE CODE. Or, if you are the original author of the code. The problem is, and there will be people that disagree with me, is that unit tests make refactoring of other people's code a lot harder. STAY WITH ME! If the unit tests were "good" and helped document what the code does, then they don't. You won't believe this, but in dogmatic high-breadth-coverage (low depth coverage), there are tons of test code that is SO TIED TO IMPLEMENTATION rather than interface than any monkeying of the presumed encapsulated logic breaks the unit tests, so you have double the things to fix. You'll never believe what happens next. Some developer in some Agile thing that got assigned 2 unicorn shits for the task panics because the unit tests are SERIOUSLY slowing down his "velocity". So what does he do? Delete tests, change tests to make them work at any costs.
• bedobi 3y ago
  unfortunately legitimate use cases for unit tests (like this) are pretty rare in corporate codebases, overwhelmingly, unit tests are just mocked tests that enforce a certain implementation at the class or even individual method/function level and pretend that it works, making it impossible to refactor anything or even fix bugs without breaking tests such tests are not just useless, they're positively harmful https://gist.github.com/androidfred/501d276c7dc26a5db09e893bb7c9f3ee https://gist.github.com/androidfred/501d276c7dc26a5db09e893b...
  • bobthepanda 3y ago
    The point of unit tests is not to CYA during refactors, but to confirm that the implementation is consistent between small changes without weird side effects. A coworker once thought unit tests were dumb, and ended up writing code that repeated the call to an application 10x for the same info. This didn’t result in a changed UI because it was a read, but it’s not good to just suddenly 10x your reads for no good reason. TFA also describes discovering weird side effect race conditions as a result of unit tests.
    • domano 3y ago
      Using mockserver etc. you can cover for these things in component-test cases even more easily through your whole application while being more flexible with bigger code changes than unit tests allow.
      • bobthepanda 3y ago
        A lot of places on the internet treat component testing and unit testing as synonyms. I’ve never heard of the former and it basically sounds like the unit tests as we write.
    • bedobi 3y ago
      > confirm that the implementation is consistent between small changes without weird side effects not sure what this is referring to, but I'll give an example say you have a requirement that says if you call POST /user with a non-existing user, a user should be created and you should get a 2xx response with some basic details back you could test this by actually hitting the endpoint with randomly generated user data known to not already exist, check that you get the expected 2xx response in the expected format, and then use the user id you got back to call the GET /user/userId endpoint and check that it's the same user that was just created this is a great test! it enforces actual business logic while still allowing you to change literally everything about the implementation - you could change the codebase from Java Spring Boot to Python Flask if you wanted to, you could change the persistence tech from MySQL to MariaDB or Redis etc etc - the test would still pass when the endpoint behaves as expected and fail when it doesn't, and it's a single test that is cheap to write, maintain and run OR you could write dozens of the typical corporate style unit test i'm referring to, where you create instances of each individual layer class, mocks of every class it interacts with, mocked database calls etc etc which 1) literally enforce every single aspect of the implementation, so now you can't change anything without breaking the tests 2) pretend that things work when they actually don't (eg, it could be that the CreateUserDAO actually breaks because someone stuffs up a db call, but guess what, the CreateUserResource and CreateUserService unit tests will still pass, because they just pretend (through mocks) that CreateUserDao.createUser returns a created user
• bicijay 3y ago
  We can't even have a consensus on what "unit" tests really are... Every company i work for has a different meaning for it. Some places consider a test "unit" when all the dependencies are mocked, some places consider a whole feature a "unit".
  • bedobi 3y ago
    Kent Beck (the originator of test-driven development) defines a unit test as "a test that runs in isolation from other tests". This is very different from the popular and completely misguided definition of a unit test as "a test that tests a class/method in isolation from other classes/methods". But it doesn't really matter if you want to call a given test an "integration" test or a "unit" test. The point of any test is to fail when something breaks and pass when something works, even if the implementation is changed. If it does the opposite in either of those cases, it's not a good test.
    • randomdata 3y ago
      > The point of any test is to fail when something breaks and pass when something works The point of any test is to document API expectations for future developers (which may include you). That the documentation happens to be self-validating is merely a nice side effect.
    • troupo 3y ago
      Kent Beck also said [0] "I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence" [0] https://stackoverflow.com/a/153565 https://stackoverflow.com/a/153565
  • mdgrech23 3y ago
    I get the logic from the mocking camp e.g. we're not here to test this dependency we're just here to test this function/method whatever but when you mock you end up making assumptions about how that dependency works. This is how you end up w/ the case of all my tests are green and production is broke. I think it's hard to beat e2e testing. The thing is e2e tests are expensive to write and maintain and in my opinion you really need a software engineer to write them and write them well. Now manual e2e testing is cheap and can be outsourced. All the companies I've worked for in the US have had testing departments and they did manage to write a few tests but they were developers and so to be frank they were really bad at writing them. They did probably 80 or 90% of their testing manually. At that point who we kidding. Just say you do manual testing, pay your people accordingly and move on.
• the_sleaze9 3y ago
  Good story. I for one do not believe in Unit Tests and try to get LLM tooling to write them for me as much as possible. Integration Tests however, (which I would argue is what this story is actually praising) are _critical components of professional software. Cypress has been my constant companion and better half these last few years.
  • randomdata 3y ago
    In reality, unit tests and integration tests are different names for the same thing. All attempts at post facto differentiation fall flat. For example, the first result on Google states that a unit test calls one function, while an integration test may call a set of functions. But as soon as you have a function that has side effects, then it will be necessary to call other functions to observe the change in state. There is nothing communicated by calling this an integration test rather than a unit test. The intent of the test is identical.
    • cjfd 3y ago
      No. Or maybe only if you also consider 'village' and 'city' to be the same thing.
      • WendyTheWillow 3y ago
        Clusters of humans cohabiting a confined space? If you squint hard enough…
      • __MatrixMan__ 3y ago
        There are only two kinds of tests: ones you need and ones you don't. Splitting hairs over names of types of tests is only useful if you're trying to pad a resume.
      • randomdata 3y ago
        Implying that integration tests (or vice versa) are legally incorporated like cities, while unit tests are not? What value is there in recognizing a test as a legal entity? Does the, assuming US, legal system even allow incorporation of code? Frankly, I don't think your comparison works.
• EZ-E 3y ago
  I barely ever have unit tests flagging real issues. It's always a chore to update them. Feature/end to end tests though... Plenty of real issue flagged.
  • toenail 3y ago
    > I barely ever have unit tests flagging real issues That sounds like you work alone and haven't worked for a long time on a code base with unit tests. Or the unit tests are bad.
    • domano 3y ago
      Dont take this the wrong way, but this is the answer i would get from enterprise devs usually when pointing this out. Then i would realize that their definition of a real issue was completely removed from any business or user impact, but geared more towards their understanding of the process detail in question. I would argue that there certainly are some good places for unit tests, like if you have some domain-driven design going and can have well defined unit-tests for your business logic, but this usually is the smallest part of the codebase. Mocking things that talk to databases etc. usually gives a false sense of security while that thing could break for a whole number of reasons in the real world. So just dropping the mock here and testing the whole stack of the application can really do wonders here in my experience.
      • bebop 3y ago
        Not disagreeing with your points. One thing mocks can be good at is to simulate errors that would be difficult to reproduce in an actual stack. For example, maybe you want to try and handle transient network issues or db connection failures, a mock could throw the correct exception easily, making your full stack do this would be challenging.
      • toenail 3y ago
        > this is the answer i would get from enterprise devs usually when pointing this out Yes, exactly what I thought, that's what you would hear from somebody who has experience working on large code bases with many contributors.
• baz00 3y ago
  I believe in them. But unit tests are useless around useless humans. And there’s lots of those. A fine example is that time I wrote a test suite for a domain specific language parser. Someone wanted to break the language so they deleted the tests. New stuff was added without tests. They confidently broke everything historically and looking forward. Then blamed it on me because it was my test suite that didn’t catch it. The language should not have been broken. Everything only works if you understand what you are doing so every argument should be posed as both sides.
  • r2_pilot 3y ago
    In my opinion this is because we don't teach Chesterton's Fence early enough (or often enough) to internalize it at a societal level.
  • Steltek 3y ago
    Whenever I read, "when my code breaks the tests, I delete the tests", this is what I picture in my head. Their code changed behavior and good unit tests catch change in behavior. Someone somewhere is probably depending on that behavior.
• onetimeuse92304 3y ago
  I don't believe in unit tests as they are practiced. This, unfortunately, is the kind of thing that can work in principle, but the realities make it unusable. There are multiple problems with unit tests, as they are implemented in the industry. And to make the unit tests usable and productive you need to make them so productive that it can offset those problems. First of all, for unit tests to work everybody has to contribute quality unit tests. One team member writing unit tests well for his part of functionality is not going to move the needle -- everybody has to do this. Unfortunately, it is rarely the case that all team members are able to write quality code this is the case for unit tests. Usually, the reality is that given deadlines and scope, some developers will deprioritize focusing on writing good unit tests to instead deliver what business people do really care about -- functionality. Give it enough time and unit tests can no longer be trusted to perform its job. Second, it is my opinion that refactoring is extremely important. Being able to take some imperfect code from somebody else and improve it should be an important tool in preventing code rot. Unfortunately, unit tests tend to calcify existing code making it more expensive to change the functionality. Yes, more, not less expensive. To move a lot of stuff around, change APIs, etc. you will usually invalidate all of the unit tests that work around this code. And fixing those unit tests in my experience takes more effort than refactoring the code itself. Unit tests are good for catching errors AFTER you have made the error. But my personal workflow is to prevent the errors in the first place. This means reading the code diligently, understanding what it does, figuring out how to refactor code without breaking it. Over the years I invested a lot of effort into this ability to the point where I am not scared to edit large swaths of code without ever running it, and then have everything work correctly on the first try. Unit tests are usually standing in the way. I think where unit tests shine is small library code, utilities, where things are not really supposed to change much. But on the other hand, if they are not really supposed to change much there also isn't much need to have unit tests... The most paradoxical thing about unit tests is that teams that can write unit tests well can usually produce code of good enough quality that they have relatively little use of unit tests in the first place. What I do instead of unit tests? I do unit tests. Yes, you read that correctly. The trouble with unit tests is that everybody gets the part of what unit is wrong. Unit does not have to mean "a class". Units can be modules or even whole services. What I do is I test a functionality that matters to the client -- things I would have to renegotiate with the client anyway if I was to ever change it. These tests make sense because once they are written -- they do not need to change even as the functionality behind them is being completely rewritten. These test for what clients really care about and for this they bring a lot of bang for the buck.
• deleted 3y ago
  [deleted]
• domano 3y ago
  So at work we would run tons of tests against the real service with a real database, seeding thousands of schemas to allow for parallel testing of tests that change state. This takes 3 minutes, 1 if you use tmpfs. It only takes <10 seconds if you dont run writing tests. These actually cover most real world use cases for a query-engine we maintain. Unit tests have their place for pieces of code that run based on a well defined spec, but all in all this integration or component-level testing is really what brings me the most value always.
  • RaftPeople 3y ago
    From research I've read, unit tests (whether automated or not) tend to catch around 30% of bugs whereas end to end testing and manual code review (believe it or not) each tend to catch around 80% of bugs.
• donatj 3y ago
  I was ambivalent on unit tests until I discovered how much the mere act of writing them was finding bugs. I very vividly remember writing a test for a ~40 loc class of pure functions. I started out thinking the exercise was a waste of time. This class is simple, has no mutable state, and should have no reason to change. Why bother testing it? By the time I was done writing the test I had found three major bugs in that 40 loc, and it was a major aha moment. Truly enlightening.
  • linsomniac 3y ago
    That reminds me of this time I wrote some code to add a method to Python string objects. The first reply to my issue on it in the bug tracker was "We shouldn't accept this, it's trivial to implement in your own code, see: XXXX". The second reply was "You have a bug in your implementation in the first reply." It took a couple years to be accepted.
    • nerdponx 3y ago
      Sounds familiar. Was that str.removeprefix?
      • linsomniac 3y ago
        str.rsplit()
  • jamesu 3y ago
    I bumped into so many corner case and dumb bugs on a recent python project that I'm even more of a unit testing enthusiast than before. Past a certain level of complexity they are definitely a net benefit.
    • throwaway2037 3y ago
      You mentioned Python. I struggle with the weak(er) typing. It is a bottomless well of bugs. Did your unit tests find type issues or (business) logic / state issues?
• davnicwil 3y ago
  I think about unit tests being useful for getting more confidence that some deterministic, pure (mathematically speaking) and stateless piece of code that's data in data out actually works, particularly when you change it. If any of those conditions doesn't hold the cost/benefit certainly and even sometimes the absolute utility goes way down. If I have to mock anything, in particular, or more generally care at all about any implementation details (ie side effects) then I just think might as well make this a full on automated functional test then. As soon as fake code is introduced into the test its utility rapidly decays in time as the things it fakes themselves change.
  • corey 3y ago
    I agree that mocks are brittle and nearly useless. If you follow SOLID principles to the extreme, you'll find that your code is separated into logic code that is pure and easy to unit test, and IO code that is very simple and can be tested by a relatively few number of integration tests.
    • aleksiy123 3y ago
      To some extent this is pretty much the same as mocking. You are still injecting fake data into your pure logic functions whether its through their parameters or by them calling a mock. I agree preferable but sometimes you want to test the logic of the code thats actually making decisions about how and when the IO is called. You can do it with integration tests of course but in more complex environments with lots of complex IO dependencies mocking is cheaper. Its also hard to simulate specific failures in integration tests like a specific request failing. Pretty much mocking with extra steps. So mocking has its place as well.
• corndoge 3y ago
  I started believing in unit tests the day I finished my patch, ran the program and watched it work perfectly. I then grudgingly wrote a test, ran it and immediately observed it fail. One of the test inputs was some garbage input and that exposed a poorly written error handling path. Humbling! I still hate writing them and it grates on my aesthetic sense to structure code with consideration to making it testable, but if we want to call ourselves engineers we need to hold ourselves to engineering standards. Bridge builders do not get to skip tests.
  • cfiggers 3y ago
    > if we want to call ourselves engineers we need to hold ourselves to engineering standards. Bridge builders do not get to skip tests. Bravo. We need more of this mindset in the world, and also more collective will to encourage it in one another. YOU are the kind of engineer I want writing the code that goes in my Dad's pacemaker or the cruise control in my wife's car.
    • m3kw9 3y ago
      If you have worked in places where safety is critical, you wouldn’t say something so shallow. In those places they place human verification above all else. They have a thick book where you do a full run and is double checked, they don’t f around with unit tests and say this is good to go
      • ska 3y ago
        I don't think anyone is saying unit tests and you are good to go are they? In any critical system work, there are multiple layers and you can't really skip any of them. It's also sort of meaningless to talk about such testing without requirements and spec to test against. Traceability is as much a part of it as any of the testing. By the time you get to the "thick book/full run" as you put it, there has typically been a metric crapload of testing done already.
      • Ancapistani 3y ago
        Human verification is very expensive, compared to unit tests. It costs money to pay that human to do it, time for them to test it, time to describe issues found, time to send it back for a fix. Unit tests - actually, all automated tests - are comparatively cheap. The developer can run them immediately. All code will have bugs. The "trick" to building a productive development pipeline is to catch as many of those bugs as possible as early as possible, and thereby reduce both the temporal and monetary cost of resolving them.
• test77777g 3y ago
  [dead]
• yitchelle 3y ago
  The gem of this story is the author is not running unit test in what most folks understand a unit test is. As he also pointed out, he is executing the tests on target so it is more of an integration tests rather than unit tests. In the test that he is doing, it brings in new categories of potential faults. ie scheduling issues, memory constraints, interrupts servicing,
• agentultra 3y ago
  It's all degrees. Unit tests are great at finding examples of errors or correct behaviours. However they prove nothing and they definitely do not demonstrate the absence of errors. They are often sufficient for a great deal of projects. If all it takes to convince you it's "good enough," are a handful of examples then that's it. As much as you need and no less. However I find we programmers tend to be a dogmatic bunch and many of us out there like to cling to our favoured practices and tools. Unit tests aren't the only testing method. Integration tests are fine. Some times testing is not sufficient: you need proof. Static types are great but fast-and-loose reasoning is also useful and so you still need a few tests. What's important is that we sit down to think about specifying what it means for our programs to be, "correct." Because when someone asks, "is it correct?" You need to as, "with respect to what?" If all you have are some hastily written notes from a bunch of meetings and long-lost whiteboard sessions... then you don't really have an answer. Any behaviour is, "correct," if you haven't specified what it should be.
  • pfdietz 3y ago
    > they prove nothing If they fail, they prove there's a bug (in either the test or the code.) This is like literally any other kind of test.
    • erikpukinskis 3y ago
      A bug in test code is not a real bug. It’s just a test that’s not giving you useful information. Lots of tests don’t give you useful information. Some that fail and some that pass. It’s easy to write a test that doesn’t provide useful information across time. Harder to write a test that does.
      • baq 3y ago
        There are times for constructive advice and there are times when '...so don't do that' is the right answer. > It’s easy to write a test that doesn’t provide useful information across time. I firmly believe this is one of those times. (Currently my only issue with tests in the product I work on is that they take too long to run. Can't have it all.)
• ejb999 3y ago
  I hate unit tests, though I am forced to write them to have my CI process not fail (I need 75% coverage or it won't build) - so I have written thousands and thousands of them in the last few years - the problem I have: not a single time that I had a unit test fail that resulted in me finding a bug in my code - all I ever find are bugs in my unit test code - so pretty much seems like a waste of time to me. Either I am writing really good code so there are no bugs, or I am really bad a writing unit testing code to find those bugs.
  • rileymat2 3y ago
    A sort of non-judgmental question, in your mind are you writing them to cover lines or exercise required behavior with an intent of proving the module is broken? I ask because it seems like requiring line coverage as a metric would have the effect you are describing.
    • mrweasel 3y ago
      I've seen the same thing with comments. My boss required us to add comments to our code, to make it easier to read. That was all he asked, please add comments. My co-worker added comments like "Increase variable i by 1", while completely ignoring the 8 lines of spaghetti business logic above. Similarly I've seen people add tests that will ensure that code coverage doesn't go down, but it doesn't actually do anything to help anyone. I'd argue that the issue is that have random coverage goals is a problem on its own, but it's the only way to force some people to write even to most basic of tests.
      • pixl97 3y ago
        Ah, Goodheart's law ruins everything.
      • Izkata 3y ago
        I've thought for a long time we present coverage backwards. We shouldn't be highlighting what is covered and getting that metric up, we should highlight what isn't covered and focus on getting that metric down (like how linting is done). Present it like "Hey, here's something that no one has looked at in-depth! It's a great place for bugs to be hiding!"
• BurningFrog 3y ago
  The classic test rookie mindset is to test the functionality of the whole system, because that's what really matters. But in reality, unit testing every single function and method is where the vast majority of the benefit lies. Details really matter. It took me some time to learn this, even after being told. It's the same for most people. This little post will probably convince no one. But maybe remember it when you finally get there yourself :)
  • Izkata 3y ago
    > every single function and method Very much no, that's the bad kind of unit test that locks your code into a specific structure and makes it a pain to update because you also have to change all the related tests even if the actual interface used by the rest of the codebase didn't change. I would call this the rookie mistake of someone new to unit tests. You want to encapsulate your code with some sort of interface that matches the problem space, then test to that interface. How its internals are broken down don't matter: it could be one big function, it could be a dozen functions, it could be a class, it as long as the inputs and outputs match what the test is looking for you can refactor and add/remove features without having to spend extra time changing the tests. Makes it much less of a pain to work with in general. One way of looking at it I've used before with coworkers: For this new feature you're writing, imagine a library for it already exists. What is the simplest and most straightforward way to use that library? That's your interface, the thing you expose to the world and what you run your tests against. This is what unit testing originally meant: semantic units, not code units. It's like app Hungarian notation vs system Hungarian notation, the original idea got overtaken by people who didn't understand the idea and only mimicked the surface level appearance.
  • troupo 3y ago
    > But in reality, unit testing every single function and method is where the vast majority of the benefit lies. Details really matter. To me, this is actual rookie mentality. You end up testing the same thing multiple times over different lines of code, mocking and providing various sets of testing data... When you could just test specified and/or observable behaviour of your system, and achieve the exactly same result with fewer tests.
• mrweasel 3y ago
  It is a little sad to see so many be so dismissive of unit tests. They aren't a universal solution, which seems to be why they are written off in many cases, but they make your life so much easier in so many cases. If you need to mock out 80% of a system to make your unit test work, then yes, it's potentially pointless. In that case I'd argue that you should consider rewriting the code so that it's more testable in isolation, that will also help you debug more easily. What I like to do is write tests for anything that's just remotely complex, because it make writing the actual code easier. I can continuously find mistakes by just typing "tox" (or whatever tool you use). Or perhaps the thing I'm trying to write functionality for is buried fairly deep in an application, then it's nice to be reasonably sure about the functionality before testing it in the UI. Unit tests just makes the feedback loop much shorter. Unlike others I'd argue that MOST projects are suited for unit testing, but there might be some edge cases where they'd provide no value at all. On caveat is that some developers write pretty nasty unit tests. Their production code is nice and readable, but then they just went nuts in the unit tests and created a horrible unmaintainable mess, I don't get why you'd do that.
  • 6DM 3y ago
    I have tried to evangelize unit testing at each company I've worked at and most engineers struggle with two things. The first is getting over the hurdle of trusting that a unit test is good enough, a lot of them only trust an end-to-end test which are usually very brittle. The second reason is, I think, a lot of them don't know how to systematically breakdown test into pieces to validate e.g. I'll do a test for null, then a separate test for something else _assuming_ not null because I've already written a test for that. The best way I've been able to get buy-in for unit testing is giving a crash course on a new structure that has a test suite per function under test. This allows for a much lower loc per test that's much easier to understand. When they're ready I'll give tips on how to get the most of their tests with things like, boundary value analysis, better mocking, IoC for things like date time, etc.
    • feoren 3y ago
      > I'll do a test for null, then a separate test for something else _assuming_ not null because I've already written a test for that. Honestly, this pedantry around "unit tests must only test one thing" is counter-productive. Just test as many things as you can at once; it's fine. Most tests should not be failing. Yes, it's slightly less annoying to get 2 failed tests instead of 1 fail that you fix and then another fail from that same test. But it's way more annoying to have to duplicate entire test setups to have one that checks null and another that checks even numbers and another that checks odd numbers and another that checks near-overflow numbers, etc. The latter will result in people resting writing unit tests at all, which is exactly what you've found. If people are resisting writing unit tests, make writing unit tests easier. Those silly rules do the opposite.
• smrtinsert 3y ago
  Said it before and will say it again. There is no replacement for unit test - it is the only thing that will give you flawless deployments. Not MIT degrees, not process, not managers - tests are the literally the only thing I've seen consistently produce flawless production deployments. It's not a discussion.
• m3kw9 3y ago
  Unit tests is like buying insurance but you don’t know how much insurance has paid you if things go wrong. You spend a lot of time and effort to make your code testable, figure out what the useful test is and change the unit test when you do refactors in hope it speeds up your project, except you cannot really know if there was a net gain in speed/ reliability vs proper QA and other techniques
• dn3500 3y ago
  I think he should have credited Tom Van Vleck with the "three questions" idea. It was published in ACM SIGSOFT Software Engineering Notes, vol 14 no 5 July 1989, pages 62-63, and you can read the whole thing here: https://multicians.org/thvv/threeq.html https://multicians.org/thvv/threeq.html I hope he got permission to reproduce the comic.
• m3kw9 3y ago
  How do you all feel about the need to rewrite a unit test when code gets refactored or business logic changes, isn’t that like a huge pita?
  • macshome 3y ago
    Generally refactoring is where I find tests to be super valuable. If it’s a pure refactor then the existing tests shouldn’t break. If they start failing, then you have done something that has changed the expected behavior. For business logic I would change the tests first so that it represents the new expected result. Then you refactor the code until the tests pass.
  • kuchenbecker 3y ago
    If you're testing implementation details rather than contracts, you're susceptible to this. Make sure the unit yutare testing is the thing you want to observe the behavior of.
  • coldbrewed 3y ago
    I treat unit tests like double-entry bookkeeping; I wouldn't describe it as a particular pain and consider it more of a matter of due diligence. Not everything needs this level of rigor but there are plenty of cases where the tests are very cheap to write and reason about (for many pure functions) or are worth the cost as they validate critical behavior. Unit tests also add some design pressure to keep more logic pure/side-effect free; sure, it may take a bit more work to factor your code accordingly to keep i/o interactions separated to the shell of the application but I find this to be a useful pressure. I've found that if I'm encountering pain when writing unit tests, then the pain is due to one of the following things: 1. The code is growing too complex and I need to decompose the logic or refactor the tests 2. The code has grown too many unintentional side effects and I need to move those side effects to discrete components 3. The code under test has fundamental side effects and those side effects require testing, thus the unit tests need to be converted to an integration test 4. The code under test is sufficiently complex that it demands full system/acceptance testing There are some cases where refactoring the tests is generally too painful and I'll throw away all the tests entirely, maybe sprinkle in a few tests for logic that seems critical, and move on. Tests can accumulate technical debt, but in contrast to implementing code it's pretty cheap to cut your losses on tests and wipe them out. I see a lot of people conflating unit testing with the idea that all code must have tests, and there's a ton of code that's phenomenally painful to test and can be easily checked by the developer. Tests should be a supporting tool an an augment to the developer practices; it's better to have some tests that work well and throw out the ones that are miserable to write rather than require 95% test coverage, drown in testing, and throw out all tests entirely.
• hax0ron3 3y ago
  Unit tests are not even well defined. What is a unit?
  • Ancapistani 3y ago
    Something needn't be well-defined to be valuable. :) If it helps, think of "unit tests" and "atomic tests". Your goal in writing a unit test is to test the smallest possible amount of logic at a time, with the least possible overhead (i.e., mocking). The advantages of this approach are many: it helps keep the level of complexity of individual methods low enough to be quickly understandable, documents the interface provided by your methods, ensures that the tests run quickly, and allows new tests to be written with minimal effort. Obviously there are disadvantages, too. Unit tests - any tests - take time to write. This is sometimes offset by the time saved by catching issues as early in the development cycle as possible, but not always. For "greenfield" projects especially, I tend to take a different approach than in my other work. For those, I start by "writing the README". It doesn't matter if it's an actual README.md; the point is to write down some examples showing how you think the new functionality should be used. Once that's done, I'll stub out an implementation of that, then refining it with increasing granularity until the overall architecture of the project begins to be defined. Sometimes, that architecture is complex enough that it's worthwhile to break it into smaller pieces and start the process over for those. Other times, I get to a working "happy path" pretty quickly. Once I have a minimally working feature, I write tests for the public-facing interface. Then the interfaces between domains inside the project. Then unit tests for individual methods. I mostly work in Python, so this is also the point where I pause and apply type annotations, write/expand my docstrings, ensure that my `__all__` objects are set properly, make sure any "internal use" methods of publicly exported types are prefixed with `_`, etc. On the other hand, when I'm writing a feature or making a change to a more mature codebase, I often _start_ by writing tests. Sometimes that's a new interface that I'll be using elsewhere, so I'll write tests defining that. Sometimes it's a change in behavior on an existing implementation, so I'll write tests for that. Either way, from that point on I repeatedly run _only_ the new tests that I've written as I build out the feature. Only once the feature works and those tests pass do I re-run the whole test suite to check that I've not broken something I hadn't considered. When those pass, I'll go back over my code one more time to make sure that I've added tests for all of the relevant internal stuff before submitting the patch.
• drittich 3y ago
  I am troubled by the word belief, not just in the title, but in the comments here. Unit tests should not be doctrine, there is a time and a place. And, I feel that more often than not they are warranted. We can argue about what granularity they should be, talk about functional programming, debate whether they should hit the database or not, but IMO all of those things miss the point. For me, in order of priority, unit tests provide the following benefits: 1) Make me write better, more decoupled code 2) Serve as documentation as to the intent of the code, and provide some expected use cases 3) Validate the code works as expected, (especially when "refactoring", which is basically how I write all my code even from the start) 4) Help you when deleting code by exposing unexpected dependencies You can argue against all of those points, and I often will, myself. It depends on the scale, importance, and lifetime of the project as to whether I will write unit tests. But, as soon as I think someone else will work on the code, I will almost always provide unit tests. In that scenario, they: - Provide a way to quickly validate setup and installation was correct and the application functions - Signal that the code was "curated" in some way. Someone cared enough to setup the test environment and write some tests, and that gives me a certain comfort in proceeding to work on the code. - Provide a gateway into understanding why the application exists, and what some of the implementation details are. So, thinking about the advantages I've outlined above, for me it would be very hard to say I don't "believe" in unit tests. I just don't always use them.
• servaldeneptuno 3y ago
  I have an unrelated (and most likely dumb) question about the article. When they talk about the inheritance relationship between 'Thread' and 'MyThread' in the example code in reference to the destructor methods, particularly here: > Now, what happens when MyThread::singlepassThreadWork() uses a member variable of MyThread like foobar and we delete the MyThread object while the thread is still running? The destruction sequence is such that MyThread is deleted first and after that, the destructor of its parent object Thread runs and the thread is joined. Thus, there is a race condition: We risk accessing the vector foobar in singlepassThreadWork() after it was already deleted. We can fix the user code by explicitly stopping the thread in its destructor What does it mean when they say 'the destructor of its *parent* object Thread runs'? I've always thought that when you inherit from one class to another and then instantiate an object of said class, they're just one object, so what do they mean when they make the distinction between 'parent' and 'child' object? When you have inheritance of say two classes, those would be two distinct objects instantiated in memory? Is there something I'm missing?
  • OvbiousError 3y ago
    You're right, the wording is confusing. It should be "parent class". There is only one object, a MyThread object. In C++ when an object is destroyed, all the destructors in the hiearchy run, from bottom to top. So first ~MyThread and then ~Thread. Anyway I think it is odd design to stop the thread in the destructor. You'd normally stop the thread first and then destroy the object, not the other way around?
    • adrianmonk 3y ago
      They might be trying to encapsulate things so that they are sure threads get stopped when the objects go out of scope. But, I would probably do that by having a class that contains both the thread and the data that the thread needs to access. Then its destructor could first join the thread and then clean up the data. For example, instead of a WorkerThread that contains a vector of WorkItem, have a BackgroundWorker that contains a Thread and a vector of WorkItem.
    • servaldeneptuno 3y ago
• seanmcdirmid 3y ago
  Unit tests are great, but the way we often do unit tests is often as simple change detectors, which don’t say so much about correctness as much as they do about the code still doing what the programmer thinks it is doing (and tests often need to change if the code changes). It would be nice if unit tests were more like interlocking evidence of system correctness, but right now we just have integration tests with poorer coverage for that.
• JonChesterfield 3y ago
  Most software doesn't work the moment you stray from the expected path. Whether that's because most software isn't tested competently or because software testing practices don't deliver robust software is not yet clear. I suspect that unit tests, and tests in general, will be considered a historical artifact from the time before we worked out how to write software properly. For example, we don't generally unit test things that a static type system checks for us. Maybe good enough type systems will remove the rest of them.
  • appplication 3y ago
    I think it’s a little over optimistic to think that we will ever work out how to properly write software. Some new patterns may help, but we will always have a need for unit tests and other tests. Wrt typing, that’s a very narrow set of errors, and I would dare say even a small minority of the things that can and do go wrong in software are type related. That said, effective typing is another orthogonal tool to unit tests that can help create robust software. On that front, what we are missing is a language with robust typing that catches these type errors, but also gets out of developers way the rest of the time.
• pfdietz 3y ago
  Another thing unit tests are is focused. If you change just a small part of your code, you should only need to run a small fraction of your unit tests. Your unit test framework should support this selective execution of tests.
• corethree 3y ago
  >It is a little sad to see so many be so dismissive of unit tests. You're preaching to the choir. The overwhelming majority of people worship unit tests like dogma. There's almost no point in saying the above. It's like saying it's a little sad to see some people who are so dismissive about eating and breathing to stay alive. Your next part is the one that's interesting. Mocking 80 percent of a system to get unit tests to work. I've seen so much of this from developers who don't even realize the pointlessness of what theyre doing that it's nuts. They worship test so much that they can't see the nuance and the downside. Take this article. This article is literally presenting evidence for why unit tests are bad. He literally created an error that would not have existed in the first place we're it not for his tests. Yet he has to spin it in such a strange way to make it support the existing dogma of test test test.
  • jerrycruncher 3y ago
    Sitting on a call right now where a guy is going on about how excited he is to mock out the entirety of a large e-commerce vendor's platform. It's maddening.
• cloverich 3y ago
  I like pasting code into ChatGPT, then saying "Write unit test(s) that demonstrate the bug(s) in this code". I have pre-instructions that say "Show code only. Be concise" to keep it simple. This has resulted in many learnings for me.
• kreeben 3y ago
  I don't see how you can either believe or not believe, in a unit test. A unit test is what it is. It's a real thing. It exists. Use it, or don't. How this topic can sometimes be about belief is beyond me. It's like if a person found a screw driver and says, I now believe in screw drivers. The topic of how people believe in unit tests, to me is proof that the world is screwed. We're all screwed and everything is a screw driver.
  • chowells 3y ago
    I suppose this is sort of the complement of https://xkcd.com/169/ https://xkcd.com/169/ Pretending to misunderstand clear communication then making smug points about it isnt clever either. https://www.merriam-webster.com/dictionary/believe%20in https://www.merriam-webster.com/dictionary/believe%20in definition 2, "to have trust in the goodness or value of (something)". Words (and phrases) in English usually have more than one meaning. Ranting about correct use of a phrase because you're pretending the only extant meaning is a different one is not clever.
• TheAlchemist 3y ago
  Similar experience to this guy - didn't believe in them initially, but now I'm a believer. For what it's worth, I find Copilot to be quite an exceptional help in writing unit tests ! A real game changer for me. Not only it takes care on most boilerplate code, but also kind of 'guesses' what case I'm about to write - and sometimes even point me in a direction I would miss otherwise.
• alganet 3y ago
  Unit tests saved me many times. I'm happy that this article praises unit tests without forcing a TDD perspective to the reader. It presents it like a tool, not a religion, and that's very refreshing.
• pfdietz 3y ago
  To me an interesting distinction is not between unit and integration tests, but between tests that are run quickly as part of a gate on commits in CI, vs. tests that are run more asynchronously searching for bugs. The former must run quickly, and it's ok if the exact same test is run over and over. The latter need not run quickly, but benefits if new tests can be created and run, or if the tests incorporate randomness so they don't do the same thing each time they are run. Here, it seems he was using tests intended for the first purpose for the second purpose instead. That can work, as it did here, but I don't think it's optimal. Better to have more exploratory, randomized, property-based tests chugging away in the background to find weird new ways the code can fail.
• HankB99 3y ago
  I like them because they help me to partition my code into units that are easier to write and test. Once they're working, assembling the parts generally leads to a working project. It also motivates me to get small pieces working and tested before I get to the finish line. Each successful test is a victory!