9 ms·

The Big TDD Misunderstanding (2022)

63 points by WolfOliver 2y ago

brumar 2y ago
> Do not isolate code when you test it. > Only isolate your code from truly external services That makes tests more trustworthy but also sometimes harder to maintain I think. I have seen cases where small changes on the code base created strong ripple effects with many tests to update. Arguably, the tests were not very well written or organized and with too many high level tests. Still, this and the very large execution time of the test collection made me realized that for medium to large projects, I will be much more careful in the future before going all in with the no-mock approach.
- nobleach 2y ago
  Or, write code that _can_ work in isolation. (Pure functions). You can bet when GM builds fuel injectors, they have a harness that they can place them in to make sure that JUST the injector gets tested... putting it in a car and driving around is the wrong approach. The same is often true for software. Ensuring that things integrate is vital. Ensuring that functions run in isolation also has value.
  - battered8310 2y ago
    Absolutely, whenever there is hardware in the loop, the more of the system you test with the more expensive the test is and the harder it is to isolate issues. That’s not to say you can skip end to end testing, but the more testing pushed to the complete system level will drive up cost and schedule.
    - nobleach 2y ago
      To beat my example into the ground. If I were to put a fuel injector (system under test) into a vehicle, and drive it around a parking lot (E2E test), only to have the vehicle stall out. Is the fuel injector at fault? How would you prove that? No, you pull out the component and you "bench test" it. Believe me, after many MANY hours spend fighting bugs where the extensive E2E suite passed, we finally had to go back to adding in unit tests for the small units under the surface. Not all... but many places.
  - t0mas88 2y ago
    This makes sense when the cost of running the integration tests is high due to real hardware being in the loop. But for most business software this isn't the case and e2e tests aren't actually costly. Looking at a previous project we ran full e2e tests from an end user perspective in three major browsers in less than 15 minutes at a cloud cost of only a few hundred dollars per month. Compared to the cost of developers on the project that was a negligible amount. Adding more unit tests would make the feedback cycle shorter on some changes, but also add development time to create and maintain those tests. So in line with the article we did so only sparingly on a few critical paths that were likely to cause issues. The rest was well covered by e2e tests.
    - sarchertech 2y ago
      e2e tests that run in 15 minutes in the cloud are useless for TDD though which is what the article is talking about.
recursivedoubts 2y ago
Agree w/the author that the concept of "unit" often hurts test quality. You should be striving to balance the long-term usefulness of your tests with the debuggability of those tests. In my experience, those tests are what most people would call "integration tests" (although that name, like so much terminology in the testing world, is confusing and poorly defined.) You want to get the tests up at as high a level of abstraction as possible where the API and correctness assertions are likely to survive implementation detail changes (unlike many unit tests) while at the same time avoiding the opaque and difficult to debug errors that come with end-to-end testing (again, the language here is confusing, I assume you know what I mean.) See https://grugbrain.dev/#grug-on-testing https://grugbrain.dev/#grug-on-testing
nobleach 2y ago
Eh, I knew this would somehow come back to that Kent C Dodds stuff. I can respect the guy, but his opinion on valuable testing differs from mine and quite a lot of others. His opinion is "the more your tests represent the way your users use your app, the better they are". For E2E tests, yes, absolutely. For unit tests, no... not at all. As a very short example, an E2E test will prove that yes, I did indeed show a button on the screen. A unit test will prove the code that put it there worked. The code can "fail open" for many reasons. (In JavaScript for example, forgetting an `await` keyword can return a truthy value!). The worst part about it is that he called himself a thought leader, called his approach a "best practice" and had nothing really to back that up. Now people go around repeating it all the time. It's frustrating.
- 9rx 2y ago
  That’s not in line with the original definition of unit test. It also sounds like a pointless test using your definition. There is no advocacy for writing pointless tests.
  - nobleach 2y ago
    It's pointless if one is just trying to put "points on the board" and see a bunch of green in their console. Knowing what to test is important. Should every function be tested? Probably not. Should every function that has conditional logic be tested? Maybe? I totally get that integration tests have immense value. Where I disagree is that, "the closer your tests are to the way a user would use your app" is a good way to use testing. Functional/Acceptance/E2E tests are but one tool in an aresenal. Telling new software developers to ignore 25 years of history because React components were hard to test in isolation (Enzyme - the package Kent's React Test Library is built to replace) is irresponsible. Telling them that it's "best practice" is disingenuous.
- int_19h 2y ago
  If you have an E2E test that tests the button behavior thoroughly (i.e. not just showing it, but the fact that, when it's clicked, the app state changes accordingly), then you have a test showing that "code that put it there worked".
int_19h 2y ago
Simple rule of thumb. Always write functional tests first. Doesn't matter if they are slow - you still want something that faithfully captures the specified behavior and allows you to detect regressions automatically. Then, if your resulting test suite is too slow, add finer-grained tests in areas where the perf benefits of doing so dwarf the cost of necessary black-boxing. Getting down to the level of individual classes, never mind functions - i.e. the traditional "unit tests" - should be fairly rare in non-library code.
- jasonm23 2y ago
  aka the inverted test triangle.
- wink 2y ago
  I'm not trying to be confrontational, but even "business logic" inside a project is often clearly separated between "library" and "glue, aka controllers or whatever" - and that's kinda the point where I disagree. Not accepting "but the unit tests work" - 100% agreed. Not being able to slice it that the thing that could be accessed by the current public API or a different public API be tested on its own (and often relatively pure) is not how I want to write my software or tests.
  - int_19h 2y ago
    Thing is, trying to make everything individually testable in tiny bits takes a lot of effort that wouldn't otherwise be needed. Furthermore, with all the mocking this usually requires, you end up with tons of extraneous code that now needs to be maintained as well (and by very nature of mocks, you'll have to fix them every time the corresponding real code changes behavior observable across the boundary). In practice, the ostensible benefit of knowing exactly what the problem is as soon as you see the test failure is usually dwarfed by all these costs. And, at the same time, the more mocks you have, the less likely your tests are to catch real issues. Now, as you rightly note, in many cases there's already an obvious component boundary that is inherent in the design. And then it makes perfect sense to write functional tests for individual components along those boundaries. And in some cases, the component may be "library-like", and those tests will end up looking a lot like your typical unit test, where individual classes and even methods are covered one by one. But most classes and methods in a typical app aren't like that.
arialdomartini 2y ago
> see here and here I believe links are significantly more useful when they include descriptive text like the title or author, rather than just 'here'.
codr7 2y ago
I rarely write isolated unit tests. Most of the systems I build use a database on which all logic depends, and often a network connection. I've worked on systems where these aspects were mocked, and they eventually grind to a halt because of the effort required to make the tiniest change. First of all you need a way to create a pristine database from code, preferably in memory. Second nested nested transactions are nice, since you can simply rollback the outer transaction per test case; otherwise you need to drop/create the database which is slower. For networked servers, an easy way to start/stop servers in code and send requests to them is all you need. Given these pieces, it's easy to write integration tests that run fast enough and give a lot of bang for the buck. TDD is even more rare for me, I typically only do that when designing API's I'm unsure about, which makes imagining user code difficult. And fixing bugs, because it makes total sense to have a failing test to verify that you fixed it, and that it remains fixed.
- hitchstory 2y ago
  I still find the skepticism around TDD weird. Except for a few pretty niche scenarios (e.g. it's experimental code or manual testing is cheaper for some obscure reason) i dont really see the point of not doing it. I especially dont see what is gained by writing the test after.
  - codr7 2y ago
    Coding is not religion for me, I have no patience for fundamentalism. I will write as many tests as I need to feel confident, which depends on context. And integration tests give me a lot more confidence than mocked unit tests.
    - hitchstory 2y ago
      I hate coding fundamentalism with a passion too. The only thing I get really religious about in coding is the importance of trade offs. The cost/benefit of writing a test before just consistently exceeded doing it after for me. Same for integration, e2e or unit tests (there's never been a rule that says you can only TDD with a unit test). The cost/benefit trade off for tests with mocks vs. database is a different topic - orthogonal to the practise of red/green/refactor, and one where IMO the trade offs are much less obvious.
atum47 2y ago
My first job interview after college. It was a pair programming session but I would not touch the keyboard. I would dictate and the interviewer would type. I had to implement a dynamic array in java using TDD in one hour. From time to time I would break protocol and jump into implementation before the tests. Overall it went good. I finish the job. The interviewer however assesed that he was afraid I would not actually use TDD on the job, he was spot on. I did not get the job.
- arcanemachiner 2y ago
  Congratulations on dodging that bullet.
  - atum47 2y ago
    I mean, yeah. Now that I have a stable well paying job I can say that. Back then I was kinda sad. My first interview in english. I was extra nervous.
- charleslmunger 2y ago
  This is hilarious because it's a perfect interview (detects precisely the thing they're testing for) but also provides total adverse selection because the thing they're testing for is ridiculous.
  - hitchstory 2y ago
    Ive done this too. The exercise wasnt arrays (Im militant about only setting very realistic tasks). My task required modifying existing production-like code and tests. My hope was always that the candidates would do TDD where it seemed simple and obvious to do so. It was actually pretty rare but the candidates that defaulted to doing that always ended up being better in my opinion. They were always made offers higher than my company could afford elsewhere (so i guess in others' opinions too). In this thread https://news.ycombinator.com/item?id=43060636 https://news.ycombinator.com/item?id=43060636 I pondered why most people dont default to TDD for production code and the answer invariably seemed to be "we didnt think TDD was a thing you could do with integration/e2e tests".
hansvm 2y ago
Disclaimer: I don't have a lot of comments on TDD as a whole, other than that most software I write is very exploratory (I'll throw away the first 3-10 drafts, and by no means does that mean it takes 3-10x as long to write), and the best language for me for that exploration is often the language of the actual software I'm writing. TDD, in that environment, doesn't seem very applicable since the whole point is that we don't know what's actually possible (or, when it's possible, if the tradeoffs are worth it). The author has a lot of opinions about testing though which conflict with what I've found to work in even that sort of dynamic environment. Their rationale makes sense on the surface (e.g., I've never seen a "mock"-heavy [0] codebase reap positive net value from its tests), but the prescription for those observed problems seems sub-optimal. I'll pick on one of those complaints to start with, IMO the most egregious: > Now, you change a little thing in your code base, and the only thing the testing suite tells you is that you will be busy the rest of the day rewriting false positive test cases. If changing one little thing results in a day of rewriting tests, then either (a) the repo is structured such that small functional changes affect lots of code (which is bad, but it's correct that you'd therefore have to inspect all the tests/code to see if it actually works correctly afterward), or (b) the tests add coupling that doesn't exist otherwise in the code itself. I'll ignore (a), since I think we can all agree that's bad (or at least orthogonal to testing concerns). For (b) though, that's definitely a consequence of "mock"-heavy frameworks. Why? The author's proposal is to just test observable behavior of the system. That's an easy way to isolate yourself from implementation details. I don't disagree with it, and I think the industry (as I've seen it) discounts a robust integration test suite. What is it about "unit" tests that causes problems though? It's that the things you're testing aren't very well thought through or very well abstracted in the middle layers. Hear me out. TFA argues for integration tests at a high level, but if you (e.g.) actually had to implement a custom sorting function at your job would you leave it untested? Absolutely not. It'd be crammed to the gills with empty sets, brute-force checking every permutation of length <20, a smattering of large inputs, something involving MaxInt, random fuzzing against known-working sorting algorithms, and who knows what else the kids are cooking up these days. Moreover, almost no conceivable change to the program would invalidate those tests incorrectly. The point of a sorting algorithm is to sort, and it should have some performance characteristics (the reason you choose one sort over another). Your tests capture that behavior. As your program changes, you either say you don't need that sort any more (in which case you just delete the tests, which is O(other_code_deleted)), or you might need a new performance profile. In that latter case, the only tests that are broken are associated with that one sorting function, and they're broken _because_ the requirements actually changed. You still satisfy O(test_changes) <= O(code_changes); the thing the author is arguing doesn't happen because of mocks. Let's go back to the heavily mocked monstrosities TFA references. The problem isn't "unit" testing. Integration tests (the top of a DAG), and unit tests (like our sorting example, the bottom of a DAG) are easy. It's the code in between that gets complicated, and there might be a lot of it. What do we do then? At a minimum, I'd personally consider testing the top and bottom of your DAG of code. Even without any thought leadership or whatever garbage we're currently selling, it's easy to argue that tests at those levels are both O(other_code_written) in cost and also very valuable. At a high level (TFA's recommendation), the tests are much cheaper than the composite product, and you'd be silly not to include them. At a low level (truly independent units, like the "sorting" case study), you'd also be silly not to include them, since your developers are already writing those tests to check if it works as they implement the feature in the first place, and the maintenance cost of the tests is both proportional to the maintenance cost of the code being tested and extremely valuable in detecting defects in that code (recall that bugs are exponentially more expensive to fix the further down the pipeline the propogate before being triaged). Addressing the bottom of your DAG is something the article, in some sense, explicitly argues against. They're arguing against the inverted pyramid model you've seen for testing. That seems short-sighted. Your developers are already paying approximately the cost of writing a good test when they personally test a sorting function they're writing, and that test is likely to be long-lived and useful; why throw that away? More importantly, building on shaky foundations is much more expensive than most people give it credit for. If your IDE auto-complete suggests a function name that says it does the right thing and accepts the arguments you're giving it, you get an immediate 10x in productivity if that autocomplete is always right. Wizards in a particular codebase (I've been that wizard in a few, my current role as well; that isn't a derogatory assessment of "other" people) can always internalize the whole thing and immediately know the right patterns, but for everyone else with <2yrs of experience in your company in particular (keep in mind that average silicon valley attrition is 2-3yrs), a function doing what it says it's going to do is a godsend to productivity. Back to the problem at hand though. TFA says to integration test, and so do I. I also say to test your "leaf" code in your code DAG, since it's about the same cost and benefit. What about the shit in between? In a lot of codebases I've seen, I'd say to chock it up as a lost cause and test both the integration stuff (that TFA suggest) and also any low-level details (the extra thing I'm saying is important). Early in my career, I was implementing some CRUD feature or another and explicitly coached (on finding that the reason implementation was hard was a broken function deep in the call-stack) to do the one-liner fix to make my use case work instead of the ten-liner to make the function actually correct and the 1000-liner to then correct every caller. I don't think they were wrong in giving that advice. I'm sad that the code was in a state where that was reasonable advice. If you're working on newer projects though (or plan to be at a place for awhile and have the liberty to do some cleanup with every new feature (a pattern I wholly endorse and which has served me very well personally)), it's worth looking at that middling code and figuring out why it's so hard to work with. 99% of the time, the reason mocks look attractive isn't because they're the only solution. It's because they're the only solution that makes sense once you've already tied your hands. You don't need something to "unit" test the shutdown handler; you need something to test the total function which processes inputs and outputs and is called by the shutdown handler. You don't need to "unit" test a UI page that requires 3 different databases to produce any output; you need to unit test the functions which turn that output into that UI page (ideally, without mocks, since although those ostensibly do the same thing they usually add an extra layer of complexity and somehow break all your tests), and for something that messy you might even just need an "integration" test around that UI page asserting that it renders approximately correctly. What else? People sell all kinds of solutions. "Functional Programming" or "OOP" or whatever. Programming is imperative when you execute it, and the right representation for the human reader varies from problem to problem. I don't have any classes to sell or methodologies to recommend. I do strongly recommend taking a very close look at the abstractions you've chosen though. I've had no problem deleting 90% of them at new jobs, making the code faster, more correct, and easier to modify (I usually do so as part of a "coup," fixing things slowly with each new feature). When every new feature deletes code, the benefits tend to snowball. I see my colleagues doing that now to code I recently wrote, and I'd personally do it again. [0] People typically mean one of two things when they say they're "mocking" a dependency. The first is that they want a function to be "total" and have reasonable outputs for all possible inputs. They'll mock out many different interface implementations (or equivalent blah blah blah in your favorite language) to probe that behavior and ensure that your exponential backoff routine behaves reasonably when the clock runs backward, when 1000 of them are executed simultaneously, and whatnot. That tends to make for expensive tests, so I tend to see it reserved for risky code in teams which are risk-averse, but it's otherwise very good at its job. The other case is using some sort of "mock" library which lets you treat hard dependencies as soft dependencies and modify class instantiation, method return values, and all sorts of things to fit the test you're trying to write. This latter case is much more common, so it's what I'm referring to in a "heavily mocked" codebase. It's a powerful tool which could be used for good, but IME it's always overused enough that it would be better if it didn't exist.
motorest 2y ago
From the article: > The argument for isolating the units from each other is that it is easier to spot a potential bug. (...) In my opinion, this does not pay out because of the huge amount of false positive test cases you get and the time you need to fix them. Also, if you know the code base a little you should have an idea where the problem is. If not, this is your chance to get to know the code base a little better. This is at best specious reasoning, and to me reflects that the blogger completely misses the point of having tests. To start off, there is no such thing as a false positive test. Your tests track invariants, specially those which other components depend on. The whole point of having these tests is to have a way to automatically check for them each and every single time we touch the code, so that the tests warn us that a change we are doing will cause the application to fail. If you somehow decide to change your code so that a few invariants break, these are not "false positives". This is your tests working as expected and warning you that you must pay attention to what you are doing so that you do to not introduce regressions. It's also completely mind-boggling and absurd to argue that "knowing the code" is any argument to avoid tracking invariants. The whole point of automated test suites is that you do not want the app to fail because you missed any detail or corner case or failure mode. Knowing the code does not prevent bugs or errors or regressions. I'm perplexed by the way we have people write long articles on unit tests when they don't really seem to understand what they are supposed to achieve.
- theamk 2y ago
  Maybe in the ideal case, but in practice, a lot of unit tests track much more than invariants other components depend on. Example: a function makes two RPC calls to query two independent statuses, which are of course mocked for the unit tests. Due to the the test library used, unit tests expect query A first and query B second. The function is refactored, and it now queries B first, so unit tests start to fail. This is 100% false positive - nothing in the real world cares if A or B are queried first, those actually go to systems which don't even know each other. And yet, the change author now has to fix dozens of failed tests. Another example: there is an event-based system, and unit tests hardcode number of main loop executions ("emit event A; poll; poll; poll; ensure event B arrives"). A change which adds one more poll cycle will have zero effect on the real app, but causes many tests to break. It's a false positive. Yet another example: there are random numbers involved, so test driver makes sure to seed RNG to known value at start of each test. A single call to random() is added somewhere early in the code and boom! each test explodes. Another hundreds of false positives. I could go on an on... Should those tests be rewritten to be more robust and only check things that really matter? Yep. Does real-life code has fragile unit tests like this? Yep, almost every single codebase I have seen.
simonw 2y ago
Wow. If "unit" in "unit test" does indeed mean that the test itself should be able to run independent of the other tests then maybe I can get over my avoidance of calling them "unit tests"! I dislike that term because the most valuable tests I write are inevitably more in the shape of integration tests - tests that exercise just one function/class are probably less than 10% of the tests that I write. So I call my tests "tests", but I get frustrated that this could be confused with manual tests, so then I call them "automated tests" but that's a bit of a mouthful and not a term many other people use. I'd love to go back to calling them "unit tests", but I worry that most people who hear me say that will still think I'm talking about the test-a-single-unit-of-code version.
- voiceofunreason 2y ago
  "I call them 'unit tests' but they don't match the accepted definition of unit tests very well." -- Kent Beck, __Test Driven Development By Example__ The short version is that "unit test" did actually mean something (see Glenford Myers, __The Art of Software Testing__ or Boris Beizer, __Software Testing Techniques__), although it wasn't necessarily clear how those definitions applied to object-oriented programming (see Robert Binder, __Testing Object-Oriented Systems__). The Test-First/TDD/XP community later made an effort to pivot to the language of "programmer test", but by the time that effort began it was already too late. So I think you should continue to call your tests "tests" (or "checks", if you prefer the framing of James Bach and Michael Bolton). As best I can tell - there's no historicity to the idea that "unit test" was a reference to the isolation of a tests from its peers; it's just a ret-con.
- hitchstory 2y ago
  Ive had this experience with team-specific vocab where certain terms organically end up having terms with two or more conflicting meanings and it was horrendous. It led to all sorts of bugs, misunderstandings and even arguments. Even worse, most people didnt realize there was a problem coz they always knew what they meant. The only time I managed to work past it was by convincing everyone to never use that term again - burning it to the ground - and agreeing to replace it with two or more new, unambiguous terms. Id love to burn "unit test" and "integration test" to the ground but nobody outside my team listens to me :) Id probably replace them with: * code coupled * interface coupled * high level * low level * xUnit * faked infrastructural * deployed infrastructural * hermetic / non hermetic * declarative / non declarative
Supermancho 2y ago
> The argument for isolating the units from each other is that it is easier to spot a potential bug. That's not the only argument. The important result of this, is ensuring the "unit" of code is written to be testable. This happens to require it be simple and extensible. It does not enforce making the code or tests comprehensible. When you don't trust someone's code, have them write detailed unit tests. They will find most of their problems on their own and learn better practices, along the way. I am, in no way, implying that unit tests are a replacement for integration or behavioral or E2E testing et al...depending on how you want to define those.
deterministic 2y ago
What works great for me: I auto test the API of the server/system/library/module I am responsible for. Nothing else. No auto testing of internal details. It lets me completely rewrite internals without breaking the tests. The API tests needs to be so good that another developer could implement the same server/system/library/module using the tests only. And the API tests needs to try as hard as possible to break the code being tested. Using this method I have had zero bugs in production for the last 5+ years.

63 points by WolfOliver 2y ago

> Do not isolate code when you test it. > Only isolate your code from truly external services That makes tests more trustworthy but also sometimes harder to maintain I think. I have seen cases where small changes on the code base created strong ripple effects with many tests to update. Arguably, the tests were not very well written or organized and with too many high level tests. Still, this and the very large execution time of the test collection made me realized that for medium to large projects, I will be much more careful in the future before going all in with the no-mock approach.

nobleach 2y ago
Or, write code that _can_ work in isolation. (Pure functions). You can bet when GM builds fuel injectors, they have a harness that they can place them in to make sure that JUST the injector gets tested... putting it in a car and driving around is the wrong approach. The same is often true for software. Ensuring that things integrate is vital. Ensuring that functions run in isolation also has value.
- battered8310 2y ago
  Absolutely, whenever there is hardware in the loop, the more of the system you test with the more expensive the test is and the harder it is to isolate issues. That’s not to say you can skip end to end testing, but the more testing pushed to the complete system level will drive up cost and schedule.
  - nobleach 2y ago
    To beat my example into the ground. If I were to put a fuel injector (system under test) into a vehicle, and drive it around a parking lot (E2E test), only to have the vehicle stall out. Is the fuel injector at fault? How would you prove that? No, you pull out the component and you "bench test" it. Believe me, after many MANY hours spend fighting bugs where the extensive E2E suite passed, we finally had to go back to adding in unit tests for the small units under the surface. Not all... but many places.
- t0mas88 2y ago
  This makes sense when the cost of running the integration tests is high due to real hardware being in the loop. But for most business software this isn't the case and e2e tests aren't actually costly. Looking at a previous project we ran full e2e tests from an end user perspective in three major browsers in less than 15 minutes at a cloud cost of only a few hundred dollars per month. Compared to the cost of developers on the project that was a negligible amount. Adding more unit tests would make the feedback cycle shorter on some changes, but also add development time to create and maintain those tests. So in line with the article we did so only sparingly on a few critical paths that were likely to cause issues. The rest was well covered by e2e tests.
  - sarchertech 2y ago
    e2e tests that run in 15 minutes in the cloud are useless for TDD though which is what the article is talking about.

recursivedoubts 2y ago

Agree w/the author that the concept of "unit" often hurts test quality. You should be striving to balance the long-term usefulness of your tests with the debuggability of those tests. In my experience, those tests are what most people would call "integration tests" (although that name, like so much terminology in the testing world, is confusing and poorly defined.) You want to get the tests up at as high a level of abstraction as possible where the API and correctness assertions are likely to survive implementation detail changes (unlike many unit tests) while at the same time avoiding the opaque and difficult to debug errors that come with end-to-end testing (again, the language here is confusing, I assume you know what I mean.) See https://grugbrain.dev/#grug-on-testing https://grugbrain.dev/#grug-on-testing

nobleach 2y ago

Eh, I knew this would somehow come back to that Kent C Dodds stuff. I can respect the guy, but his opinion on valuable testing differs from mine and quite a lot of others. His opinion is "the more your tests represent the way your users use your app, the better they are". For E2E tests, yes, absolutely. For unit tests, no... not at all. As a very short example, an E2E test will prove that yes, I did indeed show a button on the screen. A unit test will prove the code that put it there worked. The code can "fail open" for many reasons. (In JavaScript for example, forgetting an `await` keyword can return a truthy value!). The worst part about it is that he called himself a thought leader, called his approach a "best practice" and had nothing really to back that up. Now people go around repeating it all the time. It's frustrating.

9rx 2y ago
That’s not in line with the original definition of unit test. It also sounds like a pointless test using your definition. There is no advocacy for writing pointless tests.
- nobleach 2y ago
  It's pointless if one is just trying to put "points on the board" and see a bunch of green in their console. Knowing what to test is important. Should every function be tested? Probably not. Should every function that has conditional logic be tested? Maybe? I totally get that integration tests have immense value. Where I disagree is that, "the closer your tests are to the way a user would use your app" is a good way to use testing. Functional/Acceptance/E2E tests are but one tool in an aresenal. Telling new software developers to ignore 25 years of history because React components were hard to test in isolation (Enzyme - the package Kent's React Test Library is built to replace) is irresponsible. Telling them that it's "best practice" is disingenuous.
int_19h 2y ago
If you have an E2E test that tests the button behavior thoroughly (i.e. not just showing it, but the fact that, when it's clicked, the app state changes accordingly), then you have a test showing that "code that put it there worked".

int_19h 2y ago

Simple rule of thumb. Always write functional tests first. Doesn't matter if they are slow - you still want something that faithfully captures the specified behavior and allows you to detect regressions automatically. Then, if your resulting test suite is too slow, add finer-grained tests in areas where the perf benefits of doing so dwarf the cost of necessary black-boxing. Getting down to the level of individual classes, never mind functions - i.e. the traditional "unit tests" - should be fairly rare in non-library code.

jasonm23 2y ago
aka the inverted test triangle.
wink 2y ago
I'm not trying to be confrontational, but even "business logic" inside a project is often clearly separated between "library" and "glue, aka controllers or whatever" - and that's kinda the point where I disagree. Not accepting "but the unit tests work" - 100% agreed. Not being able to slice it that the thing that could be accessed by the current public API or a different public API be tested on its own (and often relatively pure) is not how I want to write my software or tests.
- int_19h 2y ago
  Thing is, trying to make everything individually testable in tiny bits takes a lot of effort that wouldn't otherwise be needed. Furthermore, with all the mocking this usually requires, you end up with tons of extraneous code that now needs to be maintained as well (and by very nature of mocks, you'll have to fix them every time the corresponding real code changes behavior observable across the boundary). In practice, the ostensible benefit of knowing exactly what the problem is as soon as you see the test failure is usually dwarfed by all these costs. And, at the same time, the more mocks you have, the less likely your tests are to catch real issues. Now, as you rightly note, in many cases there's already an obvious component boundary that is inherent in the design. And then it makes perfect sense to write functional tests for individual components along those boundaries. And in some cases, the component may be "library-like", and those tests will end up looking a lot like your typical unit test, where individual classes and even methods are covered one by one. But most classes and methods in a typical app aren't like that.

arialdomartini 2y ago

> see here and here I believe links are significantly more useful when they include descriptive text like the title or author, rather than just 'here'.

codr7 2y ago

I rarely write isolated unit tests. Most of the systems I build use a database on which all logic depends, and often a network connection. I've worked on systems where these aspects were mocked, and they eventually grind to a halt because of the effort required to make the tiniest change. First of all you need a way to create a pristine database from code, preferably in memory. Second nested nested transactions are nice, since you can simply rollback the outer transaction per test case; otherwise you need to drop/create the database which is slower. For networked servers, an easy way to start/stop servers in code and send requests to them is all you need. Given these pieces, it's easy to write integration tests that run fast enough and give a lot of bang for the buck. TDD is even more rare for me, I typically only do that when designing API's I'm unsure about, which makes imagining user code difficult. And fixing bugs, because it makes total sense to have a failing test to verify that you fixed it, and that it remains fixed.

hitchstory 2y ago
I still find the skepticism around TDD weird. Except for a few pretty niche scenarios (e.g. it's experimental code or manual testing is cheaper for some obscure reason) i dont really see the point of not doing it. I especially dont see what is gained by writing the test after.
- codr7 2y ago
  Coding is not religion for me, I have no patience for fundamentalism. I will write as many tests as I need to feel confident, which depends on context. And integration tests give me a lot more confidence than mocked unit tests.
  - hitchstory 2y ago
    I hate coding fundamentalism with a passion too. The only thing I get really religious about in coding is the importance of trade offs. The cost/benefit of writing a test before just consistently exceeded doing it after for me. Same for integration, e2e or unit tests (there's never been a rule that says you can only TDD with a unit test). The cost/benefit trade off for tests with mocks vs. database is a different topic - orthogonal to the practise of red/green/refactor, and one where IMO the trade offs are much less obvious.

atum47 2y ago

My first job interview after college. It was a pair programming session but I would not touch the keyboard. I would dictate and the interviewer would type. I had to implement a dynamic array in java using TDD in one hour. From time to time I would break protocol and jump into implementation before the tests. Overall it went good. I finish the job. The interviewer however assesed that he was afraid I would not actually use TDD on the job, he was spot on. I did not get the job.

arcanemachiner 2y ago
Congratulations on dodging that bullet.
- atum47 2y ago
  I mean, yeah. Now that I have a stable well paying job I can say that. Back then I was kinda sad. My first interview in english. I was extra nervous.
charleslmunger 2y ago
This is hilarious because it's a perfect interview (detects precisely the thing they're testing for) but also provides total adverse selection because the thing they're testing for is ridiculous.
- hitchstory 2y ago
  Ive done this too. The exercise wasnt arrays (Im militant about only setting very realistic tasks). My task required modifying existing production-like code and tests. My hope was always that the candidates would do TDD where it seemed simple and obvious to do so. It was actually pretty rare but the candidates that defaulted to doing that always ended up being better in my opinion. They were always made offers higher than my company could afford elsewhere (so i guess in others' opinions too). In this thread https://news.ycombinator.com/item?id=43060636 https://news.ycombinator.com/item?id=43060636 I pondered why most people dont default to TDD for production code and the answer invariably seemed to be "we didnt think TDD was a thing you could do with integration/e2e tests".

hansvm 2y ago

Disclaimer: I don't have a lot of comments on TDD as a whole, other than that most software I write is very exploratory (I'll throw away the first 3-10 drafts, and by no means does that mean it takes 3-10x as long to write), and the best language for me for that exploration is often the language of the actual software I'm writing. TDD, in that environment, doesn't seem very applicable since the whole point is that we don't know what's actually possible (or, when it's possible, if the tradeoffs are worth it). The author has a lot of opinions about testing though which conflict with what I've found to work in even that sort of dynamic environment. Their rationale makes sense on the surface (e.g., I've never seen a "mock"-heavy [0] codebase reap positive net value from its tests), but the prescription for those observed problems seems sub-optimal. I'll pick on one of those complaints to start with, IMO the most egregious: > Now, you change a little thing in your code base, and the only thing the testing suite tells you is that you will be busy the rest of the day rewriting false positive test cases. If changing one little thing results in a day of rewriting tests, then either (a) the repo is structured such that small functional changes affect lots of code (which is bad, but it's correct that you'd therefore have to inspect all the tests/code to see if it actually works correctly afterward), or (b) the tests add coupling that doesn't exist otherwise in the code itself. I'll ignore (a), since I think we can all agree that's bad (or at least orthogonal to testing concerns). For (b) though, that's definitely a consequence of "mock"-heavy frameworks. Why? The author's proposal is to just test observable behavior of the system. That's an easy way to isolate yourself from implementation details. I don't disagree with it, and I think the industry (as I've seen it) discounts a robust integration test suite. What is it about "unit" tests that causes problems though? It's that the things you're testing aren't very well thought through or very well abstracted in the middle layers. Hear me out. TFA argues for integration tests at a high level, but if you (e.g.) actually had to implement a custom sorting function at your job would you leave it untested? Absolutely not. It'd be crammed to the gills with empty sets, brute-force checking every permutation of length <20, a smattering of large inputs, something involving MaxInt, random fuzzing against known-working sorting algorithms, and who knows what else the kids are cooking up these days. Moreover, almost no conceivable change to the program would invalidate those tests incorrectly. The point of a sorting algorithm is to sort, and it should have some performance characteristics (the reason you choose one sort over another). Your tests capture that behavior. As your program changes, you either say you don't need that sort any more (in which case you just delete the tests, which is O(other_code_deleted)), or you might need a new performance profile. In that latter case, the only tests that are broken are associated with that one sorting function, and they're broken _because_ the requirements actually changed. You still satisfy O(test_changes) <= O(code_changes); the thing the author is arguing doesn't happen because of mocks. Let's go back to the heavily mocked monstrosities TFA references. The problem isn't "unit" testing. Integration tests (the top of a DAG), and unit tests (like our sorting example, the bottom of a DAG) are easy. It's the code in between that gets complicated, and there might be a lot of it. What do we do then? At a minimum, I'd personally consider testing the top and bottom of your DAG of code. Even without any thought leadership or whatever garbage we're currently selling, it's easy to argue that tests at those levels are both O(other_code_written) in cost and also very valuable. At a high level (TFA's recommendation), the tests are much cheaper than the composite product, and you'd be silly not to include them. At a low level (truly independent units, like the "sorting" case study), you'd also be silly not to include them, since your developers are already writing those tests to check if it works as they implement the feature in the first place, and the maintenance cost of the tests is both proportional to the maintenance cost of the code being tested and extremely valuable in detecting defects in that code (recall that bugs are exponentially more expensive to fix the further down the pipeline the propogate before being triaged). Addressing the bottom of your DAG is something the article, in some sense, explicitly argues against. They're arguing against the inverted pyramid model you've seen for testing. That seems short-sighted. Your developers are already paying approximately the cost of writing a good test when they personally test a sorting function they're writing, and that test is likely to be long-lived and useful; why throw that away? More importantly, building on shaky foundations is much more expensive than most people give it credit for. If your IDE auto-complete suggests a function name that says it does the right thing and accepts the arguments you're giving it, you get an immediate 10x in productivity if that autocomplete is always right. Wizards in a particular codebase (I've been that wizard in a few, my current role as well; that isn't a derogatory assessment of "other" people) can always internalize the whole thing and immediately know the right patterns, but for everyone else with <2yrs of experience in your company in particular (keep in mind that average silicon valley attrition is 2-3yrs), a function doing what it says it's going to do is a godsend to productivity. Back to the problem at hand though. TFA says to integration test, and so do I. I also say to test your "leaf" code in your code DAG, since it's about the same cost and benefit. What about the shit in between? In a lot of codebases I've seen, I'd say to chock it up as a lost cause and test both the integration stuff (that TFA suggest) and also any low-level details (the extra thing I'm saying is important). Early in my career, I was implementing some CRUD feature or another and explicitly coached (on finding that the reason implementation was hard was a broken function deep in the call-stack) to do the one-liner fix to make my use case work instead of the ten-liner to make the function actually correct and the 1000-liner to then correct every caller. I don't think they were wrong in giving that advice. I'm sad that the code was in a state where that was reasonable advice. If you're working on newer projects though (or plan to be at a place for awhile and have the liberty to do some cleanup with every new feature (a pattern I wholly endorse and which has served me very well personally)), it's worth looking at that middling code and figuring out why it's so hard to work with. 99% of the time, the reason mocks look attractive isn't because they're the only solution. It's because they're the only solution that makes sense once you've already tied your hands. You don't need something to "unit" test the shutdown handler; you need something to test the total function which processes inputs and outputs and is called by the shutdown handler. You don't need to "unit" test a UI page that requires 3 different databases to produce any output; you need to unit test the functions which turn that output into that UI page (ideally, without mocks, since although those ostensibly do the same thing they usually add an extra layer of complexity and somehow break all your tests), and for something that messy you might even just need an "integration" test around that UI page asserting that it renders approximately correctly. What else? People sell all kinds of solutions. "Functional Programming" or "OOP" or whatever. Programming is imperative when you execute it, and the right representation for the human reader varies from problem to problem. I don't have any classes to sell or methodologies to recommend. I do strongly recommend taking a very close look at the abstractions you've chosen though. I've had no problem deleting 90% of them at new jobs, making the code faster, more correct, and easier to modify (I usually do so as part of a "coup," fixing things slowly with each new feature). When every new feature deletes code, the benefits tend to snowball. I see my colleagues doing that now to code I recently wrote, and I'd personally do it again. [0] People typically mean one of two things when they say they're "mocking" a dependency. The first is that they want a function to be "total" and have reasonable outputs for all possible inputs. They'll mock out many different interface implementations (or equivalent blah blah blah in your favorite language) to probe that behavior and ensure that your exponential backoff routine behaves reasonably when the clock runs backward, when 1000 of them are executed simultaneously, and whatnot. That tends to make for expensive tests, so I tend to see it reserved for risky code in teams which are risk-averse, but it's otherwise very good at its job. The other case is using some sort of "mock" library which lets you treat hard dependencies as soft dependencies and modify class instantiation, method return values, and all sorts of things to fit the test you're trying to write. This latter case is much more common, so it's what I'm referring to in a "heavily mocked" codebase. It's a powerful tool which could be used for good, but IME it's always overused enough that it would be better if it didn't exist.

motorest 2y ago

From the article: > The argument for isolating the units from each other is that it is easier to spot a potential bug. (...) In my opinion, this does not pay out because of the huge amount of false positive test cases you get and the time you need to fix them. Also, if you know the code base a little you should have an idea where the problem is. If not, this is your chance to get to know the code base a little better. This is at best specious reasoning, and to me reflects that the blogger completely misses the point of having tests. To start off, there is no such thing as a false positive test. Your tests track invariants, specially those which other components depend on. The whole point of having these tests is to have a way to automatically check for them each and every single time we touch the code, so that the tests warn us that a change we are doing will cause the application to fail. If you somehow decide to change your code so that a few invariants break, these are not "false positives". This is your tests working as expected and warning you that you must pay attention to what you are doing so that you do to not introduce regressions. It's also completely mind-boggling and absurd to argue that "knowing the code" is any argument to avoid tracking invariants. The whole point of automated test suites is that you do not want the app to fail because you missed any detail or corner case or failure mode. Knowing the code does not prevent bugs or errors or regressions. I'm perplexed by the way we have people write long articles on unit tests when they don't really seem to understand what they are supposed to achieve.

theamk 2y ago
Maybe in the ideal case, but in practice, a lot of unit tests track much more than invariants other components depend on. Example: a function makes two RPC calls to query two independent statuses, which are of course mocked for the unit tests. Due to the the test library used, unit tests expect query A first and query B second. The function is refactored, and it now queries B first, so unit tests start to fail. This is 100% false positive - nothing in the real world cares if A or B are queried first, those actually go to systems which don't even know each other. And yet, the change author now has to fix dozens of failed tests. Another example: there is an event-based system, and unit tests hardcode number of main loop executions ("emit event A; poll; poll; poll; ensure event B arrives"). A change which adds one more poll cycle will have zero effect on the real app, but causes many tests to break. It's a false positive. Yet another example: there are random numbers involved, so test driver makes sure to seed RNG to known value at start of each test. A single call to random() is added somewhere early in the code and boom! each test explodes. Another hundreds of false positives. I could go on an on... Should those tests be rewritten to be more robust and only check things that really matter? Yep. Does real-life code has fragile unit tests like this? Yep, almost every single codebase I have seen.

simonw 2y ago

Wow. If "unit" in "unit test" does indeed mean that the test itself should be able to run independent of the other tests then maybe I can get over my avoidance of calling them "unit tests"! I dislike that term because the most valuable tests I write are inevitably more in the shape of integration tests - tests that exercise just one function/class are probably less than 10% of the tests that I write. So I call my tests "tests", but I get frustrated that this could be confused with manual tests, so then I call them "automated tests" but that's a bit of a mouthful and not a term many other people use. I'd love to go back to calling them "unit tests", but I worry that most people who hear me say that will still think I'm talking about the test-a-single-unit-of-code version.

voiceofunreason 2y ago
"I call them 'unit tests' but they don't match the accepted definition of unit tests very well." -- Kent Beck, __Test Driven Development By Example__ The short version is that "unit test" did actually mean something (see Glenford Myers, __The Art of Software Testing__ or Boris Beizer, __Software Testing Techniques__), although it wasn't necessarily clear how those definitions applied to object-oriented programming (see Robert Binder, __Testing Object-Oriented Systems__). The Test-First/TDD/XP community later made an effort to pivot to the language of "programmer test", but by the time that effort began it was already too late. So I think you should continue to call your tests "tests" (or "checks", if you prefer the framing of James Bach and Michael Bolton). As best I can tell - there's no historicity to the idea that "unit test" was a reference to the isolation of a tests from its peers; it's just a ret-con.
hitchstory 2y ago
Ive had this experience with team-specific vocab where certain terms organically end up having terms with two or more conflicting meanings and it was horrendous. It led to all sorts of bugs, misunderstandings and even arguments. Even worse, most people didnt realize there was a problem coz they always knew what they meant. The only time I managed to work past it was by convincing everyone to never use that term again - burning it to the ground - and agreeing to replace it with two or more new, unambiguous terms. Id love to burn "unit test" and "integration test" to the ground but nobody outside my team listens to me :) Id probably replace them with: * code coupled * interface coupled * high level * low level * xUnit * faked infrastructural * deployed infrastructural * hermetic / non hermetic * declarative / non declarative

Supermancho 2y ago

> The argument for isolating the units from each other is that it is easier to spot a potential bug. That's not the only argument. The important result of this, is ensuring the "unit" of code is written to be testable. This happens to require it be simple and extensible. It does not enforce making the code or tests comprehensible. When you don't trust someone's code, have them write detailed unit tests. They will find most of their problems on their own and learn better practices, along the way. I am, in no way, implying that unit tests are a replacement for integration or behavioral or E2E testing et al...depending on how you want to define those.

deterministic 2y ago

What works great for me: I auto test the API of the server/system/library/module I am responsible for. Nothing else. No auto testing of internal details. It lets me completely rewrite internals without breaking the tests. The API tests needs to be so good that another developer could implement the same server/system/library/module using the tests only. And the API tests needs to try as hard as possible to break the code being tested. Using this method I have had zero bugs in production for the last 5+ years.