13 ms·

The best talk on this topic, IMO is Ian Cooper: "TDD, Where did it all go wrong?" https://www.youtube.com/watch?v=EZ05e7EMOLM https://www.youtube.com/watch?v=EZ

• TeeMassive 6y ago
  Yes this is the best talk so far. Right now I am in a company that has strict class-method testing strategy. This makes refactoring a pain in the A and decreases the programmer time / quality insured by the test ratio. In "unit testing", the unit doesn't mean "the smallest thing we test which we decided is a class even if doesn't make sense"; it simply means "on its own" so external dependencies are not in the way. My heuristic for determining how much a test is worth writing is something like this: cost of number of outbound dependence * human understanding complexity + non-immutability + how much code coverage it adds. This leads to a grouping of tests and code that naturally fit together.
• fatnoah 6y ago
  >As such, a good trigger rule is "one test per desired external behavior of the unit, plus one test per bug fixed". The test for each bugfix comes from experience -- they delineate tricky parts of your unit and enforce workign code around them. So much this. The smoothest integration I've ever worked on was one where I owned the API and back-end and another team built the front end. I defined the API behaviors, built the tests verify those behaviors, and created a mock API for the front-end folks to use while testing. Over the course of building the API, I slowly got my test success rate from 0% to 100% and always immediately knew when I accidentally changed behaviors. When we finally integrated, there were literally 0 errors related to the client/server interface. Yes, there was some UI wonkiness and we discovered some issues of scale, but there were no issues with parameter values, HTTP response codes, error messages, etc. It was a) amazing and b) the only time I've had the luxury to build something in that manner.
  • dkarlovi 6y ago
    Did you build it around OpenAPI? I'm interested to try that approach.
    • fatnoah 6y ago
      No, this effort pre-dated the development of that spec.
• Insanity 6y ago
  +1 especially to the final point. If there was a bug, this bug should be replicated in the test. You then solve the bug, make sure the test (and others pass), and you'll be (relatively) sure the bug will not be reintroduced with a later change. Every bug you find is an "edge-case" you didn't anticipate. Leave it in the test for the future. I find that the "table test" approach of Go works surprisingly well with this. You just add a case to the table, and often that's all you have to do.
• rpadovani 6y ago
  > one test per bug fixed this, and you should try to write the test before fixing the bug - you cannot trust a test that you haven't seen failing
  • draw_down 6y ago
    I agree with the last part, but the ordering isn’t crucial. For example you can stash your changes in the implementation to check that the test fails when the change is not present and passes when it is. For this (among other reasons) it is nice to have a test runner that re-runs when it detects file changes. The ordering is one part of TDD that always bugged me- you have to write the tests first. But I often prefer to experiment and try a couple approaches before deciding on one. Having tests first would make a lot more overhead for that way of working.
    • wh33zle 6y ago
      To be fair though, if your tests are at the right abstraction level, the specific approach you are choosing for the implementation shouldn't matter for the test. Writing the test first also forces you to think abouy what API do you actually want to expose. Once you got the API right, there is still room for experimenting.
      • draw_down 6y ago
        That’s only true when you have decided what the abstraction will be. That’s my point, a lot of times you don’t know yet!
        • wrs 6y ago
          In this approach you decide on the abstraction (i.e., the API) by writing example code (i.e., some tests) that uses the API. The tests are how you decide which abstraction seems to make the most sense. It sounds like you actually implement the abstraction to decide whether it seems like the right one, which is a lot more work.
• tmountain 6y ago
  Great advice. Most tests are too discrete.
• humanrebar 6y ago
  It's worth noting that the greater the granularity of your unit, the more likely you will be able to write tests targeted at bugs. For instance, if you are only testing through the API of the service, you may have a hard-to-impossible time confirming you recover gracefully from certain exceptions. You generally don't have service-level APIs to throw various exceptions intentionally. Point being, the overzealous testers do have some good points, even if they miss the forest for the trees.
  • VHRanger 6y ago
    Yeah, sometimes for practical reasons you don't want to, or can't test directly across the API, as good testing practice would dictate. Taken to the extreme, the philosophy I laid out leads to something that looks like "only integration and end-to-end tests" depending on your architecture. So I try to be pragmatic whenever possible, but I think leaning towards BDD works better, after 18 months of doing it.
  • gvjddbnvdrbv 6y ago
    The bug triggered the exception. The test case encapsulates reproducing the bug. The bug is fixed. The bug can no longer be reproduced. As long as the bug remains fixed the test passes.. Another way of thinking about it. Unless your exceptions are a documented part of your API no one cares about them - they only care about the outcome they actually expect. If you construct tests that pass for positive outcomes or fail for any other outcome then your exceptions remain implementation details.
    • snewman 6y ago
      I think GP is referring to nondeterministic exceptions. For instance, if the service under test depends on some other service, then you may need to test the scenario where the other service is unavailable. The exception is not triggered by a bug, it is triggered by an operational externality.
      • gvjddbnvdrbv 6y ago
        Then test the externalities. They probably have healthcheck endpoints.... Don't overthink everything. KISS
• mumblemumble 6y ago
  If I recall correctly, another very important point he makes in that talk is that it's fine to delete tests. TDD tends to result in a lot of tests being created as a sort of scaffolding to support initial development. The thing about scaffolding is, when you're done using it, you tear it down. I don't think he mentions it during the talk, but the next step after deleting all those tests is a little bit more refactoring for maintainability. Now that you've deleted all the redundant tests, you can then guard against future developers unwittingly becoming tightly coupled to your implementation details, by taking all the members that used to only be exposed for testing purposes, and either deleting them or making them private.
  • VHRanger 6y ago
    Yes, you should delete tests for everything that isn't a required external behavior, or a bugfix IMO. Otherwise you're implicitly testing the implementation, which makes refactoring impossible. A big smell here is if the large majority of your tests are mocked. This might mean you're testing at too fine-grained a level.
    • mumblemumble 6y ago
      Agreed. I would take an even stronger position, and say that a high degree of mocking actually implies two things: First, yes, you're testing at too fine-grained a level. Second, it's a code smell that suggests you may be working with a fundamentally untestable design that relies overmuch on opaque, stateful behavior.
      • 411111111111111 6y ago
        Mocks are worthwhile though. Otherwise you end up not being able to unittest anything which accesses an external api such as databases, rest services etc.
        • deleted 6y ago
          [deleted]
        • 6y ago
• phkahler 6y ago
  >> When people talk about "unit tests", a unit doesn't refer to the common pattern of "a single class". A unit is a piece of the software with a clear boundary. It might be a whole microservice, or a chunk of a monolith that is internally consistent. Its OK to dislike unit testing, but please don't redefine the term to avoid it. That's not helpful. Instead try to find the papers (by NASA or IBM?) That show unit testing finds only very few actual bugs, making it low value. That said, there are IMHO some units worth testing more.
  • bcrosby95 6y ago
    They aren't redefining it though. The term has always been fuzzy. A common boundary of unit testing has always been a module's publicly exposed interface.
    • Izkata 6y ago
      Wasn't the "clear boundary" definition the original, that was later interpreted as syntactic boundary (function/class) instead of semantic boundary (a chunk of business logic)?
  • sixdimensional 6y ago
    Integration tests vs. unit tests.
• rad_gruchalski 6y ago
  > When people talk about "unit tests", a unit doesn't refer to the common pattern of "a single class". A unit is a piece of the software with a clear boundary. It might be a whole microservice, or a chunk of a monolith that is internally consistent. Yes, this. However, when we go in the direction of testing a complete service, is it not more convenient to call it an integration test?
  • digitalsushi 6y ago
    The parent above you didn't necessarily imply that the whole microservice was being integrated before testing it. It is reasonable that a microservice can exist as whole, but nonintegrated with its other deliverable components, and remains sufficiently unit-testable. It's also extremely likely that the microservice needs to be integrated to test it :)
  • KallDrexx 6y ago
    My personal view is that it's an integration test only if the test actively involves external components (like a real database, some other api, etc..). I don't use integration test terminology if all those external components are mocked/stubbed out (even if it encompasses a large "unit"). I also call those mocked surface area tests "functional" tests, partially to differentiate them from non-mocked integration tests, and partially because people get too hung up on the "unit" term.
    • closeparen 6y ago
      Often it's convenient to test the functions in your database access layer against a real or convincingly simulated database (e.g. sqlite). This is "like a unit test" in that it focuses on the details of individual functions, but "like an integration test" in that it crosses a system boundary rather than mocking it. I've not found it productive to use either term when talking about it.
• dkarl 6y ago
  > TDD, much like scrum, got corrupted by the "Agile Consulting Industry" > Overzealous testers test for each new public method in a class. The Agile Consulting Industry can be summed up as: know only one rigid way to do things, and if it doesn't work, blame the context. In the case of unit testing, if my blind mechanical rules for unit testing don't apply sanely to your code, then your code is wrong.
  • tilolebo 6y ago
    This. This is why I stopped following agile and other XP gurus who just live from consulting. It's easy to have strict principles when you never need to apply them yourself.
    • randompwd 6y ago
      Uncle Bob comes to mind
      • VHRanger 6y ago
        He's to blame for popularizing a lot of the unit testing insanity IMO, yes.
  • api 6y ago
    It can also be summed up as: "we see there are managers with budget, and we have a solution to this problem..." Any place you have money, you will find someone selling snake oil.
  • rvense 6y ago
    Agile is a religion. Well-meaning, smart people wrote some good, general guidelines. Somehow it got turned into an industry of people who want money to tell me how I'm living my life wrong.
    • TeeMassive 6y ago
      Yeah. Agile can be summed up by making little releases which you show to the client and so the developers can make decisions on what to develop next quickly. Everything else naturally derive from this. All the other stuff are just ways, which you can adopt or not, to accomplish that. You don't need to hire a consultant to tell you that you're not doing your poker story points meeting wrong because of metaphysical explanation.
• cjfd 6y ago
  I think most disenchantment with TDD comes from the second point that you notice. If one attempts to test every method of every class one ends up testing implementation details that are very much subject to change. Also, one can easily end up testing trivial points like 'is the processor still capable of adding two integers'. As you note in your third point it seems much more productive to test properties of code that the customer could potentially recognize as something they value. TDD really isn't dead for me. I do it pretty much every day. Both in work and in personal projects. I am not sure talks/conversations like these are very valuable. In the end it turns out that every practical question has the answer 'it depends'. Maybe the important thing to realize is that most questions do have the answer 'it depends' and that one can never stop using ones brain.
• koonsolo 6y ago
  > plus one test per bug fixed I used to be a big proponent of this, until I suggested this to my manager: "We ran the stats on our bugtracker, and bugs coming back are really rare, so we like to focus our effort on testing with a higher ROI". And in the end I agreed with him on this.
  • stkdump 6y ago
    Yeah, I never understood this safeguarding against bugs resurfacing after being fixed once. I only saw bugs coming back at a company that didn't use version control and instead copied source code back and forth with a USB stick. I can understand something like test driven bug fixing, where you basically create a simple test to reproduce the bug quickly and then fix the bug using that. In many cases that is the most efficient workflow. The test succeeding can then serve as evidence of the bugfix (though it might not be enough). So if you have already written the test, you might as well leave it in there, because it doesn't bother anyone usually, and the tiny chance that someone breaks this exact same thing again, while tiny isn't non-existant. But fixing a bug and then putting extra work just for a test, if there is another easier way to prove that the bug is fixed? No, thanks.
    • crazygringo 6y ago
      > Yeah, I never understood this safeguarding against bugs resurfacing after being fixed once. In my experience it's not infrequent for bugs to unknowingly only get half-fixed, not realizing that the true problem actually lies a level deeper, or has a mirror case, or whatever. Maybe a good example is that a parameter to a command is 0, the bugfix sets it to be 1, but a later bugfix changes it back to 0, when the correct bugfix would set it to be 0 in some cases and 1 in others. And that if you fix the bug without a test, then the second related bug crops up a couple months later, and somebody else tries to fix it similarly naively, and can wind up re-introducing the first bug if there isn't a test for it. Basically, in practice bugs have this nasty habit of clustering and affecting each other -- if the code was trickier than usual to write in the first place, it's going to be trickier than usual to fix, and more likely than usual to continue to have problems. So keeping tests for fixed bugs is kind of like applying extra armor where your tank gets hit -- statistically, it's going to pay off.
• spaetzleesser 6y ago
  “TDD, much like scrum, got corrupted by the "Agile Consulting Industry". Sticking to the original principles as laid out by Kent Beck results in fairly sane practices.” Totally agree. Somehow every good idea gets converted to a rigid ideology after a while. Same for OOP. It’s a solid idea but then the ideologues pushed it way too far. And instead of dialing back a little we see other ideologues declare “X is dead“ and the pendulum swings into another extreme direction. My company is generally behind the curve so now people have been bitten by the REST, JSON, Microservice bug. They don’t know why or what it really is but things have to be done that way. That together with calling themselves “agile” without understanding what it means besides using JIRA and having fixed sprints.
  • reading-at-work 6y ago
    > My company is generally behind the curve so now people have been bitten by the REST, JSON, Microservice bug. They don’t know why or what it really is but things have to be done that way. This resonates with me. My first job out of college was with a big, very old insurance company. My team lead became obsessed with using microservices for some reason, even though we were only building internal web apps that would have about 1,000 users on a busy day. There would be no performance concerns whatsoever that would warrant "breaking up a monolith" to make it more scalable. But microservices were a great way for the team to feel like we were using trendy tech despite not having any idea how to really go about doing it or any particular reason for doing so.
    • Whatanacc123 6y ago
      As a team lead, I struggle with the trade off of using trendy software vs the thing that works. I have a really easy web app framework that everybody hates because it looks terrible on a resume. So I have above average turnover.
• radicalbyte 6y ago
  I've lost so much time making these arguments with people. Unfortunately the combination of dogma + an industry of sham consultants who have monetized that have created a monster.
  • geebee 6y ago
    I think I won an argument with an interviewer, but I lost any chance at the job ;)
• u801e 6y ago
  > When people talk about "unit tests", a unit doesn't refer to the common pattern of "a single class". A unit is a piece of the software with a clear boundary. It might be a whole microservice, or a chunk of a monolith that is internally consistent. Every example I've read pertaining to unit testing uses a function as the unit to test. The easiest functions to test are ones that don't have side-effects (network, I/O, disk, etc). Could you point me to an example where a unit test applies to something beyond a function?
  • pydry 6y ago
    This bugs me as well. The people who argue about whether unit tests works will often redefine "unit" to mean anything from "the entire application" to a single function. Colloquially, however, anything above the level of a self contained function or class is called something else - typically an integration test.
    • UK-Al05 6y ago
      Unit tests, integration tests were never fully defined that well. They change based on who you talk to. Especially integration tests. Better definition is the small, medium and large tests from Google's testing blog.
  • bmh100 6y ago
    I've used unit tests on attributes and methods of a mock class instance, in order to test the class's construction method.
• linearVexponent 6y ago
  Avoiding too many layers of useless indirection is not hard: avoid refactoring effort that looks like a line by line, linear normalization of syntax. Target tests and refactor for the confusing, incomplete chunks according to the team if there is one. Semantic understanding of the system will improve. Instead of fetishizing code patterns, fetishize systemic understanding. Personally, that habit has made it so I write better code from the start. It’s acted like a forcing function to reconsider if a habit is useful or just a habit. My code went from deep OOP hierarchies of indirection, to composable, more functional, chunks. I import less, define fewer objects to begin with, compute a larger variety of useful objects, and can pull together features faster. Have standardized machine? Nursing my own symbol library is where most of the fun is. With respect to Martin Fowler and the rest, who are great engineers in their own right, but this all smells like pandering to efficiency, importing a shared model, which impacts resiliency. We shouldn’t have people think within the same context box everyday. Software philosophy has been taken over by the equivalent of popular bean counters. Focused on minimize the idea space for the perception of productivity gains. It’s cognitive indirection, IMO
• brightball 6y ago
  This is an excellent point. I once had to deal with an external contract firm on a project that I was hired to fix. We had issues of production code breaking so badly that it brought down the entire server (N+1 query issue that triggered 50k queries). The tests passed. When I emergency patched the issue and deployed it to production, the contract firm got mad at me for breaking their tests...to fix a production emergency. It’s put me on guard again militant test ideologues with no concept of real priorities ever since.
  • TeeMassive 6y ago
    > The tests passed. When I emergency patched the issue and deployed it to production, the contract firm got mad at me for breaking their tests...to fix a production emergency. I get the point, but it depends on which tests failed. Tests for unreleased features and trivial UX stuff is not the same as breaking a test making sure not every customer gets a 50% discount.
  • Benjammer 6y ago
    Be careful not to confuse militant ideology and local incentive structures.
    • brightball 6y ago
      Fair point. In this particular case the primary developer for them wanted to “publicly shame” me in Slack. Seemed much more ideology driven at the time.
      • bmh100 6y ago
        That may have just been the rationalization for an ulterior motive.
      • saagarjha 6y ago
        Public shaming of a developer is rarely the right thing to do–and I would expect most design paradigms do not include a section on it ;)
  • 6y ago
• brobdingnagians 6y ago
  Tedu had an interesting blog post about testing recently: https://flak.tedunangst.com/post/against-testing https://flak.tedunangst.com/post/against-testing What I got out of it was that tests for regressions are really good, but there are lots of considerations to make when determining which other tests to write, and why you are doing it. A good read nevertheless.
• edem 6y ago
  It would be great if it worked this way. What I see instead is that managers are complaining if coverage is less than 80% and you have to write tooling to compensate for generated code and all that crap. It is the 10th circle of hell.
  • bluGill 6y ago
    Generated code is easy, just generate the tests. I wish I was kidding.
• stx 6y ago
  > What triggers writing a test is what matters. Overzealous testers test for each new public method in a class. I have seen this before. At one company I forced by a couple of other developers to write tests for accessors(get/set) on model classes. They would reject my PR if I didn't do it. To leadership it looked like I was against automated testing. To me its more important to write tests for places where your most likely to make mistakes. For example calculations or business logic or how the view renders your model as it changes. Not every single small method.
• simiones 6y ago
  > - When people talk about "unit tests", a unit doesn't refer to the common pattern of "a single class". A unit is a piece of the software with a clear boundary. It might be a whole microservice, or a chunk of a monolith that is internally consistent. I don't understand how this matches to the idea of "don't write one line of code unless it's necessary to make a failing test to pass".
  • Ma8ee 6y ago
    What is it that you think is contradictory?
    • simiones 6y ago
      All of the design advantages I've ever heard advertised for TDD come from writing a bit of test, a bit of code, a bit more test, a bit more code. If instead you are writing a few tests and then an entire module, you're doing test-first development, but it's definitely not test-driven development as I've ever seen it presented by proponents.
      • Ma8ee 6y ago
        You need to write a module. You write tests that tests the functionality of the module. But to pass those tests you need a couple of classes. You write tests for the classes (or at least the one you intend to work on next). The class needs some methods, you write tests for those. You write code for the methods until all methods tests pass. Hopefully the tests for the class pass, otherwise you might need to update methods and tests for those. Unt so weiter.
• mytailorisrich 6y ago
  The main issue apart from the "Consulting Industry" is that too many people try to follow a methodology to the letter, but superficially, without trying to understand the point and deeper meaning. That's why people argue about rule or whether this or that can or cannot be done instead of trying to understand the aim first and then the cost/benefit balance.
  • bluGill 6y ago
    Which is a problem when you see something that seems like it should help so you hire a consultant who can push the letter of the rules but doesn't understand it well enough to figure out how it should work in your organization. I wasted a lot of time trying to get acceptance tests in here, they seem like a good idea but I can't get traction on them and the consultants preached the rules not how or why so the rules got roboticly followed with no helpful results. I'm still not sure if the concept is flawed or not just the execution but we threw it out.
• coredog64 6y ago
  But how can I measure that with a red/yellow/green stoplight chart based on code coverage? I’m actually supposed to trust that the developers will do the right thing?!
• ransom1538 6y ago
  Sorry. How is this hard? You test a function you write and write it down. Run the test. Why is this on Youtube? This is nuts - this isn't an existential developer crisis.
• agumonkey 6y ago
  OOP Class really devalued the idea of module. Focusing on terminology and not the concept caused harm.
• closeparen 6y ago
  Test coverage is the most conveniently accessible numeric value in the neighborhood of software quality, therefore it is software quality. Merging a change that reduces test coverage is reducing software quality. That's okay, sometimes we all have to cut corners, but defending this choice means you don't value quality, and are therefore not a culture fit. /s
  • GordonS 6y ago
    This is a rampant problem in the enterprise world, and it drives me nuts. I regularly have to work for clients who mandate using Sonar Qube (or/and other SAST tools) with strict policies, and also require 85%-100% test coverage on all projects regardless of how much sense it makes. Predictably, teams have to spend way too much energy getting "waivers" approved by some ridiculous group, and inevitably end up creating tests that don't actually test anything, just to get the coverage figures up.
    • lowbloodsugar 6y ago
      Can you give us an example of where it doesn't make sense?
      • GordonS 6y ago
        A trivial example would be getters and setters for plain old classes.
        • lowbloodsugar 6y ago
          Surely the getters and setters of POJOs would be exercised in the tests of other classes? If not, why are they there?
          • dropofwill 6y ago
            E.g. it’s public api in a library and they’re only used by other applications.
• wnevets 6y ago
  >- TDD, much like scrum, got corrupted by the "Agile Consulting Industry". Sticking to the original principles as laid out by Kent Beck results in fairly sane practices. Trying to explain this to people just gets exhausting after a while.
• mmcnl 6y ago
  Corrupted is not the right way to describe it. Both TDD and agile provide something amazing to management: a way to make the black hole called "software engineering" into something tangible and quantifiable. This ofcourse also makes it possible to execute some bad management practices on software engineering as well. People like to complain about agile (and apparently TDD) but I would argue that there are also huge success stories that don't make it to HN.
  • markpeppers 6y ago
    If you have a reference, I’d love to read about some of these success stories.
  • nickbauman 6y ago
    Let's keep in mind this Fowler post has no science in it. It's just some "lauded practitioners'" view of TDD. Our industry is driven by this kind of discourse. There's very little good scientific research to answer questions like these in software in general. How much is any good on TDD? One paper? Two?
    • nickbauman 6y ago
      Replying to myself (facepalm): just to make sure people know what I think. I've been doing TDD since 2002. I'm of the "You will take this out of my cold, dead hands" variety on the usefulness of TDD. That doesn't mean that there's any science that proves it.
• theptip 6y ago
  > When people talk about "unit tests", a unit doesn't refer to the common pattern of "a single class". A unit is a piece of the software with a clear boundary. It might be a whole microservice, or a chunk of a monolith that is internally consistent. This is a really important aspect, and I think is one of the key things that separates "journeyman" level from "master" on the subject of testing. The most concise piece I've found on this is Bob Martin's "Testing Contra-Variance" (https://blog.cleancoder.com/uncle-bob/2017/10/03/TestContravariance.html https://blog.cleancoder.com/uncle-bob/2017/10/03/TestContrav...), if you can look past the slightly forced "Socratic dialog" style of the article. > The structure of the tests must not reflect the structure of the production code, because that much coupling makes the system fragile and obstructs refactoring. Rather, the structure of the tests must be independently designed so as to minimize the coupling to the production code. The "one test class per class" or "one test file per file" approach is an extremely common anti-pattern, and it's insidious because a lot of engineers think it's the obviously correct way of writing tests.