6 ms·

Yeah even in the webdev world, the zeitgeist has shifted from "write as many tests as possible, 100% code coverage, in fact 100% branching code coverage. Use a

• sph 4y ago
  To add to Guillermo's mantra, a good suggestion from the video I posted is "write tests exercising the internal logic before a big refactor. Then remove them." Because otherwise, the more tests you have against implementation details, the more times your tests will break just because you've moved some code around. Tests need to go red when there is a bug, not because you've renamed a couple of private methods and now the mocks are broken.
  • arkh 4y ago
    To go farther: tests should not break during a refactoring. They're here to let you know if your refactoring broke something. They should even be green when you decide to scratch your software to rewrite it in another language. Code coverage? Use it to determine what is dead code you can remove: if your tests don't go through some code, that code is useless.
    • GTP 4y ago
      I agree with the first part of your comment, but not the second. Achieving 100% code coverage is very hard and sometimes unreasonable. E.g. How do you test a GUI? I think that often it is better to write tests for the application logic and test a GUI by using it.
    • anthonypasq 4y ago
      this is simply not possible. if you move a method from one class to another every unit test breaks.
    • 3a2d29 4y ago
      question: How do unit tests stay green when you rewrite in another language? Are your tests an external API you call?
      • wstuartcl 4y ago
        Rewrites are not super common -- but IMHO if you must do one the first step is to move the tests over. Unless your goal with the rewrite is to rearchitect the interfaces at which point your basically greenfielding anyways.
  • marginalia_nu 4y ago
    Depends a lot on what it is you're doing. Like if you're writing some finnicky library code, exotic algorithms or data structures or whatever, you'll probably want more test lines than code lines because that sort of stuff is notoriously difficult to get right. Even something as pedestrian as a binary search is downright gnarly to get working in all cases[1], the bugs are difficult to reason about and the code doesn't reduce to more comprehensible primitives. On the other hand, if you do that with application code, you're essentially submerging your code in a tar pit. Any change in behavior will require changing dozens of tests. Fixing broken tests will become so routine that they stop telling you anything about your code. Development will become "I changed from foo to bar, now let's update the 73 assertions that failed to whatever value the test runner says they have now". [1] https://stackoverflow.com/a/6393352 https://stackoverflow.com/a/6393352
  • hinkley 4y ago
    If you’re using a lot of mocks you’re gonna have a bad time. I think I can agree with the sentiment that mock heavy tests should be short lived, and why you don’t want to keep those around. Not sure I can agree with the rest. I aim for max of two stubs per test, and many of those end up with a TODO. Pure functions need no stubs, and most well factored code should only need at most one. One stub one test is pretty sustainable. It’s easy to rewrite such a test if the requirements change. Unit tests should absolutely be disposable, but you don’t dispose of them all at the same time. Just the ones that don’t fit the new rules. I do run into a steady stream of people who can’t seem to understand that the tests should affect the structure of your code. “And then a miracle happens” is what you have there - long intervals where the system produces no verifiable state or output is bad. That’s not an architecture. It’s lack of it. A functional programming style makes this easier to avoid, but it’s not a cure, because the disease is in their heads, the code is the symptom. There’s a substantial overlap between people with untestable code and people with undocumentable code. They can’t explain the code to the test framework any better than they can explain it to each other. All that said, testing is hard. It shouldn’t be this hard and we need to keep looking for ways to improve that situation. But even here we have people who reach for the least expressive solution quite frequently, such as assert over more reflective matchers, which make for much more useful red tests. At least BDD style seems to be winning out.
• icedchai 4y ago
  I worked at one place that had so many mocks, your tests were barely testing any real code. In one case, a dude checked in a 5000 line test suite for an "internal API." The server was mocked. All the tests were doing was checking that calls echoed back their own parameters. What was the point? Well, the API client now had 100% coverage.
  • hinkley 4y ago
    Going into new segments of our code, I’ve had to rewrite an awful lot of tests. It was a mess, and each one had at least a few tests that were only testing the mocks. Hand written mocks at that. Those people need to be stopped.
  • closeparen 4y ago
    This is 95% of tests I read and write at my job.
    • icedchai 4y ago
      It wasn't that bad at a previous job, but it was close. The sad part is nobody else would comment on the useless nature of these tests that didn't actually test anything.
      • closeparen 4y ago
        It’s very tricky to be against any kind of testing in a professional setting… opens you up for other engineers to question your maturity, professionalism, an and commitment to reliability. Or at least to look substantially more mature, professional, and committed to reliability than you. In front of management that can be death. People with a lot of clout have absorbed the virtues of automated testing in general and applied it to unit testing in particular. It’s hard to swim upstream on that one.
        • icedchai 4y ago
          It's true. I mean, I wouldn't comment on them either. I'd just roll my eyes when asked to review another 4000 line "test suite" that upped the coverage but did not test anything meaningful. I wrote many useless tests myself, assigned tasks like "upping test coverage for module XYZ." They'd all get thumbs up and looks-good in reviews.
• _whiteCaps_ 4y ago
  https://www.npr.org/2008/01/01/17725932/in-defense-of-food-author-offers-advice-for-health https://www.npr.org/2008/01/01/17725932/in-defense-of-food-a... Just want to point out that advice is inspired by Michael Pollan's diet advice: "Eat food. Not too much. Mostly plants."
• wankle 4y ago
  Completely agree. Integration tests over unit tests all day.
  • wizofaus 4y ago
    Are your integration tests able to run quickly/reliably enough to include them in your ci pipeline?
    • wankle 4y ago
      Yup. Very much so. In Java. Can't speak for other languages.
    • jeltz 4y ago
      Yes, and in my experience from the Ruby world integration tests are often as fast or faster than unit tests.
• apaprocki 4y ago
  To play devil's advocate here, what usually suffers the most in integration-only testing is testing error handling. In some codebases it's extremely important to test all the error handling paths because they are not extremely exceptional events. Many times, the disproportionate amount of failures are from error handling not precisely doing the right thing even though it looks reasonable upon code review. It might not mean 100% coverage, but in these situations, ensuring comprehensive tests of all the error conditions (inducing OOMs and other resource limits) helps make sure the failure paths work just as well as the normal ones. Failure of a failure path can manifest at best in ways such as hiding the true source of an issue, telemetry problems, logging statements without salient values; at worst, crashing because of a chain of events triggered by the seemingly innocuous error handling that passed a code review. OOM modeling to test every code path that allocates in a function is the most tedious thing in a gtest, but often yields the most surprising, actionable fixes. Sadly, most programmers just ignore OOMs and hand-waive "if allocation fails, the system is broken and my software doesn't need to work correctly".
  • bvrmn 4y ago
    This. So many "do not cover" hints for exception handler blocks. So many lost root causes when handler tries to format a error.
• hn_throwaway_99 4y ago
  For the webdev world, I think a big reason for this change is that many web systems talk to so many underlying 3rd party services that often times it's subtle, backwards incompatible changes in those other services that break things. If you're just only testing everything with mocks, you're never going to catch what will often be the worst breakages and bugs that your users experience.
• mejutoco 4y ago
  I think better type systems, like the one in Typescript have made mainstream the use of types to remove some basic bugs. This makes some basic tests not needed anymore, and gives more time to focus on the important ones.
• KronisLV 4y ago
  > Write as many tests as possible, 100% code coverage, in fact 100% branching code coverage. In my eyes this still hold true. Software that has expected behavior should be tested to make sure that the behavior isn't broken due to changes to any of the involved hot code paths or data formats. I once wrote a code library for some boring business system that handled integration between the system and a JWT library, which would make sure that certain requests should be serviced. Using a library without tests wouldn't be acceptable (security related/adjacent code is perhaps the best example of such circumstances). Neither would my code not having tests, either, given that this library would be used across multiple services within the system. Thus, I wrote tests until I got pretty close to 100% coverage and doing that actually helped me discover a few bugs while I was actually writing the tests! Not only that, but once the need to refactor something arose due to changing requirements, the tests breaking told me exactly what I had overlooked while doing those changes. Not only that, but if I'm long gone and someone comes to make changes to the library, the CI will tell them about the things they might overlook themselves, aside from any boring Wiki that they wouldn't read or other docs. The tests also demonstrate all of the ways how the code can actually be used, so aside from the occasional code comments, they also serve as living documentation. There absolutely are cases where testing something won't be viable (e.g. different file systems the code implementations for which depend on the runtime that's installed on the system, whereas all you get is a leaky abstraction in front of these and your test setup doesn't contain every covered platform, for example, checking which file paths are parsed as valid and which aren't across different file systems on different platforms), but in most systems they're not the majority. > Use an absurd amount of mocks to achieve this You also hit the nail on the head here - this is a problem and a symptom of us perhaps developing all of our systems wrong. The main reason for not writing tests (one that I can understand) is the fact that it's not easy to do so. You end up with various mocking frameworks and libraries that try to take away some of the pain caused by the fact that your entire system is not testable, but end up with more complexity to dance around in the end. I think the only way around this is to do data driven design that's coupled with functional programming in ample amounts, with as many pure functions as you can get. This would be completely un-idiomatic for many of the languages out there (e.g. those that rely on injecting services/repositories/whatever in fields, instead of passing everything a function needs in the parameters), but is also the only way how you could make testing easier. Maybe passing interfaces to "services" (many seem to use the service) pattern would be wrong and instead you'd need to pass in separate methods that your code will use. So instead of passing in UserService you'd pass in UserService::getUserById. So in a sense, it's a struggle to find the balance between code that is absolutely untestable without being in mock hell, to the point where you test mocks and your tests are useless and ending up having to write code that goes fully against how things are done in any given language and the frameworks you'll use within it, probably ending up with more code meant for decoupling those parts than you have the time to maintain. > write tests, not too many. mostly integration In an imperfect world, I guess we can just pretend that this is okay, because it will give you the most results, compared to the amount of work you need to put in. At the end of the day, nobody wants to pay 10x more for systems that are nearly perfectly tested, they just want something that is vaguely decent and will accept hand-wavy apologies for everything constantly breaking, as long as the breakages are small and non-critical enough. Devs also don't seem to typically enjoy writing tests, in part due to some systems not being testable easily, but also because of many tools, in particular mock frameworks and even integration testing tools (like Selenium, which thankfully has more and more alternatives), just being unpleasant to use.
• hinkley 4y ago
  So we’ve forgotten the lesson of the testing ice cream cone already. Good to know.