16 ms·

Write tests. Not too many. Mostly integration (2017)

perfunctory 8y ago
I believe that contrary to the conventional wisdom one should write tests from top down. First integration tests, then move down to testing individual functions if necessary. Not the other way around. On my current project, for the first time in my long career, I have 100% code coverage. How I achieved it? By ignoring best practices on what constitutes a unit test. My "unit" tests talk to the databases, read and write files etc. I'll take 100% coverage over unit test purity any day of the week.
- evv 8y ago
  The TDD style bottom-up approach can be handy too. On a recent project I write most of the tests first, to consider the API before implementation. These should come in handy when future developers join the codebase and need to understand the intended behavior of these components.
  - sorokod 8y ago
    Would a future developer care if you wrote your tests before or after the API was implemented?
    - xiphias2 8y ago
      Actually yes, it's really hard to write a test without mocking for an existing (non-pure) API.
- fwip 8y ago
  100% coverage doesn't mean much if you aren't testing that the low-level code is doing the right thing. For example, imagine you're testing a calculator app. Your integration tests make sure that it never crashes, the UI works, basic math is working, etc, but maybe the sin() function is only accurate to two decimal places. Edit: I do not mean to imply that you, specifically, are missing things. Rather, it is possible to write tests that have 100% code coverage while missing many possible bugs, and I think those risks are increased without the presence of unit tests.
  - perfunctory 8y ago
    > 100% coverage doesn't mean much That's absolutely right. 100% coverage doesn't say much. In fact, by itself it's a pretty meaningless measure of code quality. But the top-down idea still holds. In practice bugs are more likely to happen at the seams between components, and therefore it's where one should test first, not last. It of course depends on the type of the project but I believe it's true for the majority of projects out there.
    - achamayou 8y ago
      100% means nothing, but 30% means something is wrong. Coverage tells you nothing, but the absence of coverage is interesting and useful.
- jmromer 8y ago
  Top-down / outside-in test-first development is BDD.
- bluntfang 8y ago
  how big is your project? what's your test suite run time? I honestly think unit test purity is very worthwhile if you have a large project. It means the difference between a 1 build time vs a 6 minute build time.
  - alayne 8y ago
    What is your stack? I feel like you are doing something odd if total build times are a bottleneck on development.
- pmarreck 8y ago
  Repeatedly talking to the database (i.e. "retesting known database functionality over and over and over") is not a problem intrinsically. If you have a smallish codebase it's not a problem, that is. But get into the hundred-thousand-line and beyond point, and they become a very serious time-consumer on test runs, which need to be fast for high team productivity. Learned this the hard way by working on a million-line codebase for a couple years on a team. 40 minute test suite runtimes before you know if you broke something become a serious flow-breaker. It is of course up to you if you want to design for scalability in the code sense, however.
- regularfry 8y ago
  Interestingly, if you go right back to Kent Beck's TDD book, you'll see you're in agreement with him. Also with rails testing as it comes out-of-the-box. Testing individual functions is orthogonal to testing a unit.
maccio92 8y ago
Wow honestly maybe rethink your upsells here.. literally had to scroll half way through the page to get to the actual article. And what's the purpose of having a full screen splash image of a pineapple with sunglasses..
- frou_dh 8y ago
  https://news.ycombinator.com/item?id=9238739 https://news.ycombinator.com/item?id=9238739
- dajohnson89 8y ago
  That pineapple isn't even ripe.
- wishinghand 8y ago
  Does Medium force unrelated images into headers? I've never written on it but I can't remember the last time I saw a relevant header image.
  - dguo 8y ago
    You don't have to add a header image, but doing it no matter what seems to be the general thinking on Medium, presumably to get people's attention. See freeCodeCamp's guide on writing for them: https://medium.freecodecamp.org/how-to-get-published-in-the-freecodecamp-medium-publication-9b342a22400e#590c https://medium.freecodecamp.org/how-to-get-published-in-the-... "Medium will automatically make the first image in your article its featured image. This means your article’s og-image meta property will be this image. This image will serve as your story’s ambassador everywhere: social media news feeds, Reddit, Google News — even RSS readers." "Start considering images early in the writing process. And never publish without at least one image. Otherwise your story will be all but invisible in news feeds." I agree the header images are rarely relevant, and I wish they weren't considered must haves.
- aboutruby 8y ago
  80% of the article is unrelated images and self-promotion but some of the content is quite good.
calebegg 8y ago
It might be obvious to some, but for reference the title (and associated tweet) is a reference to Michael Pollan's oft-quoted advice for eating healthy: "Eat food. Not too much. Mostly plants." https://www.nytimes.com/2007/01/28/magazine/28nutritionism.t.html https://www.nytimes.com/2007/01/28/magazine/28nutritionism.t...
- adpirz 8y ago
  Was hoping the article would reference it, clearly a nod to Pollan!
- dharmach 8y ago
  With intermittent fasting concepts, now it is more like "Eat food. Not too frequently. Mostly plants."
- IfOnlyYouKnew 8y ago
  Which, just in case it isn’t obvious, is a timeless classic. Even if you disagree with the advice itself, the argument he makes and the brief survey of the history of nutritional science make it a worthwhile read. As this article shows, the mode of his argument has also been influential: it is a perfect case study of skepticism without cynicism; or considering competing arguments while avoiding the trap of “bothsiderism”. Matt Levine, another writer held in high regard in this community, is very similar in this regard. There is also a book expanding on the article.
  - malkia 8y ago
    This is the first time I'm reading (and "hearing") the phrase "bothsiderism" :) I love it! Definitely bothsidering to use it from now on!
    - killjoywashere 8y ago
      bothsiderism is essentially the outcome of what is in my mind the most infamous vendor lock-in move of all: the FCC's equal time rule. By requiring broadcasters give equal time to both sides, the ruling American parties (Democrat and Republican) effectively granted themselves a duopolony on mass media in the US and excluded all competing views. This is also the kernel of truth at the origin of the highly manipulable term "mainstream media".
dang 8y ago
Discussed at the time: https://news.ycombinator.com/item?id=15565875 https://news.ycombinator.com/item?id=15565875
hathawsh 8y ago
My rule of thumb: tests cast away fear. Whenever I get that sinking feeling that I'll break things when I change the code, I write tests until my doubts disappear. It works every time. In response to the article: it's true that "you should very rarely have to change tests when you refactor code", however, most of the time, most coders are changing the expected behavior of the code due to changed or added requirements, which is not refactoring. Tests should change when the requirements change, of course. I am not contradicting the article, only clarifying that the quoted statement does not apply outside refactoring. (Refactoring is improving the design or performance of code without changing its required behavior.)
- duncanfwalker 8y ago
  The level of experience of the people writing (and maintaining) the code is also a factor I think. As other commenters have said it's all about risk reduction. I definitely agree that you can get a lot of value writing integration tests. At the same time I'm a slightly concerned that if I'd read this article when I was first starting out as a developer I'd have thought unit tests were a side-note or a chore, rather than the building blocks for an application.
- delusional 8y ago
  Doesn't a lot of refactoring involve change in responsibility between classes? At least I find that a lot in my code. In those cases, the interface of those classes might change (as the responsibility shifts elsewhere), which will of course cause test changes.
  - hathawsh 8y ago
    My understanding is that refactoring may require changes to unit tests, but not integration tests. On the other hand, there have been long, unfruitful Internet discussions about the precise definition of refactoring, and I find it's better to just find an agreement among your immediate peers. :-)
  - marcosdumay 8y ago
    Why are you testing how the responsibility is divided between classes? Yes, some times you have to make an unclean cut into your code to test something. But this is not the default situation. That's why the article has that "mostly integration" part on its title.
jb3689 8y ago
The number of times I've said "oh I bet this works" just to be wrong when I wrote the tests is countless. For me it's more about having small well-defined interfaces with very strong tests Moreover, tests that don't break are useless
- munk-a 8y ago
  Though brittle tests are also useless, we have some snapshot result tests (take a large set of assumptions, push it through the pipes, get a large result set - in our case it's a webpage, but this holds for other settings) and almost every change to code covered by one of those tests breaks it, then the developer updates the test to work again, was their change logically correct - it can be hard to tell if 500 lines of a 2k expected result changed in minor ways. So a too-brittle test can be valueless.
stcredzero 8y ago
The current version of Unit Testing came out of the Chrysler C3 project, which was written using VisualWorks Smalltalk. (Eventually sparking Extreme Programming and SUnit, which was the ancestor of JUnit in Java land.) Here's the thing about Unit Testing in an environment like that. The best way to code and refactor code would also automatically refactor all of the Unit Tests. In an environment like that, Unit Tests are pretty nimble. There are no long waits for compile times. The entire test suite can run at the press of a button from a widget integrated in your standard development environment. Though, from what I read, the entire Unit Test suite would take an entire 10 minutes to run. However, you could easily just run the tests for the classes you were working on at the time, and reserve the whole suite for checkin time. So what happens when you move the practice away from this particular kind of Smalltalk environment? Refactorings in most languages are slower without the Refactoring Browser, and often your Unit Tests effectively double the amount of work involved. The velocity of change slows down. Unit Tests might be less nimble to run. A long compile time might be involved. Given those changes, it makes perfect sense that a larger granularity of tests and fewer tests might be more convenient.
- keithnz 8y ago
  it may been born in the smalltalk world, but it really was worked fleshed out and challenged and hashed and rehashed in the world of java ( and mirrored in other languages like C++, C#, and ruby ). The XP forum was very active in building what it meant to unit test and how to do test first development, it was highly motivated by the idea of being able to robustly respond to change with quick feedback loops. What became apparent is that the smalltalk world of small modular highly composable designs is kind of critical and much of the C++ / Java world struggled to achieve that, but there was a lot of advice on design techniques. Over time what's been left is more of an emphasis on test than design when talking about that pros and cons of unit testing. So we now want to validate our software works more than we want techniques to quickly evolve our designs and adapt to change in a robust way. Both achieve the idea of working software. So unit testing as a technique to test your software works may not be as good as integration testing and E2E testing, but that wasn't its sole goal, adaptable design was. Not that design has been left behind, if anything we are seeing quite a lot of focus on things like functional programming and new design ideas for putting software together. The key thing is not really the dogma of practices but the ideas of working software and adapting to change and that the idea of robustness and good design is embraced in the core of what you do.
pmarreck 8y ago
I would say mostly unit and very few integration, if the app codebase is designed well, anyway
notacoward 8y ago
I so agree with this. I've been fighting this exact battle at work lately. People on my team have decided to take testing seriously, which is fantastic, but many team members' understanding of what that means is still at the "watch unit-test coverage numbers go up" stage. So let me be very clear where I stand. * 100% unit-test coverage is a garbage goal. * I don't hate unit tests. They can have enormous value shaking out edge cases in a self contained piece of code - usually a "leaf" in the module dependency graph - and making it future proof. Love it. However, unit tests don't tell you anything about whether the module's behavior in combination with others leads to a correct result. It's possible to chain a bunch of unit tests each with 100% coverage together, and still have the combination fail spectacularly. In operations, a lot of bugs live in the interstices between modules. Even worse, 100% line coverage means a lot less than people seem to think. Let's say one line to compute an expression will compute the wrong value for some inputs. You might have a zillion unit tests that cause that line to be counted as covered, but still be missing the test that reveals the error case. I see this particularly with ASSERT or CHECK types of macros, which get counted toward 100% coverage even though no test ever exercises the failure case. Striving too hard for 100% unit test coverage often means a lot of work - many of our unit tests have 3x more tricky mock code than actual test code - to get a thoroughly artificial and thus useless result. The cost:benefit ratio is abominable. By all means write thorough unit tests for those modules that have a good ratio, but in general I agree with the admonition to focus more effort on functional/integration tests. In my 30+ years of professional program, that has always been a way to find more bugs with less effort.
- arkh 8y ago
  Test coverage is useful: it helps you see what code you should remove as it is never hit.
  - notacoward 8y ago
    No argument there. Thanks for pointing that out. But is it worth writing a hundred lines of mock code trying to get at it before concluding it's dead? I suggest that there are more efficient ways to deal with that particular issue.
  - 8y ago
jamougha 8y ago
We have a new guy at work who loves unit tests. Yammers about them constantly. If he's working on a feature that needs changes to classes A, B and C then his workflow is: make all the changes to class A, along with lots of unit tests. Then do the same for class B. Finally, do the same for class C. And then, after days of work during which the application doesn't even compile, he runs the integration tests and discovers that the application is broken. Then someone comes over and rewrites his code so it works.
- munk-a 8y ago
  So... that rewrite of his code is pretty quick and easy because the components you're integrating are well covered by tests at that point? Unless the tests on A, B and C are ineffective I think this comment isn't really about tests - some hired developers just aren't great at writing software but if they all used vim that wouldn't mean the cause of their errors was the fact that they used vim.
  - jamougha 8y ago
    The tests are ineffective. That's because, as the article notes, unit tests - tests of single functions, with dependencies mocked - give very little confidence that code actually works. Problems usually appear in the relationships between parts. And because they're tightly bound to the internal implementation of the code they test the tests often have to be rewritten when the code is rewritten. Does this mean unit tests are bad? No. They way they're frequently used I believe they are bad, but you're correct that that wasn't the point I was making. I'm really commenting on the way that common wisdom pushes gimmicky methodologies but doesn't talk about the fundamentals of how to make code changes in an effective way.
    - munk-a 8y ago
      I agree with you then - there is nothing inherently beneficial from the quantity of unit or integration tests you have, the quality of the tests to accurately target specific edge/error cases is what makes them valuable. As an aside I find TDD a particularly bad trend for these sorts of bad habits, TDD is a great idea in theory, but you'll often see TDD unit tests come out mimicking the code under test, i.e. "How to test if this function containing `return n + n ^ 2` is correct... well, let's generate some random numbers and then see if the output is equal to that random number plus it's square! That's like full coverage!" Having tests that duplicate the code under test is a pretty easy trap to fall into with micro-tests and it should make you suspicious of any test that covers a very small unit of code. This is similar to a statistical problem involving over-fitted trend lines, you can construct a mathematical formula that can go through any number of arbitrary (X,Y) points, leading to an average deviance of 0, but this formula will probably break as soon as you take another sample.
tretiy3 8y ago
This is good. But russian translation is terrible (were translations some marketing move?). Please ignore translation. Read original only
Dangeranger 8y ago
As a counter point to this opinion of Kent's, who I respect greatly, I'd like to propose J. B. Rainsberger's article: "Integrated Tests Are A Scam" with presentation of the same name.[0] [0] https://blog.thecodewhisperer.com/permalink/integrated-tests-are-a-scam https://blog.thecodewhisperer.com/permalink/integrated-tests...
mpweiher 8y ago
Counterpoint: Integrated Tests are a Scam https://blog.thecodewhisperer.com/permalink/integrated-tests-are-a-scam https://blog.thecodewhisperer.com/permalink/integrated-tests... I found that argument more convincing, and it also aligns better with my experience. The point about 100% coverage not being a good goal is pretty solid.
- wenc 8y ago
  For the benefit of others, it's also worth pointing out that the author isn't talking about "integration tests" but "integrated tests" (as he defines it). Integration tests = testing individual software modules as a group/combination. Integrated tests = instead of testing one behavior at a time, multiple behaviors are tested simultaneously. (or in the author's words: "any test whose result (pass or fail) depends on the correctness of the implementation of more than one piece of non-trivial behavior.") Integration tests are very important.
  - mpweiher 8y ago
    It is also worth pointing out that the author used to use the term "integration tests" in this post, and only changed to "integrated tests" after writing. So a claim that these are completely distinct/disjoint ideas is not really supportable. As far as I can tell, he doesn't explain the name change, but having watched the talk before he changed the name, it looks like "integrated" is a somewhat more general term. Specifically: if you are "testing individual software modules as a group/combination", how are you going to do that without "depending on the correctness of the implementation of more than one piece of non-trivial behavior." I see exactly one way: your "integration" tests have to be trivial, really more akin to characterisation tests. "When I hit this external API with this value, do I get this reaction?" As far as I can tell, he doesn't talk about this in the post, but he does in the talk/video. These are interface/boundary tests. So you test each unit in isolation, and you test the boundaries. If your unit tests are sufficient and your boundary tests are sufficient, plugging things together will work. So no integrated/integration tests. Now I am actually not that strict, I let my boundary tests leak a little towards integration/integrated tests, because it is possible to just not hook up the components right. So you want to check that with some smoke tests. However, once they are hooked up, the more complex behaviours Just Work™.
1337shadow 8y ago
Maybe but do you think 100% coverage is such a bad experience to have at all ? This can be a rich exercise, and a rite of passage to know what tests matters, which should be automated (fuzzers, dbunit etc), which can wait, etc... and how to get to 90% with the effort of doing doing only 30. Integration tests should write themselves. It depends on the projects, but personally i use to get 65% of coverage for the price of 5%. For example you want to test a bunch of commands, then you can make a function like autotest('some command', 'some_fixture.txt'): call "some command", capture the output, and write some_fixture.txt with the captured output, and fail complaining that it had to create some_fixture.txt. The next run it will find some_fixture.txt and compare the output and fail only if it differs. Unit tests should of course be hand written, but to cover everything that matters like for bugs, or for what you want to refactor in TDD. Of course any line that's not covered can potentially break when upgrading versions, but this kind of breaks are likely to be revealed by the 90% of coverage that I think you can get with minimal effort by applying this recipe. Then, you can afford 0day updates when underlying libraries upstream make a new release candidate.
thom 8y ago
The thing that changed my approach to testing, at least in OO languages, was being shown that use cases/tasks/"things the system can do" should be first class objects in the model. At that point, you have a single layer that exposes everything of real value in the application, giving you a really simple test surface. Underneath that surface you can refactor to your heart's delight, without worrying about having to maintain lots of pointless unit tests for trivial behaviours on each object - all that matters is maintaining the end to end functionality. So yes, I agree integration tests are the key, and you can architect your application to make this easier. Not over-testing your underlying model is just good old fashioned information-hiding. Testing the UI on top I leave as a matter of taste.
castlegloom 8y ago
Yeah this is article comes out and says that it's based on gut, not science. So speaking for anecdotal evidence, unit tests that don't break and frequent deploys are way better than finessing one more Gherkin test.
kords 8y ago
100% coverage doesn't prove the code is fully tested, just like having unit/integration/acceptance/smoke and other types of tests doesn't prove that application works. But this doesn't mean we stop adding tests to our suite, so why stopping adding more coverage? I totally agree with "just stop mocking so much stuff". Most of the time we can use real implementation and by doing this, we'll also increase coverage.
munificent 8y ago
I think one mental model for tests is that they're simply another kind of automation. The fundamental question of automation is: will I repeat this task enough such that the amortized saving outweighs the cost of writing a script to do it? Whenever you're thinking of adding a test X, quickly consider how often you and your team are likely to need to manually test X if you don't. Also factor in the cost of writing test X (though that's tricky because sometimes you need to build test architecture Y, which lowers the costs of writing many tests...). If it's a piece of code that's buggy, changing frequently, brittle, or important, then you're likely to need to validate its behavior again and again. It's probably worth writing a unit test for it. If it's an end user experience that involves a lot of services maintained by different teams and tends to fall apart often, it's probably worth writing an integration test instead of having to keep manually testing every time the app goes down. If it's an API with lots of important users using it for all sorts of weird edge cases and you don't want to have to manually repro each of the existing weird edge cases any time you add a new one, it's probably worth writing some black box tests for it. But if it's code that's simple, boring, stable, short-lived, or unimportant, your time may be better spent elsewhere. Another model is that your tests represent the codified understanding of what the system is. This can be very helpful if you have a lot of team churn. How do you make sure a new person doesn't break something when "something" isn't well-defined somewhere? Tests are a great way to pin that down. If this is a priority, then it becomes more important to write tests for things where the test actually very rarely fails. It's more executable system specification than it is automation.
- chriswarbo 8y ago
  I agree with the idea that we're mostly automating manual work. The really nice thing about automated tests is that they can also go much further than I would ever bother to manually. For example, I might manually check that a couple of product pages are showing, with an image, price and description. With automated testing I can check all product pages, every time. It can also test a tedious amount of interactions, e.g. various interleavings of "add to cart", "remove from cart", "checkout", "press back button", etc. Manual testing still gives me confidence that automated tests can't, e.g. that an "unknown unknown" hasn't taken things down. Yet that's all I need to check manually; if pages are loading and look right, I don't feel a need to manually check all of the ecommerce arithmetic, permission logic, etc.
aboutruby 8y ago
It's confusing with Rails, because integration tests are now system tests, and the kind of integration tests described in this article are named controller and requests tests.
luord 8y ago
> you find yourself spending time testing things that really don’t need to be tested The thing with 100% code coverage is not that one should write more tests, but that you should refactor everything that isn't business logic (and, thus, nothing that would be part of the functionality of the application to test) into third party libraries, if it doesn't already come from third party libraries, which it probably should.
NicoJuicy 8y ago
At work, the unit tests and data are locked, even the webserver. Every change of data or requirements causes them to fail or fail because of human error. At home, I do integration testing and it seems to work much cleaner and more complete.
gravypod 8y ago
> I’ve heard managers and teams mandating 100% code coverage for applications. That’s a really bad idea I hope many managers and programmers out there don't take this the wrong way. I've been an engineer on a project that was attempting to get 100% code coverage on a piece of software I was writing. I heard constant remarks during this period that were similar to "You don't need 100% code coverage, it doesn't do anything!" These engineers who I was working with had only read articles like this and didn't stop to think about what the article was trying to say. From my experience there is no safe rule of thumb for for how many tests should be implemented for a project. There should be just enough to feel safe (as hathawsh has said). If you're recommending to engineers on your team to stop implementing tests when they say "I'm at 60% coverage and it'll take ~2 days to get to 100%" I'd really hope you take the time to understand why they want 100% coverage. The software I was working on when my coworkers started telling me to stop writing tests was code designed to trigger alarms when patient's vital signs met certain criteria set by doctors. I am very thankful that I did hit 100% coverage because between 60% and 100% there were many small edge cases that, had they caused a death, I wouldn't have been able to sleep well. Had they said I was explicitly disallowed from working on the tests I would have come in on a weekend (or taking a PTO) and implemented them then. It's our ethical responsibility to know when and where paranoia is worth the marginal time penalty.
- bulkan 8y ago
  > ...trigger alarms when patient's vital signs met certain criteria set by doctor Most of us aren't working on life or death code like this. My React app doesn't need 100% code coverage but you put it well when you said "There should be just enough to feel safe"
  - gravypod 8y ago
    What I really want to drive home is that you should listen to the engineers who are working on the project. Testing is one of those things that is entirely subject to project needs. Like you say, a React app doesn't need 100% coverage, neither does code running a CI, something far out of the serving path, etc. But we shouldn't have a knee-jerk reaction to an engineer saying we should have 100% coverage.
garganzol 8y ago
Those sticky banners and elements injected by Medium are ridiculous. They make a good article hardly readable on HD desktop monitor. Sad state of things, I remember how once upon a time it was a great platform to be at.
camnora 8y ago
I’m not so sure I agree. Often isolated TDD can lead to much cleaner code and succinct solutions. There is also a benefit of quicker test execution. I’ve seen a noticeable difference in the quality of software delivered to my clients with this approach. Gary Bernhardt covers these topics in great depth and highly recommend to anyone interested.
neo2006 8y ago
I agree with the writer about the statement that you don't need a 100% test coverage but you still need to check your coverage. I mean you need to run coverage and go through your code and see if there is any gap in your testing. That said I don't agree with the statement "mostly integration" the reason is that he don't take into an account the ROI of a test, he only take into account the outcome. I mean e2e tests are the best to catch bugs but they are harder to achieve harder to debug and harder to maintain. Same go for integration test they are harder to debug, maintain and perform than unit test. Developpers forget that their time is money and that if they spend time on a test just because it make them feel safe that mean that the project will cost more money. The general rule I use is simple. When you test, test behavior not code and do your test at the lowest level possible.
trixie_ 8y ago
Integration tests really are the best bang for the buck. So many times people think they tested their code by mocking something like the database layer and are surprised when the application in production breaks. Everything can be tested in isolation and work perfectly, but often it's how the components work together that determines if the system works at all. Integration test what you can. Unit test core pieces of logic. Avoid mocks.
- sime2009 8y ago
  Fully agree. The majority of us are writing code with has to talk to a lot of other systems which we didn't write and don't control. These interface areas between systems are huge gathering places for bugs. Also, mocks are self-deception at best and lies at worst.
je42 8y ago
have to disagree here... not writing unit tests means architecture change will break your integration tests. hence, you don't have coverage left during an architecture change. typing information on a unit (class or function) is nice, except it doesn't find all bugs. If it were to find all bugs about a unit, it would be a complete type checker ( which usually comes in a system like https://en.wikipedia.org/wiki/Isabelle_(proof_assistant) https://en.wikipedia.org/wiki/Isabelle_(proof_assistant), etc, which is a pain to use... ) thus the only "efficient way" to cover the gap between type information and a full blown type checking system are unit-tests. integration test run slower, hence, you are also wasting more time. integration test are more difficult to understand by other people, especially if they break. integration test are generally less stable than unit-test because they involve more units. integration tests usually don't cover all (error) scenerios of the involved particular units. you do that in unit-test to get the branch coverage to 100%. his issue regarding mocking potentially can be solved differently. change your interfaces and code architecture such that you minimize mocking and make unit-testing easy. i.e. experiment moving the boundary between units.
sivanmz 8y ago
Maybe the author of Socket.io and Zeit.co just happens to work on integration problems?
deleted 8y ago
[deleted]
raverbashing 8y ago
100% unit test code coverage is just Uncle Bob BS and it is infeasible for the most part And while it will give some warranties about correctness, it will not fundamentally guarantee the application does what is supposed to do. Integration tests are certainly better I can only see the headline being a "great wisdom" to people who have accepted Uncle Bob and TDD crap for too long and without questioning it. Because it is obvious. Why it is obvious? Because that's how things were done most of the time When you had 640k of RAM and a C compiler writing unit tests was not impossible, but pretty hard. Testing that your app reacted to inputs and acted correctly was doable and easily automatable. And what wasn't automatable would be tested manually. Now here comes the "holy highnesses" of testing gurus, gaslighting developers and saying that code with no tests (which tests? automated? unit? the definition is purposefully vague) doesn't work or that the only blessed code is the one that is produced through TDD onanism? No thank you
kwhitefoot 8y ago
I know this is off topic but why is it that people who, presumably, want us to read their words of wisdom put such a pile of crap at the top of the page that I have to hit the page down key five times before I get to the article?
jondubois 8y ago
>> Write tests. Not too many. Mostly integration. If a group of reputable programmers got together and published a book about programming which had a single page which only contained this line in a large font, it would be the most useful and valuable programming book ever written.
- edem 8y ago
  Definitely not. See my comment on the parent.
edem 8y ago
My comment from the original post verbatim: I agree with the part that you should write tests, but I definitely disagree with the part that most of your tests should be integration tests. As you pointed out the testing pyramid suggests that you should write more unit tests. Why? Because if you have ever tried TDD you know that unit tests make you write good (or at least acceptable) code. The reason for this is that testing bad code is hard. By writing mostly integration tests you lose one of the advantages of unit testing and you sidestep the bad code checking part. The other reason is that unit tests are easy to write. If you have interfaces for your units of code then mocking is also easy. I recommend stubbing though, I think that if you have to use mocks it is a code smell. Also the .gif with the man in pieces is a straw man. Just because you have to write at least 1 integration test to check whether the man has not fallen apart is not a valid reason to write mostly integration tests! You can’t test your codebase reliably with them and they are also very costly to write, run and maintain! The testing pyramid exists for a reason. It is a product of countless hours of research, testing and head scratching. You should introspect your own methods instead and you might arrive at the conclusion that the codebase you are working on is bad and it is hard to unit test, that’s why you have chosen to write mostly integration tests.
blauditore 8y ago
I think the root of many testing problems is that people write code and continuously test manually until they think it works correctly. Then, they start writing some automated tests. What I think is much more effective is to write test instead of manual testing while developing. That is, what you would do by hand just implement in a test. And only at the very end, double-check manually.
- executesorder66 8y ago
  Isn't that just TDD?
  - blauditore 8y ago
    Similar to it, but not as strict. TDD implies that you write tests first, which also leads the designing of your application code. What I'm describing is more general, where testing happens alternatingly with writing domain code, "roughly" at the same time.
vbsteven 8y ago
This is pretty similar to my approach. In the context of developing an API I use integration tests more than unit tests. It is straightforward to spin up an application server against a test database with some test data. Run some example requests against it and verify the results. I frequently have Test classes which contain a full scenario related to a specific resource or action. For example: * create resource * lookup resource * modify resource * list all resources * delete resource The JSON files for the integration tests are then used as examples in the API documentation. Unit tests are reserved mostly for verifying business logic components. There is no need to setup a complex ControllerTest with mocked out Services or Repositories as you don't care about the internals anyway. Just the input vs output and the integration tests cover those already.
ttty 8y ago
> Well, when you strive for 100% all the time, you find yourself spending time testing things that really don’t need to be tested I noticed people skip testing what is hard to test and just test what it's easy... Not what really matters
SomeHacker44 8y ago
Even at 100% code coverage it is possible to have missed many code paths that involve nonlocal control flow. The most common issue here would be exceptions in languages that support it. In most operating systems there are various interrupts that a program can also encounter (e.g., a lovely SEGFAULT in many *nix type operating systems) and other operating system and hardware level issues (like OOM killer in Linux or an NMI or ECC memory error in embedded systems). So, 100% code coverage may not even be enough for some applications as was discussed in another too level thread (medical device) or other hard to fix (spacecraft) or life-threatening situations.
natashas 8y ago
In general automated tests are used mostly to save time as the earlier you find the bug the more savings you have when fixing it. When your testers find it, they need to create and fill the ticket, assign it to developer, then developer needs to reproduce, fix it and assign it back and tester needs to retest it. And it's much more expensive if it happens in production! Integration/unit tests can catch a lot of those. There are some diminishing returns so 100% coverage is not needed and integration tests are more effective in catching bugs so I agree with the article's idea. Use unit tests for some real units - algorithms, calculations etc, and don't just test mocking framework. With additional layer of end to end tests and manual testing system should be able to achieve pretty good quality without spending unreasonable amount of time for it.

I believe that contrary to the conventional wisdom one should write tests from top down. First integration tests, then move down to testing individual functions if necessary. Not the other way around. On my current project, for the first time in my long career, I have 100% code coverage. How I achieved it? By ignoring best practices on what constitutes a unit test. My "unit" tests talk to the databases, read and write files etc. I'll take 100% coverage over unit test purity any day of the week.

evv 8y ago
The TDD style bottom-up approach can be handy too. On a recent project I write most of the tests first, to consider the API before implementation. These should come in handy when future developers join the codebase and need to understand the intended behavior of these components.
- sorokod 8y ago
  Would a future developer care if you wrote your tests before or after the API was implemented?
  - xiphias2 8y ago
    Actually yes, it's really hard to write a test without mocking for an existing (non-pure) API.
fwip 8y ago
100% coverage doesn't mean much if you aren't testing that the low-level code is doing the right thing. For example, imagine you're testing a calculator app. Your integration tests make sure that it never crashes, the UI works, basic math is working, etc, but maybe the sin() function is only accurate to two decimal places. Edit: I do not mean to imply that you, specifically, are missing things. Rather, it is possible to write tests that have 100% code coverage while missing many possible bugs, and I think those risks are increased without the presence of unit tests.
- perfunctory 8y ago
  > 100% coverage doesn't mean much That's absolutely right. 100% coverage doesn't say much. In fact, by itself it's a pretty meaningless measure of code quality. But the top-down idea still holds. In practice bugs are more likely to happen at the seams between components, and therefore it's where one should test first, not last. It of course depends on the type of the project but I believe it's true for the majority of projects out there.
  - achamayou 8y ago
    100% means nothing, but 30% means something is wrong. Coverage tells you nothing, but the absence of coverage is interesting and useful.
jmromer 8y ago
Top-down / outside-in test-first development is BDD.
bluntfang 8y ago
how big is your project? what's your test suite run time? I honestly think unit test purity is very worthwhile if you have a large project. It means the difference between a 1 build time vs a 6 minute build time.
- alayne 8y ago
  What is your stack? I feel like you are doing something odd if total build times are a bottleneck on development.
pmarreck 8y ago
Repeatedly talking to the database (i.e. "retesting known database functionality over and over and over") is not a problem intrinsically. If you have a smallish codebase it's not a problem, that is. But get into the hundred-thousand-line and beyond point, and they become a very serious time-consumer on test runs, which need to be fast for high team productivity. Learned this the hard way by working on a million-line codebase for a couple years on a team. 40 minute test suite runtimes before you know if you broke something become a serious flow-breaker. It is of course up to you if you want to design for scalability in the code sense, however.
regularfry 8y ago
Interestingly, if you go right back to Kent Beck's TDD book, you'll see you're in agreement with him. Also with rails testing as it comes out-of-the-box. Testing individual functions is orthogonal to testing a unit.

maccio92 8y ago

Wow honestly maybe rethink your upsells here.. literally had to scroll half way through the page to get to the actual article. And what's the purpose of having a full screen splash image of a pineapple with sunglasses..

frou_dh 8y ago
https://news.ycombinator.com/item?id=9238739 https://news.ycombinator.com/item?id=9238739
dajohnson89 8y ago
That pineapple isn't even ripe.
wishinghand 8y ago
Does Medium force unrelated images into headers? I've never written on it but I can't remember the last time I saw a relevant header image.
- dguo 8y ago
  You don't have to add a header image, but doing it no matter what seems to be the general thinking on Medium, presumably to get people's attention. See freeCodeCamp's guide on writing for them: https://medium.freecodecamp.org/how-to-get-published-in-the-freecodecamp-medium-publication-9b342a22400e#590c https://medium.freecodecamp.org/how-to-get-published-in-the-... "Medium will automatically make the first image in your article its featured image. This means your article’s og-image meta property will be this image. This image will serve as your story’s ambassador everywhere: social media news feeds, Reddit, Google News — even RSS readers." "Start considering images early in the writing process. And never publish without at least one image. Otherwise your story will be all but invisible in news feeds." I agree the header images are rarely relevant, and I wish they weren't considered must haves.
aboutruby 8y ago
80% of the article is unrelated images and self-promotion but some of the content is quite good.

calebegg 8y ago

It might be obvious to some, but for reference the title (and associated tweet) is a reference to Michael Pollan's oft-quoted advice for eating healthy: "Eat food. Not too much. Mostly plants." https://www.nytimes.com/2007/01/28/magazine/28nutritionism.t.html https://www.nytimes.com/2007/01/28/magazine/28nutritionism.t...

adpirz 8y ago
Was hoping the article would reference it, clearly a nod to Pollan!
dharmach 8y ago
With intermittent fasting concepts, now it is more like "Eat food. Not too frequently. Mostly plants."
IfOnlyYouKnew 8y ago
Which, just in case it isn’t obvious, is a timeless classic. Even if you disagree with the advice itself, the argument he makes and the brief survey of the history of nutritional science make it a worthwhile read. As this article shows, the mode of his argument has also been influential: it is a perfect case study of skepticism without cynicism; or considering competing arguments while avoiding the trap of “bothsiderism”. Matt Levine, another writer held in high regard in this community, is very similar in this regard. There is also a book expanding on the article.
- malkia 8y ago
  This is the first time I'm reading (and "hearing") the phrase "bothsiderism" :) I love it! Definitely bothsidering to use it from now on!
  - killjoywashere 8y ago
    bothsiderism is essentially the outcome of what is in my mind the most infamous vendor lock-in move of all: the FCC's equal time rule. By requiring broadcasters give equal time to both sides, the ruling American parties (Democrat and Republican) effectively granted themselves a duopolony on mass media in the US and excluded all competing views. This is also the kernel of truth at the origin of the highly manipulable term "mainstream media".

dang 8y ago

Discussed at the time: https://news.ycombinator.com/item?id=15565875 https://news.ycombinator.com/item?id=15565875

hathawsh 8y ago

My rule of thumb: tests cast away fear. Whenever I get that sinking feeling that I'll break things when I change the code, I write tests until my doubts disappear. It works every time. In response to the article: it's true that "you should very rarely have to change tests when you refactor code", however, most of the time, most coders are changing the expected behavior of the code due to changed or added requirements, which is not refactoring. Tests should change when the requirements change, of course. I am not contradicting the article, only clarifying that the quoted statement does not apply outside refactoring. (Refactoring is improving the design or performance of code without changing its required behavior.)

duncanfwalker 8y ago
The level of experience of the people writing (and maintaining) the code is also a factor I think. As other commenters have said it's all about risk reduction. I definitely agree that you can get a lot of value writing integration tests. At the same time I'm a slightly concerned that if I'd read this article when I was first starting out as a developer I'd have thought unit tests were a side-note or a chore, rather than the building blocks for an application.
delusional 8y ago
Doesn't a lot of refactoring involve change in responsibility between classes? At least I find that a lot in my code. In those cases, the interface of those classes might change (as the responsibility shifts elsewhere), which will of course cause test changes.
- hathawsh 8y ago
  My understanding is that refactoring may require changes to unit tests, but not integration tests. On the other hand, there have been long, unfruitful Internet discussions about the precise definition of refactoring, and I find it's better to just find an agreement among your immediate peers. :-)
- marcosdumay 8y ago
  Why are you testing how the responsibility is divided between classes? Yes, some times you have to make an unclean cut into your code to test something. But this is not the default situation. That's why the article has that "mostly integration" part on its title.

jb3689 8y ago

The number of times I've said "oh I bet this works" just to be wrong when I wrote the tests is countless. For me it's more about having small well-defined interfaces with very strong tests Moreover, tests that don't break are useless

munk-a 8y ago
Though brittle tests are also useless, we have some snapshot result tests (take a large set of assumptions, push it through the pipes, get a large result set - in our case it's a webpage, but this holds for other settings) and almost every change to code covered by one of those tests breaks it, then the developer updates the test to work again, was their change logically correct - it can be hard to tell if 500 lines of a 2k expected result changed in minor ways. So a too-brittle test can be valueless.

stcredzero 8y ago

The current version of Unit Testing came out of the Chrysler C3 project, which was written using VisualWorks Smalltalk. (Eventually sparking Extreme Programming and SUnit, which was the ancestor of JUnit in Java land.) Here's the thing about Unit Testing in an environment like that. The best way to code and refactor code would also automatically refactor all of the Unit Tests. In an environment like that, Unit Tests are pretty nimble. There are no long waits for compile times. The entire test suite can run at the press of a button from a widget integrated in your standard development environment. Though, from what I read, the entire Unit Test suite would take an entire 10 minutes to run. However, you could easily just run the tests for the classes you were working on at the time, and reserve the whole suite for checkin time. So what happens when you move the practice away from this particular kind of Smalltalk environment? Refactorings in most languages are slower without the Refactoring Browser, and often your Unit Tests effectively double the amount of work involved. The velocity of change slows down. Unit Tests might be less nimble to run. A long compile time might be involved. Given those changes, it makes perfect sense that a larger granularity of tests and fewer tests might be more convenient.

keithnz 8y ago
it may been born in the smalltalk world, but it really was worked fleshed out and challenged and hashed and rehashed in the world of java ( and mirrored in other languages like C++, C#, and ruby ). The XP forum was very active in building what it meant to unit test and how to do test first development, it was highly motivated by the idea of being able to robustly respond to change with quick feedback loops. What became apparent is that the smalltalk world of small modular highly composable designs is kind of critical and much of the C++ / Java world struggled to achieve that, but there was a lot of advice on design techniques. Over time what's been left is more of an emphasis on test than design when talking about that pros and cons of unit testing. So we now want to validate our software works more than we want techniques to quickly evolve our designs and adapt to change in a robust way. Both achieve the idea of working software. So unit testing as a technique to test your software works may not be as good as integration testing and E2E testing, but that wasn't its sole goal, adaptable design was. Not that design has been left behind, if anything we are seeing quite a lot of focus on things like functional programming and new design ideas for putting software together. The key thing is not really the dogma of practices but the ideas of working software and adapting to change and that the idea of robustness and good design is embraced in the core of what you do.

pmarreck 8y ago

I would say mostly unit and very few integration, if the app codebase is designed well, anyway

notacoward 8y ago

I so agree with this. I've been fighting this exact battle at work lately. People on my team have decided to take testing seriously, which is fantastic, but many team members' understanding of what that means is still at the "watch unit-test coverage numbers go up" stage. So let me be very clear where I stand. * 100% unit-test coverage is a garbage goal. * I don't hate unit tests. They can have enormous value shaking out edge cases in a self contained piece of code - usually a "leaf" in the module dependency graph - and making it future proof. Love it. However, unit tests don't tell you anything about whether the module's behavior in combination with others leads to a correct result. It's possible to chain a bunch of unit tests each with 100% coverage together, and still have the combination fail spectacularly. In operations, a lot of bugs live in the interstices between modules. Even worse, 100% line coverage means a lot less than people seem to think. Let's say one line to compute an expression will compute the wrong value for some inputs. You might have a zillion unit tests that cause that line to be counted as covered, but still be missing the test that reveals the error case. I see this particularly with ASSERT or CHECK types of macros, which get counted toward 100% coverage even though no test ever exercises the failure case. Striving too hard for 100% unit test coverage often means a lot of work - many of our unit tests have 3x more tricky mock code than actual test code - to get a thoroughly artificial and thus useless result. The cost:benefit ratio is abominable. By all means write thorough unit tests for those modules that have a good ratio, but in general I agree with the admonition to focus more effort on functional/integration tests. In my 30+ years of professional program, that has always been a way to find more bugs with less effort.

arkh 8y ago
Test coverage is useful: it helps you see what code you should remove as it is never hit.
- notacoward 8y ago
  No argument there. Thanks for pointing that out. But is it worth writing a hundred lines of mock code trying to get at it before concluding it's dead? I suggest that there are more efficient ways to deal with that particular issue.
- 8y ago

jamougha 8y ago

We have a new guy at work who loves unit tests. Yammers about them constantly. If he's working on a feature that needs changes to classes A, B and C then his workflow is: make all the changes to class A, along with lots of unit tests. Then do the same for class B. Finally, do the same for class C. And then, after days of work during which the application doesn't even compile, he runs the integration tests and discovers that the application is broken. Then someone comes over and rewrites his code so it works.

munk-a 8y ago
So... that rewrite of his code is pretty quick and easy because the components you're integrating are well covered by tests at that point? Unless the tests on A, B and C are ineffective I think this comment isn't really about tests - some hired developers just aren't great at writing software but if they all used vim that wouldn't mean the cause of their errors was the fact that they used vim.
- jamougha 8y ago
  The tests are ineffective. That's because, as the article notes, unit tests - tests of single functions, with dependencies mocked - give very little confidence that code actually works. Problems usually appear in the relationships between parts. And because they're tightly bound to the internal implementation of the code they test the tests often have to be rewritten when the code is rewritten. Does this mean unit tests are bad? No. They way they're frequently used I believe they are bad, but you're correct that that wasn't the point I was making. I'm really commenting on the way that common wisdom pushes gimmicky methodologies but doesn't talk about the fundamentals of how to make code changes in an effective way.
  - munk-a 8y ago
    I agree with you then - there is nothing inherently beneficial from the quantity of unit or integration tests you have, the quality of the tests to accurately target specific edge/error cases is what makes them valuable. As an aside I find TDD a particularly bad trend for these sorts of bad habits, TDD is a great idea in theory, but you'll often see TDD unit tests come out mimicking the code under test, i.e. "How to test if this function containing `return n + n ^ 2` is correct... well, let's generate some random numbers and then see if the output is equal to that random number plus it's square! That's like full coverage!" Having tests that duplicate the code under test is a pretty easy trap to fall into with micro-tests and it should make you suspicious of any test that covers a very small unit of code. This is similar to a statistical problem involving over-fitted trend lines, you can construct a mathematical formula that can go through any number of arbitrary (X,Y) points, leading to an average deviance of 0, but this formula will probably break as soon as you take another sample.

tretiy3 8y ago

This is good. But russian translation is terrible (were translations some marketing move?). Please ignore translation. Read original only

Dangeranger 8y ago

As a counter point to this opinion of Kent's, who I respect greatly, I'd like to propose J. B. Rainsberger's article: "Integrated Tests Are A Scam" with presentation of the same name.[0] [0] https://blog.thecodewhisperer.com/permalink/integrated-tests-are-a-scam https://blog.thecodewhisperer.com/permalink/integrated-tests...

mpweiher 8y ago

Counterpoint: Integrated Tests are a Scam https://blog.thecodewhisperer.com/permalink/integrated-tests-are-a-scam https://blog.thecodewhisperer.com/permalink/integrated-tests... I found that argument more convincing, and it also aligns better with my experience. The point about 100% coverage not being a good goal is pretty solid.

wenc 8y ago
For the benefit of others, it's also worth pointing out that the author isn't talking about "integration tests" but "integrated tests" (as he defines it). Integration tests = testing individual software modules as a group/combination. Integrated tests = instead of testing one behavior at a time, multiple behaviors are tested simultaneously. (or in the author's words: "any test whose result (pass or fail) depends on the correctness of the implementation of more than one piece of non-trivial behavior.") Integration tests are very important.
- mpweiher 8y ago
  It is also worth pointing out that the author used to use the term "integration tests" in this post, and only changed to "integrated tests" after writing. So a claim that these are completely distinct/disjoint ideas is not really supportable. As far as I can tell, he doesn't explain the name change, but having watched the talk before he changed the name, it looks like "integrated" is a somewhat more general term. Specifically: if you are "testing individual software modules as a group/combination", how are you going to do that without "depending on the correctness of the implementation of more than one piece of non-trivial behavior." I see exactly one way: your "integration" tests have to be trivial, really more akin to characterisation tests. "When I hit this external API with this value, do I get this reaction?" As far as I can tell, he doesn't talk about this in the post, but he does in the talk/video. These are interface/boundary tests. So you test each unit in isolation, and you test the boundaries. If your unit tests are sufficient and your boundary tests are sufficient, plugging things together will work. So no integrated/integration tests. Now I am actually not that strict, I let my boundary tests leak a little towards integration/integrated tests, because it is possible to just not hook up the components right. So you want to check that with some smoke tests. However, once they are hooked up, the more complex behaviours Just Work™.

1337shadow 8y ago

Maybe but do you think 100% coverage is such a bad experience to have at all ? This can be a rich exercise, and a rite of passage to know what tests matters, which should be automated (fuzzers, dbunit etc), which can wait, etc... and how to get to 90% with the effort of doing doing only 30. Integration tests should write themselves. It depends on the projects, but personally i use to get 65% of coverage for the price of 5%. For example you want to test a bunch of commands, then you can make a function like autotest('some command', 'some_fixture.txt'): call "some command", capture the output, and write some_fixture.txt with the captured output, and fail complaining that it had to create some_fixture.txt. The next run it will find some_fixture.txt and compare the output and fail only if it differs. Unit tests should of course be hand written, but to cover everything that matters like for bugs, or for what you want to refactor in TDD. Of course any line that's not covered can potentially break when upgrading versions, but this kind of breaks are likely to be revealed by the 90% of coverage that I think you can get with minimal effort by applying this recipe. Then, you can afford 0day updates when underlying libraries upstream make a new release candidate.

thom 8y ago

The thing that changed my approach to testing, at least in OO languages, was being shown that use cases/tasks/"things the system can do" should be first class objects in the model. At that point, you have a single layer that exposes everything of real value in the application, giving you a really simple test surface. Underneath that surface you can refactor to your heart's delight, without worrying about having to maintain lots of pointless unit tests for trivial behaviours on each object - all that matters is maintaining the end to end functionality. So yes, I agree integration tests are the key, and you can architect your application to make this easier. Not over-testing your underlying model is just good old fashioned information-hiding. Testing the UI on top I leave as a matter of taste.

castlegloom 8y ago

Yeah this is article comes out and says that it's based on gut, not science. So speaking for anecdotal evidence, unit tests that don't break and frequent deploys are way better than finessing one more Gherkin test.

kords 8y ago

100% coverage doesn't prove the code is fully tested, just like having unit/integration/acceptance/smoke and other types of tests doesn't prove that application works. But this doesn't mean we stop adding tests to our suite, so why stopping adding more coverage? I totally agree with "just stop mocking so much stuff". Most of the time we can use real implementation and by doing this, we'll also increase coverage.

munificent 8y ago

I think one mental model for tests is that they're simply another kind of automation. The fundamental question of automation is: will I repeat this task enough such that the amortized saving outweighs the cost of writing a script to do it? Whenever you're thinking of adding a test X, quickly consider how often you and your team are likely to need to manually test X if you don't. Also factor in the cost of writing test X (though that's tricky because sometimes you need to build test architecture Y, which lowers the costs of writing many tests...). If it's a piece of code that's buggy, changing frequently, brittle, or important, then you're likely to need to validate its behavior again and again. It's probably worth writing a unit test for it. If it's an end user experience that involves a lot of services maintained by different teams and tends to fall apart often, it's probably worth writing an integration test instead of having to keep manually testing every time the app goes down. If it's an API with lots of important users using it for all sorts of weird edge cases and you don't want to have to manually repro each of the existing weird edge cases any time you add a new one, it's probably worth writing some black box tests for it. But if it's code that's simple, boring, stable, short-lived, or unimportant, your time may be better spent elsewhere. Another model is that your tests represent the codified understanding of what the system is. This can be very helpful if you have a lot of team churn. How do you make sure a new person doesn't break something when "something" isn't well-defined somewhere? Tests are a great way to pin that down. If this is a priority, then it becomes more important to write tests for things where the test actually very rarely fails. It's more executable system specification than it is automation.

chriswarbo 8y ago
I agree with the idea that we're mostly automating manual work. The really nice thing about automated tests is that they can also go much further than I would ever bother to manually. For example, I might manually check that a couple of product pages are showing, with an image, price and description. With automated testing I can check all product pages, every time. It can also test a tedious amount of interactions, e.g. various interleavings of "add to cart", "remove from cart", "checkout", "press back button", etc. Manual testing still gives me confidence that automated tests can't, e.g. that an "unknown unknown" hasn't taken things down. Yet that's all I need to check manually; if pages are loading and look right, I don't feel a need to manually check all of the ecommerce arithmetic, permission logic, etc.

aboutruby 8y ago

It's confusing with Rails, because integration tests are now system tests, and the kind of integration tests described in this article are named controller and requests tests.

luord 8y ago

> you find yourself spending time testing things that really don’t need to be tested The thing with 100% code coverage is not that one should write more tests, but that you should refactor everything that isn't business logic (and, thus, nothing that would be part of the functionality of the application to test) into third party libraries, if it doesn't already come from third party libraries, which it probably should.

NicoJuicy 8y ago

At work, the unit tests and data are locked, even the webserver. Every change of data or requirements causes them to fail or fail because of human error. At home, I do integration testing and it seems to work much cleaner and more complete.

gravypod 8y ago

> I’ve heard managers and teams mandating 100% code coverage for applications. That’s a really bad idea I hope many managers and programmers out there don't take this the wrong way. I've been an engineer on a project that was attempting to get 100% code coverage on a piece of software I was writing. I heard constant remarks during this period that were similar to "You don't need 100% code coverage, it doesn't do anything!" These engineers who I was working with had only read articles like this and didn't stop to think about what the article was trying to say. From my experience there is no safe rule of thumb for for how many tests should be implemented for a project. There should be just enough to feel safe (as hathawsh has said). If you're recommending to engineers on your team to stop implementing tests when they say "I'm at 60% coverage and it'll take ~2 days to get to 100%" I'd really hope you take the time to understand why they want 100% coverage. The software I was working on when my coworkers started telling me to stop writing tests was code designed to trigger alarms when patient's vital signs met certain criteria set by doctors. I am very thankful that I did hit 100% coverage because between 60% and 100% there were many small edge cases that, had they caused a death, I wouldn't have been able to sleep well. Had they said I was explicitly disallowed from working on the tests I would have come in on a weekend (or taking a PTO) and implemented them then. It's our ethical responsibility to know when and where paranoia is worth the marginal time penalty.

bulkan 8y ago
> ...trigger alarms when patient's vital signs met certain criteria set by doctor Most of us aren't working on life or death code like this. My React app doesn't need 100% code coverage but you put it well when you said "There should be just enough to feel safe"
- gravypod 8y ago
  What I really want to drive home is that you should listen to the engineers who are working on the project. Testing is one of those things that is entirely subject to project needs. Like you say, a React app doesn't need 100% coverage, neither does code running a CI, something far out of the serving path, etc. But we shouldn't have a knee-jerk reaction to an engineer saying we should have 100% coverage.

garganzol 8y ago

Those sticky banners and elements injected by Medium are ridiculous. They make a good article hardly readable on HD desktop monitor. Sad state of things, I remember how once upon a time it was a great platform to be at.

camnora 8y ago

I’m not so sure I agree. Often isolated TDD can lead to much cleaner code and succinct solutions. There is also a benefit of quicker test execution. I’ve seen a noticeable difference in the quality of software delivered to my clients with this approach. Gary Bernhardt covers these topics in great depth and highly recommend to anyone interested.

neo2006 8y ago

I agree with the writer about the statement that you don't need a 100% test coverage but you still need to check your coverage. I mean you need to run coverage and go through your code and see if there is any gap in your testing. That said I don't agree with the statement "mostly integration" the reason is that he don't take into an account the ROI of a test, he only take into account the outcome. I mean e2e tests are the best to catch bugs but they are harder to achieve harder to debug and harder to maintain. Same go for integration test they are harder to debug, maintain and perform than unit test. Developpers forget that their time is money and that if they spend time on a test just because it make them feel safe that mean that the project will cost more money. The general rule I use is simple. When you test, test behavior not code and do your test at the lowest level possible.

trixie_ 8y ago

Integration tests really are the best bang for the buck. So many times people think they tested their code by mocking something like the database layer and are surprised when the application in production breaks. Everything can be tested in isolation and work perfectly, but often it's how the components work together that determines if the system works at all. Integration test what you can. Unit test core pieces of logic. Avoid mocks.

sime2009 8y ago
Fully agree. The majority of us are writing code with has to talk to a lot of other systems which we didn't write and don't control. These interface areas between systems are huge gathering places for bugs. Also, mocks are self-deception at best and lies at worst.

je42 8y ago

have to disagree here... not writing unit tests means architecture change will break your integration tests. hence, you don't have coverage left during an architecture change. typing information on a unit (class or function) is nice, except it doesn't find all bugs. If it were to find all bugs about a unit, it would be a complete type checker ( which usually comes in a system like https://en.wikipedia.org/wiki/Isabelle_(proof_assistant) https://en.wikipedia.org/wiki/Isabelle_(proof_assistant), etc, which is a pain to use... ) thus the only "efficient way" to cover the gap between type information and a full blown type checking system are unit-tests. integration test run slower, hence, you are also wasting more time. integration test are more difficult to understand by other people, especially if they break. integration test are generally less stable than unit-test because they involve more units. integration tests usually don't cover all (error) scenerios of the involved particular units. you do that in unit-test to get the branch coverage to 100%. his issue regarding mocking potentially can be solved differently. change your interfaces and code architecture such that you minimize mocking and make unit-testing easy. i.e. experiment moving the boundary between units.

sivanmz 8y ago

Maybe the author of Socket.io and Zeit.co just happens to work on integration problems?

deleted 8y ago

[deleted]

raverbashing 8y ago

100% unit test code coverage is just Uncle Bob BS and it is infeasible for the most part And while it will give some warranties about correctness, it will not fundamentally guarantee the application does what is supposed to do. Integration tests are certainly better I can only see the headline being a "great wisdom" to people who have accepted Uncle Bob and TDD crap for too long and without questioning it. Because it is obvious. Why it is obvious? Because that's how things were done most of the time When you had 640k of RAM and a C compiler writing unit tests was not impossible, but pretty hard. Testing that your app reacted to inputs and acted correctly was doable and easily automatable. And what wasn't automatable would be tested manually. Now here comes the "holy highnesses" of testing gurus, gaslighting developers and saying that code with no tests (which tests? automated? unit? the definition is purposefully vague) doesn't work or that the only blessed code is the one that is produced through TDD onanism? No thank you

kwhitefoot 8y ago

I know this is off topic but why is it that people who, presumably, want us to read their words of wisdom put such a pile of crap at the top of the page that I have to hit the page down key five times before I get to the article?

jondubois 8y ago

>> Write tests. Not too many. Mostly integration. If a group of reputable programmers got together and published a book about programming which had a single page which only contained this line in a large font, it would be the most useful and valuable programming book ever written.

edem 8y ago
Definitely not. See my comment on the parent.

edem 8y ago

My comment from the original post verbatim: I agree with the part that you should write tests, but I definitely disagree with the part that most of your tests should be integration tests. As you pointed out the testing pyramid suggests that you should write more unit tests. Why? Because if you have ever tried TDD you know that unit tests make you write good (or at least acceptable) code. The reason for this is that testing bad code is hard. By writing mostly integration tests you lose one of the advantages of unit testing and you sidestep the bad code checking part. The other reason is that unit tests are easy to write. If you have interfaces for your units of code then mocking is also easy. I recommend stubbing though, I think that if you have to use mocks it is a code smell. Also the .gif with the man in pieces is a straw man. Just because you have to write at least 1 integration test to check whether the man has not fallen apart is not a valid reason to write mostly integration tests! You can’t test your codebase reliably with them and they are also very costly to write, run and maintain! The testing pyramid exists for a reason. It is a product of countless hours of research, testing and head scratching. You should introspect your own methods instead and you might arrive at the conclusion that the codebase you are working on is bad and it is hard to unit test, that’s why you have chosen to write mostly integration tests.

blauditore 8y ago

I think the root of many testing problems is that people write code and continuously test manually until they think it works correctly. Then, they start writing some automated tests. What I think is much more effective is to write test instead of manual testing while developing. That is, what you would do by hand just implement in a test. And only at the very end, double-check manually.

executesorder66 8y ago
Isn't that just TDD?
- blauditore 8y ago
  Similar to it, but not as strict. TDD implies that you write tests first, which also leads the designing of your application code. What I'm describing is more general, where testing happens alternatingly with writing domain code, "roughly" at the same time.

vbsteven 8y ago

This is pretty similar to my approach. In the context of developing an API I use integration tests more than unit tests. It is straightforward to spin up an application server against a test database with some test data. Run some example requests against it and verify the results. I frequently have Test classes which contain a full scenario related to a specific resource or action. For example: * create resource * lookup resource * modify resource * list all resources * delete resource The JSON files for the integration tests are then used as examples in the API documentation. Unit tests are reserved mostly for verifying business logic components. There is no need to setup a complex ControllerTest with mocked out Services or Repositories as you don't care about the internals anyway. Just the input vs output and the integration tests cover those already.

ttty 8y ago

> Well, when you strive for 100% all the time, you find yourself spending time testing things that really don’t need to be tested I noticed people skip testing what is hard to test and just test what it's easy... Not what really matters

SomeHacker44 8y ago

Even at 100% code coverage it is possible to have missed many code paths that involve nonlocal control flow. The most common issue here would be exceptions in languages that support it. In most operating systems there are various interrupts that a program can also encounter (e.g., a lovely SEGFAULT in many *nix type operating systems) and other operating system and hardware level issues (like OOM killer in Linux or an NMI or ECC memory error in embedded systems). So, 100% code coverage may not even be enough for some applications as was discussed in another too level thread (medical device) or other hard to fix (spacecraft) or life-threatening situations.

natashas 8y ago

In general automated tests are used mostly to save time as the earlier you find the bug the more savings you have when fixing it. When your testers find it, they need to create and fill the ticket, assign it to developer, then developer needs to reproduce, fix it and assign it back and tester needs to retest it. And it's much more expensive if it happens in production! Integration/unit tests can catch a lot of those. There are some diminishing returns so 100% coverage is not needed and integration tests are more effective in catching bugs so I agree with the article's idea. Use unit tests for some real units - algorithms, calculations etc, and don't just test mocking framework. With additional layer of end to end tests and manual testing system should be able to achieve pretty good quality without spending unreasonable amount of time for it.