23 ms·
Write tests. Not too many. Mostly integration
- methodover 9y agoIs there solid evidence to back up some of the assertions that have been made about testing? It feels like an area where lots of people have opinions, and there are not much in the way of facts.
- p0nce 9y agoThere are very serious books about software quality with actual data, but it's much easier to tell each other anecdotic experiences on the internet - in a weird mix of bragging and strawmen arguments. That's how our field is stagnating.
- alien_at_work 9y agoMicrosoft put out a study where they had done TDD or at least extensive unit testing. I don't recall the numbers but development time took longer and there were a lot less bugs. Which is what I would have expected.
- C7H8N4O2 9y ago"Eat food. Not too much. Mostly plants." -Michael Pollan
- mzzter 9y agoI knew the title sounded familiar. Thanks for reminding me of where it came from!
- tantalor 9y agoSo much for "deep" & "profound".
- emodendroket 9y agoWhat? Does it take away from it somehow that it's a twist on someone else's famous phrase? I immediately recognized the reference despite hardly being a Pollan-head, so I doubt anyone's idea was to conceal the connection.
- hinkley 9y agoGood lord. Why integration tests? I think the biggest thing you can do to write more integration tests is to just stop mocking so much stuff. Okay. The biggest problem I see with people trying to write unit tests is that they don’t want to change how they write code. They just want tests for it. It’s like watching an OO person try their hardest to write OO code in a functional language. So they try to write E2E tests which work for about 5 or 6 quarters and then fall apart like a cheap bookcase. If you can find a new job before then, you never have to learn to write good tests! I agree with the author that the trick is to stop using mocks all the time, but you don’t have to write integration tests to get rid of mocks. You have to write better code. Usually if I have a unit test with more than one mock it’s bcause I’m too invested in the current shape of the code and I need to cleave the function in two, or change the structure of he question asked (eg, remake two methods into two other methods). Almost always when I accept that the code is wrong, I end up with clearer code and easier tests. Unit tests run faster, are written faster, and not only can they be fixed faster, they can be deleted and rewritten if the requirements change. The most painful thing to watch by far is someone spending hours trying to recycle an old test because they spent 3 hours on it last time and they’ll be damned if they’re going to just delete it now.
- jasim 9y agoNo true Scotsman. You might be speaking about those cases where people write large god classes, pervasively side-effectful code, zero API design, and a general lack of pure abstractions - then their tests would equally be bad. Tests are code, so one's ability to design programs would reflect on their tests and vice versa. But a reasonable programmer who cares enough about what they do can still end up with a brittle test suite because of other factors. Unit tests written in a dynamic language is a major drag on refactoring. It is not so much that we test the internals of the system or get tied up with the shape of code internal to an object. Even if you follow Sandi Metz's wonderful guidelines around testing: i) do not test internals, ii) test incoming queries by asserting on their return value, iii) test incoming commands by asserting on their side effects, iv) mock outgoing commands, and v) stub outgoing queries, you end up with a brittle test suite that is hard to refactor thanks to connascence. Whenever you refactor your names, or shuffle the hierarchy of your domain elements, you are now left with a thankless chore of hunting and pecking your unit tests and making them reflect the new reality of your code. Here integration tests help you know that your system still works end to end, and unit tests simply remain that one thing that refuses to budge until you pay it its respects. Unit testing complex views is still a hard problem. There are no well-defined stable "units" to speak of in an ever changing HTML UI. We have snapshot tests, we try to extract simple components on whom we can assert presence/absence of data, and we have integration tests that blindly railroads over everything and make sure the damn thing worked. But in a different context unit testing is the one true answer. If your system is statically typed (say in Haskell or OCaml), and your functions are pure and compositional, you don't so much worry about mocking and stubbing. You can make simple assertions on pure functions and as the granularity of your functions increase, they end up covering more parts of your system and get closer to an integration test. Static types form another sort of guarantee, the most basic one being that it takes a class of bugs away in form of undefined data types, the system going into invalid states, and of course the clerical mistake of named connascence. We often abuse unit tests in dynamic languages to cover these scenarios, leading to huge test suites with very brittle tests. I think it is important to call out that the value of unit tests are still contextual - "it depends" like everything in the world, and despite our best efforts, they can become hard to refactor. There is a case to be made for writing integration tests because they deliver more business value at a cheaper price than a pervasive set of unit tests when dealing with a highly effectful dynamic system. This lets us also think about other forms of testing like generative testing and snapshot testing that come to the same problem from different angles.
- jph 9y agoBehavior driven development (BDD) and test driven development (TDD) enable much faster coding when done well in my experience, and that includes full coverage fast unit tests, functional tests, benchmark tests, and integration tests. Unit tests are worth their weight in gold for quickly finding issues, both in our team's code and especially in cases of subtle changes among language releases, or unanticipated input changes, or dependency changes that are supposed to work but don't. IMHO unit tests lead to better functional approaches, better long range maintainability, better security, and much better handling of corner cases.
- hacker_9 9y agoI think the main selling point for me on TDD was breaking the boring build loop where I would have to wait for the project to compile, the webpage to load, click on all the buttons to get to the bit I'm testing, and then finally see if my code worked. Then repeat as many times as it fails. With TDD testing functionality is always just one button click away - I actually have fun again at work.
- Chris2048 9y ago> enable much faster coding Faster than what?
- ntrepid8 9y agoFor large code bases in dynamic languages I think the advantage grows larger over time. Even a modest amount of test coverage will allow you to build new features without breaking existing features. It's a lot faster to catch bugs when the tests run in your local development environment or in the CI pipeline than to wait until Q/A (hopefully) catches them in end-to-end integration testing and sends it back to you for rework. The other advantage I've noticed is that the tests can serve as an additional form of documentation. New team members can look through the tests to see how code is supposed to work from the examples there. For some web API projects I've even been able to generate the documentation from the test cases.
- moultano 9y agoI'd take a slightly different take: - Structure your code so it is mostly leaves. - Unit test the leaves. - Integration test the rest if needed. I like this approach in part because making lots of leaves also adds to the "literate"-ness of the code. With lots of opportunities to name your primitives, the code is much closer to being self documenting. Depending on the project and its requirements, I also think "lazy" testing has value. Any time you are looking at a block of code, suspicious that it's the source of a bug, write a test for it. If you're in an environment where bugs aren't costly, where attribution goes through few layers of code, and bugs are easily visible when they occur, this can save a lot of time.
- mannykannot 9y ago> Integration test the rest if needed. Is there any situation where there is integration, but no need to test it? You seem to be suggesting that if the leaves are thoroughly tested, nothing can go wrong in their integration, but at the same time, I cannot imagine someone believing that.
- 1_player 9y agoIntegration tests are always needed in some form, because you need to make sure the leaves are actually called, since unit tests are executed in a vacuum, the functions might work but might never be called at all, or might not work because of weird bugs that appear when only testing the whole tree.
- therealdrag0 9y agoExactly, most bugs I see are in integration, mis-matches in data-models or state. But I also work on business-y applications which tend to be more integrations than local business logic.
- chewbacha 9y agoThis is my exact mentality as well! In fact, I like it so much that I apply it to system design as well. Structure the pieces of code into a directed acyclic graph for great success. A tree structure that terminated leaves is a DAG. https://en.m.wikipedia.org/wiki/Directed_acyclic_graph https://en.m.wikipedia.org/wiki/Directed_acyclic_graph
- branko_d 9y agoIntegration testing is especially important when talking to a database. People seem to like mocking the data, but that completely misses the subtleties of how databases actually work, including aspects such as concurrency, transaction isolation levels, locking and such. There should be a few well crafted tests that modify the database from several parallel threads, and afterwards verify that no invariants have been broken. This is pretty much the only opportunity to catch a race condition in a controlled environment.
- gipp 9y agoThis is the part I don't get when people preach the gospel of ultra-isolated test suites where nothing talks to an external system. The most nontrivial parts of your code, the most likely to break, are those that talk to external systems. Building a mock of that system sophisticated enough to sufficiently capture even a significant fraction of the "real" failure modes is just as, if not more error-prone and daunting a task. Totally isolated tests make it impossible to test the most failure-prone parts of your code in anything approaching a satisfactory manner. Maybe I'm just misunderstanding people's advice but it drives me crazy when I see that.
- MartinCron 9y agoA lot of the ultra-isolated test advocates seem to come from the kind of development world where you can exercise most of the value without needing to interact with external dependencies. With aggressive enough mocking, you can deceive yourself into thinking that any application is in that domain. I know I have done it, and the application worked PERFECTLY until I hooked it up to a real database.
- throwaway5752 9y agoWhy does everyone rethink a working strategy. Write lots of unit tests that are fast. Write a good amount of integration tests that are relatively fast. Write fewer system integration tests that are slower. The testing pyramid works. He even talks about it in this post, and then ignores the point of it. You write lots of unit tests because you can run them inline pre-commit or in a component build. If you integration tests are numerous and interesting enough, that won't work. They are better suited to gating feature branch merges. System integration (the actual name for "end to end") take longer and usually gate progressively more stable branches upstream, or nightlies depending on where you are.
- a-saleh 9y agoBecause for certain kind of project the strategy stops working. I work as QE on a fairly large, ~7 years old project. Micorservice architecture has been attempted. We always merge to master, which means that everything more-or-less is a feature-branch merge. We have too many repositories to count. And what we learned is, that most of the components we have are just too thin to allow for useful unit-test coverage. Almost everything is [Gui]--request->[Middleware]--request-->[Proxy]--request-->[Backend]->[Database]. In reality, [Middleware] and [Backend] probably should have been a single component, but devs wanted to do microservices, and be scalable, but they didn't really understand the bounded contexts of their services. All of this leads us to a place, where unit-tests don't tell us much. On the other hand, we managed to spawn [Middleware]->[Backend]->[Database], and we can run a useful integration tests-suite in ~2 minutes. So, on one hand, if we desined this better, the good-old pyramid might be a working strategy. On the other hand, if I can get actual services running in minute, and test them end-to-end, I don't think I will bother with true unit-tests on my next projects. I.e. why mock the database, if I can spawn it seconds :-)
- miskin 9y agoSo, if I understand it correctly, Middleware and Backend should have been single component since it's one bounded context and splitting it makes one of those feature envy? Is there some benefit keeping these separate or is the cost of change too high at this point? If it's not about features, but more about API, have you tried Consumer-driven contract testing approach?
- oweiler 9y agoAs always it depends. There are projects where integration is the hard part and others where the business logic is impossible to get right without unit tests. Unit tests also have the added benefit that they tell you exactly the reason they failed (because there can only be one) whereas integration tests can fail for multiple reasons.
- toasterlovin 9y agoYeah, but usually an integration test tells you what line it failed on, which gets you fairly close to knowing the exact reason it failed.
- cbhl 9y agoI feel like there's definitely an art to mocking. Rather than mocking out the module being called (A calls B, A_test mocks B), you want to mock out its dependencies (A calls B calls C, A_test mocks C). But if you swap out said dependency, now you have to go update other modules' integration tests to mock out the new service.
- merb 9y agoI think the answer would be it heavily depends on what you are doing. if you are creating a library that operates on a protocol, unit tests are necessary / extremly important. if you are writing a ERP where a lot of your code NEEDS to operate WITH the database you are better of with integration tests, because mocking away the database would lead to so much bugs, especially if your database is extremly important (and not just a dumb datastore) Edit: having any tests is always better than having none.
- pleasecalllater 9y agoSounds good in theory. In practice there is one problem with having integrations tests only. The test are generally simple: they pass or they fail. A unit test tests just a small functionality, so when it fails, it's quite easy to find out the problem. When an integration test fails, then we can spend hours debugging the whole stack of layers trying to find out the real problem. I had this situation once. Every failing integration test ended with hours spent on writing unit tests for all the places used by the test.
- fpoling 9y agoFrom my experience an integration test failure that requires significant efforts to investigate can only be covered with unit tests after one knows where the problem comes from. One cannot realistically write a bunch of unit tests and expect them to cover the problem unless one already knows about the problem.
- ndh2 9y agoIt's called shotgun unit testing.
- alien_at_work 9y agoFor me, one of the biggest issues with integration tests is the code coverage numbers mean nearly nothing. I've seen an "integration only" tester proudly display his single test with 90% coverage. I asked him to run it again and it was 2% because a condition changed. So this means that for all the branches your code can take, an integration test is taking one specific one at each point all the way through for that test. All over branches, through the entire call stack are unverified.
- MichaelMoser123 9y agoThe problem with this kind of advise is that projects are all very different - there is no one fit way to test it.
- misja111 9y agoThis is the only true statement that can be made about this whole integration-/ unit testing debate but it will never be popular because it doesn't seem to be simple enough. Most people like black and white guidelines. 'Unit testing is good, integration testing is bad', something along those lines. Simple to remember, simple to apply. Unfortunately it doesn't match reality. The truth is, every testing strategy is a tradeoff. Integration tests are great at catching regressions and verifying business requirements, but they can take a lot of time to construct, tend to be slow and don't give you much clues where to find the error when they fail. Unit tests are fast, easy to build and when they are well written they will point you right to the point of your code that is wrong. However, the more specific they are, the more they tend to test your implementation choices instead of real business requirements. But even the above is not always right. When you are building a library, you might be able to create unit tests that test your api and then you have the best of both worlds, your unit tests do actually verify your business requirements. On the other hand, it might be that you are building an application which doesn't depend on a lot of data and which can be orchestrated pretty well in a test set up, and therefore integration tests are actually easy to build; in that case you'd like to focus more on integration tests. Tl;Dr; there is no silver bullet, in the end the only right thing to do is let your test strategy depend on the characteristics of the project.
- rb808 9y agoAbsolutely - this is the key. There is a huge difference in the software for a game engine vs a website vs an MIS report vs a trading system vs an autopilot. You can't blanket statement which testing technique is better for all software.
- egeozcan 9y agoThis is my version: Don't stop writing tests. Not until 100% coverage. Mostly unit. Especially if you are developing a library. An untested code branch is a ticking time bomb, in my book. An application for the end users indeed does benefit from integration tests, a lot. The problem is running them efficiently. If they take an hour to run, nobody will care to analyse them.
- edem 9y agoI agree with the part that you should write tests, but I definitely disagree with the part that most of your tests should be integration tests. As you pointed out the testing pyramid suggests that you should write more unit tests. Why? Because if you have ever tried TDD you know that unit tests make you write good (or at least acceptable) code. The reason for this is that testing bad code is hard. By writing mostly integration tests you lose one of the advantages of unit testing and you sidestep the bad code checking part. The other reason is that unit tests are easy to write. If you have interfaces for your units of code then mocking is also easy. I recommend stubbing though, I think that if you have to use mocks it is a code smell. Also the .gif with the man in pieces is a straw man. Just because you have to write at least 1 integration test to check whether the man has not fallen apart is not a valid reason to write mostly integration tests! You can't test your codebase reliably with them and they are also very costly to write, run and maintain! The testing pyramid exists for a reason! It is a product of countless hours of research, testing and head scratching! You should introspect your own methods instead and you might arrive at the conclusion that the codebase you are working on is bad and it is hard to unit test, that's why you chosen writing mostly integration tests.
- didibus 9y agoI agree very much with this. I'd add one thing, adding tests and testing your code are not the same thing. You should write tests, mostly integration tests, testing public well defined boundaries of your class, your component, your service. Mock only IO if needed, but not other classes. But also tests the rest, but no need to add a test for them. Just run the code, try out the private functions, make sure they work. And also go read Testivus: https://www.artima.com/weblogs/viewpost.jsp?thread=204677 https://www.artima.com/weblogs/viewpost.jsp?thread=204677
- breatheoften 9y agoI think the testing pyramid reflects a false correlation — it seems to assert that higher up the pyramid tests are more expensive to write/maintain and longer to run. In reality the execution time of a test says nothing about how hard the test is to write. Sometimes a very fast to execute unit test can be much harder to write/maintain than a longer running test that avoids mocking an api and perhaps utilizes abstractions in the test definition that are already written to support the program’s features. I think test suite execution speed is the real metric to focus on for most projects — to get the most value, test suites should accelerate the time to useful feedback. Write tests in the simplest way that provides useful feedback into the behavior of the system and runs quickly enough that you can receive that feedback with low latency during development. I quite like tools like jest and wallabyjs that use code coverage data to figure out which tests to rerun as code changes — means you can have a test suite that includes slow(ish) to execute tests but still get feedback quickly in reasonable time as you make changes to the code.
- Vinnl 9y ago> to get the most value, test suites should accelerate the time to useful feedback Well, they should also optimise the usefulness of the feedback they provide. Typically, tests higher up the pyramid are also more brittle (e.g. end-to-end tests might fire up an entire browser and Selenium), and thus are more likely to fail when in actuality, nothing is wrong. That's an additional reason for limiting the number of those tests.
- breatheoften 9y agoBrittle tests seem not useful in general though aren't they? I'm not sure its necessarily true that brittleness must correlate with height in pyramid or execution time -- in my experience brittleness correlates with selenium more than it does pyramid height (that's a statement about selenium more than it is a statement about any particular category of testing pyramid). Its possible to write very useful non-brittle tests using something like headless chrome ...
- 9y ago
- andmarios 9y agoIntegration tests are very important and in many cases mandatory. I'd argue though unit tests with good coverage are more important. The latter helps you make sure your application has the correct output. Nothing worst than having a piece of software you think is working and silently introducing errors. The former (integration tests) makes sure your application starts and probably works in some scenarios. With docker and containers, integration testing is easier than ever. I've written a very simple integration testing tool (a quick hack to be honest) that executes commands, checks their exit code, the output through regexp, etc and produces a nice html report. https://github.com/landoop/coyote https://github.com/landoop/coyote Originally written for testing some configuration tools and scripts, we quickly found a use for it for our Kafka connector collection (Stream Reactor). Our connector integration tests are at github as well: https://github.com/Landoop/kafka-connectors-tests https://github.com/Landoop/kafka-connectors-tests Obviously unit tests make sure that the connectors work as expected. Integration tests catch simpler errors that can be showstoppers. For example an internally renamed configuration option that didn't trickle down to the configuration parser, class shadowing issues between the connectors, unexpected errors in logs that the developers should check out etc. As the tests are written by people who don't do development, they also expose problems in the documentation. In some cases they provide us an easy way to quickly run locally a connector and catch some issues manually, like extensive CPU usage or way too much logging (e.g a function with a log statement was inside a loop that run hundreds or thousands of times per second). It gives us confidence in what we ship with every release.
- jlebrech 9y agoand keep as much logic away from templates, that way you can unit test the model. problem is templating frameworks are too smart.
- waibelp 9y agoWhy are most people using huge hero-graphics (495kb?!) for short blog postings nowadays? On topic: Testing gets worse if the codebase which needs to be tested is garbage. My experience with other developers who learn testing don't need to learn how testing works - they need to learn basic development rules: Components, loose coupling, dependency injection, ...
- ezanmoto 9y agoJust reading the comments, I agree with the common sentiment that you should have more unit tests than integration tests, but I have come around to the way of thinking that if I only have time to write a few tests then I would rather write E2E tests. This way, at the very least your entire stack is being exercised, and you have a way of ensuring that the happy path is passing consistently, which is the most important flow for an application (even if I'm personally more interested in keeping other flows sane). While I prefer unit tests due to their simplicity, speed and the speed at which they can aid debugging, these days I will only implement them after I have added a few E2E tests.
- danrspen 9y agoI started writing a response to this ... but it got a bit too long and became my first ever Medium post: https://medium.com/@danrspen/write-tests-a-sensible-amount-of-appropriate-types-d19819d39bf0 https://medium.com/@danrspen/write-tests-a-sensible-amount-o...
- lucidguppy 9y agoClean architecture: https://smile.amazon.com/Clean-Architecture-Craftsmans-Software-Structure/dp/0134494164 https://smile.amazon.com/Clean-Architecture-Craftsmans-Softw... Anything that touches the real world should be as small as possible. I've been writing code using tests for about 4+ years and I now can't think of writing code any other way. I would be scared of refactoring. Also I'm testing the code anyway - why not write it down so it gets done every time? Run integration tests too for sanity - they're testing the integration of things. To me: coding without tests is like going caving without a flash light. You don't really know what your code does until you run tests. Your confidence rises when more code is covered. No it isn't perfect - but not writing tests is not better.
- wolco 9y agoThe problem with always relying on testing is you lose your ability to create without the test crutch. Everything is red/green and your brain can get a lazy. Balance is the best.
- lucidguppy 9y agoWhat is science without tests? I say coding is similar.
- titzer 9y ago> I’ve heard managers and teams mandating 100% code coverage for applications. That’s a really bad idea. The problem is that you get diminishing returns on our tests as the coverage increases much beyond 70%... I call bullshit. I work on V8, on JITs and WebAssembly. 70% coverage for these code bases would be absurdly low. We would never ship code that is that poorly tested, and you shouldn't either. > You may also find yourself testing implementation details just so you can make sure you get that one line of code that’s hard to reproduce in a test environment. You really want to avoid testing implementation details because it doesn’t give you very much confidence that your application is working and it slows you down when refactoring. You should very rarely have to change tests when you refactor code. What in the. serious. fuck. Of course tests test implementation details. Because _implementation details_ are where the goddamn bugs are. > ... Maintaining tests like this actually really slow you and your team down. That's the whole _point_. It slows you down in the short term but it keeps you from experiencing a full-on system meltdown when everything seems to be breaking at once. Please don't follow the advice of this. It's total crap. If you've never worked on a system that has survived more than 3 years, sure, go right ahead, run against the wall. But when you work on a system survives 5, 10 (V8), or 20 years (HotSpot JVM), then you really, really want to have good tests.
- away2017throw 9y agoNo offence, but you work in a bubble of sorts. In enterprise we are absolutely expected to run against a wall - preferably fast.
- titzer 9y agoThe bubble happens to be at the bottom of everything everyone runs. If we--or kernel folks for that matter--applied the advice of the article to our development practices, our system meltdown would be your system meltdown. Sure, you have requirements from management. So do architects and engineers for building bridges. Yet they still have a duty to build bridges that don't fall down.
- Chris2048 9y ago
- kartan 9y agoSo much people in this thread is talking about different domains and are not able to see that they need different rules. It is not the same creating a library that is going to be used to launch a multi-billion rocket to Mars than developing a mostly graphical mobile app where requirements are changing daily as you A/B test your way into better business value. The article has really good points and the reasons why they work. Apply them wisely. Take the right decision for your project. Don't be dogmatic.
- protonfish 9y agoThe fight against the TDD cargo cult is vicious, barely started, and far from over.
- dang 9y agoPlease don't do this here. If you have a substantive point to make, make it thoughtfully; if you don't, please don't comment until you do. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- 3pt14159 9y agoI've programmed in many domains. There are only three places where I don't use heavy testing: 1. Very, very graphical programming - like SVG charting with animations. If it were static generation it would be easy, but throwing time into the mix makes the tests really hard plus if things go wrong in the future people can literally see it going wrong and complain, so I don't think it is worth the trouble. 2. Data analysis meant for static reporting. You know, those 2000 line SQL queries that barf out data that you pop into excel to munge through before typing up a 20 pager for upper management. 3. Small personal tools, like a CLI script that spits out equivalent yearly interest rates or what have you. Everything else I test. Libraries, backend web apps, machine learning shit, compiled, whatever. It is too easy for codebases to turn into a hellscape without tests. You get too afraid to change things. Should it change the API to the codebase? I usually don't think so, but occasionally I'll, say, make something an instance variable that I'd normally keep as a locally scoped variable. So I model test what I can and I integration test the rest. I think that he's right that integration tests cover a lot of functionality and I think he's right that mocks aren't usually great, but I think he's wrong about how much testing we should be doing. Most things should be tested most of the time. It's cheaper. Why? Because it costs money to hire support people and it costs money to validate their tickets and it costs money to fix the bugs and the bugs are harder to fix when the data is already in the system and the data is wrong. It's also more profitable to write more tests. Why? Because bugs mean lost customers and even when you keep the customer the feedback that you get from them is which problems you need to fix, not which functionality could be made better.
- luord 9y ago> You should very rarely have to change tests when you refactor code. I'm going to say this is poor, if not outright dangerous, advice. I'd argue the opposite: every time the code changes, it would be great if a test somewhere broke. Hell, if the tests originally were for a gigantic class with a ton of mocks but the code changed so there was lower coupling and the dependencies were injectable or separated in different modules, the tests should be changed. > even a strongly typed language should have tests Indeed, yet I've seen so many people in the strong typing "camp" mentioning "the compiler catches some bugs right away!" as an advantage, which always makes me think "... and?" > It doesn’t matter if your button component calls the onClick handler if that handler doesn't make the right request with the right data! ... But there should be another unit test for that handler, checking that it's using the right request with the right data. In general, I agree with some points and disagree on others.
- marcosdumay 9y ago> every time the code changes, it would be great if a test somewhere broke So, it doesn't matter if your code is correct or not? > "the compiler catches some bugs right away!" as an advantage, which always makes me think "... and?" And that leaves more time to design things right and test the stuff that actually matters.
- luord 9y ago> So, it doesn't matter if your code is correct or not? I'm going to need you to walk me so I can see how that part you quoted implies that the code being correct doesn't matter. I honestly don't see the correlation. > And that leaves more time to design things right and test the stuff that actually matters. I'm not gonna get into a religious flamewar; if you prefer static typing, all the power to you and I'm not interested in convincing you otherwise. However, I fail to see how having to write "int", "str", etc before or after a variable name impacts the design process in any way. As for testing: - A test in Go: func TestAvg(t *testing.T) { for _, tt := range []struct { Nos []int Result int }{ {Nos: []int{2, 4}, Result: 3}, {Nos: []int{1, 2, 5}, Result: 2}, {Nos: []int{1}, Result: 1}, {Nos: []int{}, Result: 0}, {Nos: []int{2, -2}, Result: 0}, } { if avg := Average(tt.Nos...); avg != tt.Result { t.Fatalf("expected average of %v to be %d, got %d\n", tt.Nos, tt.Result, avg) } } } - The exact same test in python: def test_average(): for param in [ {'nos': (2, 4), 'res': 3}, {'nos': (1, 2, 5), 'res': 2}, {'nos': (1,), 'res': 1}, {'nos': (), 'res': 0}, {'nos': (2, -2), 'res': 0}, ]: assert average(*param['nos']) == param['res'] In a real situation, tests like this one need to be written and the fact that in Go we're specifying types doesn't change a thing, so I'm not convinced that not writing the types means that in a dynamic language I have to test stuff that doesn't matter. For that matter, I'm having a hard time imagining a scenario where the type wouldn't be tested anyway so back to my "... and?" But, again, I'm also not trying to convince you otherwise. Different approaches for different folks.
- tybit 9y agoI think this is poor advice because it doesn't give context. I think unit tests are good for testing logic, and integration tests are good for testing functionality. Lots of complex logic? Then write lots of unit tests. Got a service that just wraps a database? Then you're going to want to write a lot of integration tests.
- ivan1931 9y agoI disagree with this article. Software projects can be extremely large and have wildly different requirements. Software that needs to operate at high scale and require high reliability will have differenet requirements for say - a web application that has low traffic. I think making rules of thumbs like the title of this article defeats the purpose of one of the essential tasks of being an engineer - making good tradeoffs between different approaches to solving problems. It's not hard to see that some projects will require more unit testing, some may require more integration testing and others may require more of both.
- sandos 9y agoI am currently in a project where we do not use TDD, even remotely, but we do have almost 100% coverage, since we need it for certification purposes. These tests are fairly easy to write, actually, with a test framework that helps mock everything automatically. Then we have some internal tools written that makes these tests worth a lot less, we produce a lot of the test data and then just record the outputs as "correct". We have no real idea if they are correct, but at least the tests as they are now works as regression tests. This is what happens when you realize you need unit tests for tens of thousands of line of code in a few weeks' time, instead of writing the tests when you write the code. There are of course testing on higher levels for this industrial system, multiple levels even and I have to say that the unit tests are fairly useless in comparison.
- jrs95 9y agoReminds me of this: http://david.heinemeierhansson.com/2014/tdd-is-dead-long-live-testing.html http://david.heinemeierhansson.com/2014/tdd-is-dead-long-liv...
- TimJYoung 9y agoThe one thing that I didn't see mentioned is the fact that you can test very specific branches of code with higher-level testing, but the same cannot be done in reverse. In fact, higher-level tests, especially tests driven with real--world data, routinely execute branches of code in ways that are unforeseen by the developer and unlikely to have been tested at a lower level. I suspect that fuzz testing will some day remove much of the need for lower-level testing.
- scyclow 9y agoDoes anyone have any good resources on writing front-end integration tests? I've had nothing but terrible experiences with selenium. And I recently tried Nightmare (which is Electron-based), but it wasn't much better.
- bogle 9y agoI have done a little with NightmareJS and really liked it. A bit of syntatic sugar over Mocha with the option to drop down when required. Very succinct, I thought, and easy to start with. I, too, prefer to avoid Selenium, though.
- MartinCron 9y agoI too have had selenium nightmares but have landed someplace pretty good. Here are some quick tips. YMMV. Don’t think “integration” — think “full stack” these will find configuration and connectivity bugs more than business logic bugs. These can’t be your only tests. They need to run as part of your CD/CI pipeline automatically, otherwise, they won’t get run and will decay from disuse. Headless browsers (HTMLUnit and PhantomJS) are easier to work with than “real” browsers. Haven’t used Chrome headless yet. Front end bugs are often fiddly and visual. Screenshots + human review can be a cost effective supplement to manual testing, but can never replace it. Good error logging and reporting is also key. If your front end tests break something, having the backend tell you what broke will save you time. I tend to keep these “full stack” tests to happy path scenarios, as they are slower to write and to run than lower level integration tests. Good luck.
- Jach 9y agoI don't practice TDD (see Norvig, Jeffers, and Sudoku) but I do like unit tests. I also like integration, e2e, property, mutation, and fuzz tests. I like proving specs with TLA+. Sometimes type theory proofs are useful. There's a lot of ways to improve quality. Some of those I've only been able to toy with personally, though, because the dirty secret of software engineering is that for most things we don't need very high levels, we just need most customers to not be mad. So that leads to articles like this, where people claim their experience showed the most bang for buck with a certain approach, despite not always even trying other approaches... > One thing that it doesn’t show though is that as you move up the pyramid, the confidence quotient of each form of testing increases. You get more bang for your buck. So while E2E tests may be slower and more expensive than unit tests, they bring you much more confidence that your application is working as intended. In my experience this is wrong. Working with an inverse pyramid, you'd think confidence would be high, bugs few. It's the exact opposite. > just stop mocking so much stuff Here here. But this gets into subtle arguments over "is this really a unit test if it's using RealFoo even though we're trying to test Bar, such that if RealFoo breaks this test will also break but not because of Bar?" I'm not too strict on my definition of unit test, my best attempt is something like "relatively small, isolated from the broader module/library/application, runs fast, easy to find root failure when test fails, avoids testing the implementation rather than behavior (often hard), and asserts something." It leaves open the possibility for technical 'integrations' but there's a pretty big space of possibilities between a minor integration to avoid mostly useless mocking and suddenly requiring the whole application server to have started up before you can do anything.
- laurent123456 9y agoOn a hobby project recently I've only written backend integration tests. I found these to be extremely useful though since they touch directly or indirectly to all the critical parts of the software, so if something's broken it will eventually be caught there. Also since it's relatively high level there's also rarely a need to change them whenever I change something to the backend. All in all it's definitely a time saver, both in terms of catching bugs and maintaining the tests. I think there's still a value in detailed unit tests but mostly for library code, when you want to test each function properly with various inputs.
- smizell 9y agoOne of the most transformative things I've come across for how to structure and test code has been Gary Bernhardt's talk on Boundaries [0]. I've watched it at least ten times. He also has an entire series on testing where he goes deeper into these ideas. In this video, he talks of a concept called functional core, imperative shell. The functional core is your code that contains your core logic that can be easily unit tested because it just receives plain values from the outside world. The imperative shell is the outside world that talks to disks, databases, APIs, UIs, etc. and builds these values to be used in the core. I'll stop there—Gary's video will do 100x than I can do here :) [0] https://www.destroyallsoftware.com/talks/boundaries https://www.destroyallsoftware.com/talks/boundaries
- agentultra 9y agoTesting. My current theory for why there is so much confusion about testing is because developers are not often taught the difference between specification (theorems) and implementation (proofs) and why you want to separate the two. It seems like business and investors want us to write code, more of it, and faster. Value, value, value! So my question is: what is valuable? Do you value your customers' data? Do you value their time? Their safety? Your brand and reputation? If you answered yes to any of these (and the other questions I may of forgotten or elided) then you should be encouraging your developers to write specifications. One such specification, and a weak one that developers can write and maintain on their own without involving stakeholders, are unit tests. It's a weak form of specification for a library/module/component you would like to have because it specifies properties and behaviors by example. The spec gives an example of use and the expected outcomes. A good test is a verbose Hoare triple: given some context, when this method is called, then this result is expected. Your implementation of that specification is what you're after and it's the reason why you should write the tests first. Writing tests first has little to do with productivity or design. You should write them first because your implementation should prove the theorems in your specification. Theory first. Proof after. Sometimes you have to revise your theories after attempting the proof.. but that's a story for another day. But we can write better specifications! Unit tests are weak because each test only demonstrates a single expectation. It doesn't quantify over the space of possible inputs! If you want to write a better specification for your parser or transformation try property based testing. Use a library like QuickCheck. You give it a theorem that quantifies over the input space of your function under test and it will find out for you if your proof holds (limited only by how many examples you want to try... say 10000). It's not a proof that your implementation is correct but its a much stronger guarantee than a suite of unit tests and it doesn't cost you that much more to use it. Integration tests though. I'm not sure if I agree with the advice. You should definitely write them but they can become a time sink and cost quite a bit to run if you need to test a non-trivial system. Where they lack is at level of quantification and this is where the real errors lie in software systems with many components. Integration tests, like unit tests, are proof by example. You write a specification for how a given configuration of components should interact and you supply a context and run the test and see if your assertions hold. It's wise to be aware that it won't catch most safety errors and it will never catch liveness errors. Safety errors having to do with correctness of expected values over the lifetime of a computation. Liveness having to do with maintaining invariants over the lifetime of a computation. So integration tests are useful, do write them, but I wouldn't advise spending most of your time on them if you're dealing with more than 2 or 3 components. Once you have messaging and co-ordination you're going to want a stronger specification and that would probably look more like a theorem written in a language that can be verified by something called a model checker. Something like TLA+ is making good progress breaking into industry. ... this is getting long. To summarize: developers should be taught and given time to write specifications. Most errors in software arise from poor, incorrect, or missing specifications. Weak specifications are better than none. Think of tests as specifications, write them first, and prove your software meets those specifications. Then you can change your implementation as you refine your specification and write better, faster, more reliable software.
- shados 9y agoI go the complete opposite way. I've tried various testing strategies over 15~ different companies in all sorts of environments, and unit tests are the only thing that really work (IF you can convince the team to do it...and that's a big IF). The article starts with a point I agree with: the lower in the pyramid, the cheaper the tests but the lower the confidence level they bring. That's true. Where I disagree is how much the difference on confidence and cost are. I can bang out 500 unit tests faster than I can do just a few E2E tests in most large apps. They require almost no trial and error, no real engineering (I feel strongly that abstraction in unit tests is bad), and all around are so easy to write, I don't mind if I have to toss out 150 of time when I make a significant refactor. E2E tests are amazingly brittle and require a careful understanding of the whole system. They're impossibly expensive to write. They're the only thing that tells you that stuff works though. So you want at least a few of these. Integration tests are just flat out awkward: you need understanding of a significant portion of code you did not write or touch, they often require complex fixtures (because your test will go through several code paths and might depend on a lot of arguments), they're slower (because a lot of code run), and while you don't throw them away when changing implementation details (unless they involve side effects), you still throw them away when refactoring or changing public interfaces. I've worked with a lot of people who were very vocal about these being so much better, then in the same breath complain that they spent all day writing integration tests. There's an exception here which is acceptance tests for libraries, especially when doing a full rewrite: the tests that tell you public interfaces used outside of the current context work (as opposed to public interface of objects used in the implementation). Eg: if I was to test lodash or react, that's how I'd do it. Unit tests to be are about a lot more than "is this change breaking my code". And if that's all you care about, you're missing a big part of the point. If you have 3 units, A, B and C. A calls B which calls C. If you have a test for A in the context of B, a text for B in the context of C, and a test for C, and they all pass, you know that A + B + C will work. But when writing the tests, you only had to care about itty bitty tiny pieces of code, which made things super cheap. Then you get other huge benefit: the quality of the entire code base is higher (a side effect of it having testable interfaces all across), the reasoning behind each piece of code is explicit (no one wrote a function that work and they 're not sure why, else the test would be very hard to write), you automatically have a document representing "intentions". And yes, if you change a module, even if its not expose to your customers, the public interface of that module has tests and the tests will break. But they usually take nothing but a few minutes (often a few seconds) to write. They're cheap enough to be disposable. And once you have 80%ish unit test coverage, you actually have a very high confidence level. I've gone through major refactoring of multi-million line of code apps with almost no bugs on pure unit tests. You't think the 20% of untested code would be a source of bug, but statistically, that's just not how it happens. In term of person-hour to ROI, pure unit tests just straight up win out. The reason software engineers fight back to hard against them is that they're brain dead and repetitive to write, and they can't resist overengineering. "This is such a simple test for such a simple piece of code, why should I test it?!". That's the point. All unit tests should be like this.
- nathan_f77 9y agoThe puffing-billy [1] library is awesome, and has changed the way I write integration tests. I also use VCR [2], and now my entire application (both backend and front-end) is wrapped with a proxy that records and replays every request. I can run all my tests once using test Stripe API keys, a test Recaptcha response, or any other external services that I want to test. I don't have to mock anything, which is nice. Then everything is recorded, and I can run all my integration tests offline. I've also really enjoyed using stripe-ruby-mock when testing specific webhooks, jobs, and controller actions. I don't always aim for 100% test coverage, but I try to write a LOT of tests for any code that deals with billing and subscriptions. Ooh, I've also been enjoying rswag [4]. It's quite a cool idea - You write rspec tests for your API endpoints, and the tests also serve as a Swagger definition for your API. So when your tests pass, you can use the output to generate documentation or API clients for any language. [1] https://github.com/oesmith/puffing-billy https://github.com/oesmith/puffing-billy [2] https://github.com/vcr/vcr https://github.com/vcr/vcr [3] https://github.com/rebelidealist/stripe-ruby-mock https://github.com/rebelidealist/stripe-ruby-mock [4] https://github.com/domaindrivendev/rswag https://github.com/domaindrivendev/rswag
- choward 9y agoWhy did I just have to scroll three pages to get past a pineapple. Nobody wins. It takes extra effort to add. There is no ad revenue. The reader gets annoyed. WHY?!?!?!
- maxxxxx 9y agoReading all the comments reminds me why I really don't like working in large enterprise teams. People argue a lot about semantics ("what is a unit test?") and the only right way to do things but there is almost nothing practical to learn from. I have found that in some projects unit tests are really easy to write and helpful but in others you spend more time on writing mocks and dependency injection than writing stable code. I don't even know what I want to say exactly other than that we should focus more on practical solutions to real problems and less on debating semantics.
- killjoywashere 9y agoThe title, by the way, is a play on Michael Pollan's famous essay "Unhappy Meals". The top line becomes the subtitle, the lesson, and eventually the title everyone googles for: "Eat food. Not too much. Mostly plants." http://www.nytimes.com/2007/01/28/magazine/28nutritionism.t.html http://www.nytimes.com/2007/01/28/magazine/28nutritionism.t....
- floor__ 9y agoI wouldn’t trust a person who says they kicked a pigeon.
- therealdrag0 9y agoI trust my own sense of when unit-tests and integration tests are appropriate. If I'm working on a bit of code that has actual internal logic, then I'm happy to write unit tests so I can iterate on the test with scenarios while I code. OTOH, if I'm writing a glorified CRUD pass through, api > model -> db-entity > do a thing > return result. There is nothing to unit test. Writing a few integration tests that call the API gives me much more confidence that everything is hooked up correctly than mocking a bunch of shit and asserting that a method was called that was obviously called.
- juandazapata 9y agoUnit tests force you to think about good design. Integration tests don't. If you only use integration tests, you'll end up with a big ball of mud. Certainly I wouldn't want to touch a codebase by the author of this post.
- mannykannot 9y agoThis is exactly the wrong way round. Only by thinking about interfaces can you avoid a big ball of mud, and interfaces are what integration testing tests.
- spopejoy 9y agoThe original tweet is a takeoff on the Michael Pollan maxim right? "Eat food. Not too much. Mostly plants"