16 ms·

Software Testing Anti-patterns

465 points by kkapelon 8y ago

amelius 8y ago
In cases where manual testing is the only option, is it an anti-pattern to let the testing be done by the person who developed the code? As a developer, I feel that others should test my code. Is that reasonable?
- mic47 8y ago
  Why should other test your code?
  - ben-schaaf 8y ago
    Same reason you don't review your own code, or pair program by talking to yourself.
    - koolba 8y ago
      > ... or pair program by talking to yourself. Who says I don’t? https://en.m.wikipedia.org/wiki/Rubber_duck_debugging https://en.m.wikipedia.org/wiki/Rubber_duck_debugging
      - icebraining 8y ago
        It's not pair programming, though, even if can have some of its advantages.
    - mic47 8y ago
      You can solve this by test plan: You write into test plan what you have tested, and others in code review can suggest/require to test other things you missed. In my previous job this worked quite well.
      - icebraining 8y ago
        That's fair enough; I think when people are talking about manual testing, they're thinking about the decision of what should be tested, not the actual button pressing. But it's a distinction worth making.
    - stinos 8y ago
      This seems a bit of a false generalization to me. For a long time I've been the sole developer on multiple large-ish applications. So of course I review my own code, and I write my own tests. And yes as another commenter mentions: if you don't pay attention this can lead to tunnel vision, and yes it can be hard to be consistent about it. I discovered those drawbacks fairly early, so quickly made it a habit to be super-strict and critical about pretty much every single line of code I write (both with regards to style, functionality, how it adheres to the standard practices etc). What I also do consistently is review older code: if I have to add a function to existing code, it happens more often than not that I'll re-read and review the surrounding bits. Often to come up with better code. If time permits, else I take note and go back to it later. In practice it turns out that if I haven't touched code for months, reviewing it is almost the same as reviewing someone else's code, i.e. starting with a clean slate. Which is also why I'll sometimes write something, commit it, but not yet merge it to production. Then a week or so later I'll come back and review every bit again.
      - ben-schaaf 8y ago
        Oh, I completely agree, though I wouldn't call writing code carefully and methodically a "code review" if that makes sense. I personally find that me a couple months down the track is pretty much another person anyway. In a sense you're not reviewing your own code, you're letting it be reviewed by future you. But if you're in a team with multiple developers, having someone else review your code would be more efficient.
        _asummers 8y ago
        I tend to review my code with a reviewer hat, not with the author hat, before I open the PR. Find all the places where I'd call myself out for lazy names, poor abstractions, missing docs, etc. then go in and fix those. Repeat until satisfied, then open PR.
      - rb808 8y ago
        > In practice it turns out that if I haven't touched code for months, reviewing it is almost the same as reviewing someone else's code, I really like this idea, haven't heard it before but it makes complete sense, thank you.
    - rasjani 8y ago
      Yellow rubber duck makes pair programming way more socially awkward ;)
- tempodox 8y ago
  Manually testing your own code is liable to tunnel vision. So yes, having others do it is safer. There are even production standards that make this mandatory.
- dankohn1 8y ago
  Why would manual testing be the only option? As an example, we wanted to automate the test for browser-based OAuth connections to GitHub and authorizing an app there. Here is the test that does so: https://github.com/coreinfrastructure/best-practices-badge/blob/master/test/features/github_login_test.rb https://github.com/coreinfrastructure/best-practices-badge/b...
- kalecserk 8y ago
  I have been working with teams that heavily use automated tests for over 16 years now and, at least on my experience, making others test your code would lead to untestable code and delegation of responsibility. Even for manual tests, the test spec should make part of the change request and be reviewed appropriately. Cheers!
amarant 8y ago
I can't say I agree with this. Partly because it's too generic. I'm of the opinion that you should focus on testing what you deliver. If you're building a framework or library, then unit tests are probably the way to go. If you're developing a micro service that's primarily going to be called by other services, built by other teams for example, you should probably focus on integration tests. If you have an external API, you should focus on e2e tests, as your clients won't know or care that you have several individually well tested microservices behind your API or if it's a monolith. They only care that a given API call has the expected result. The pyramid thing is pretty, but also meaningless IMHO.
- diggan 8y ago
  > I can't say I agree with this. Partly because it's too generic. Did you actually read the entire article? It specifically mentions this as a anti-pattern, having tests testing the wrong thing and you should figure out what's the most important thing to test and how. Read the section "Anti-Pattern 3 - Having the wrong kind of tests" and you'll see you actually agree with the article.
- whack 8y ago
  You've mostly repeated what he said in section #3 http://blog.codepipes.com/testing/software-testing-antipatterns.html#anti-pattern-3---having-the-wrong-kind-of-tests http://blog.codepipes.com/testing/software-testing-antipatte...
- kkapelon 8y ago
  You are just repeating exactly what I am saying as anti-pattern 3
  - amarant 8y ago
    Upon re-reading, I think I may have misread large parts of your article.. Sorry about that, these are actually great tips unless you misunderstand them ;-) at first I took your comments about "the shape is not a pyramid" as there being something wrong with that and that the shape should be a pyramid.. reading a bit more carefully I now see that's the opposite of what you're saying...I blame everything on english being my second language O:-)
kalev 8y ago
These are exactly the posts I need to level-up my game. More abstract, top level views without going into detail on setting up system-x to make it work, i can figure that out myself! Any more of these types of articles you guys and girls recommend?
- kkapelon 8y ago
  Yes I wrote it because it is my main complaint. The internet is full or articles that deal with a specific topic that is very narrow and never give the big picture away. Frankly the only good source of such material is books (especially the initial chapters where they set the stage for everything) If somebody else knows where these types of articles exist, I would also like to know where they are.
  - avinium 8y ago
    Great article. I don't agree with 100% of your points but testing is a field where everyone has their own opinion shaped by their own experience, and I think you've covered every approach (along with its pros and cons) exceptionally well. OOC, how long did this take you to write?
    - kkapelon 8y ago
      I started in November. You can see the full history here https://github.com/kkapelon/kkapelon.github.io/commits/master https://github.com/kkapelon/kkapelon.github.io/commits/maste... Most of my time was spent on thinking the examples and trying to structure the content. The actual writing was very easy when I knew in my mind what I wanted to say (as is always the case with technical writing)
  - Cofike 8y ago
    After getting into some comp sci textbooks these last few months I definitely think these books are underrated for the information they provide and how they provide it.
jamesmccann 8y ago
Really enjoyed reading this article as a Sunday afternoon long-read. Well structured and covers a lot of things I have experienced but haven't formed into such a clear description. Also really appreciate the realistic examples!
- kkapelon 8y ago
  Thank you! Yes it took me a long time to write and thinking about good examples is always hard...
al2o3cr 8y ago
Got to quibble with Antipattern #9 about test code - there's certainly a place for extracting common patterns from test code (factories, complex setup) but too much DRY in tests can make for hard-to-modify spaghetti. Sometimes it's easier to just repeat a few lines of code in the name of clarity.
- diggan 8y ago
  Sure, the point is to treat test-code as "normal" production code. The things you mentioned here applies for that code as well.
- arthurdenture 8y ago
  Similarly, overly elaborate test code can itself be buggy. A specific instance of this anti-pattern is when the test code duplicates the same logic as the code under test, logic which turns out to be incorrect. In test code, I am a lot more tolerant of straight-line repetition in the name of simplicity, in a manner that would be worth abstracting were it in non-test-code.
  - rdiddly 8y ago
    That's a sign that you need to write tests to test your tests. It's part of treating your test code as a first-class citizen. I always write a test suite for my test suite. Just kidding.
    - squeaky-clean 8y ago
      I know this is a joke, but I do have a few "don't panic" tests that do extremely simple stuff, like just run a test and return True immediately, connect to db, connect to mock db, etc. Sometimes I do something stupid like put an extra comma somewhere without noticing, or update my db driver library without also updating my mock library. I think we've all had that moment where tests fail or it doesn't compile because you thought you copy-pasted some text into an email, but it got pasted into your editor window. So when I see that literally every test has failed, I know it was some really dumb but likely simple mistake on my part. If 180/195 tests pass.... Shit.
UK-Al05 8y ago
I thought we got past the idea that a unit is a method or class. The original interpretation means a unit of behaviour. When you start thinking like that, intergration tests become less important. But still important. The idea that a unit in unit testing was a class was a misinterpretation.
- kalecserk 8y ago
  Totally agree, and this misinterpretation got crystallized into your tools that often recommend/generate tests that are named TestSubjectClassNameTest->eachMethodTest. We have to pay attention to that ;)
sorokod 8y ago
"Linux command line utility ... In this contrived example you would need: Lots and lots of unit tests for the mathematical equations. Some integration tests for the CSV reading and JSON writing No UI tests because there is no UI." As with all command line apps, the command line is the UI and requires testing ( input parameters, error messages, etc... )
- kkapelon 8y ago
  You are right. I will change the wording to say GUI instead of just UI
agentultra 8y ago
> Writing tests before the implementation code implies that you are certain about your final API, which may or may not be the case. How does this myth continue to persist? Writing the test first has nothing to do with knowing the final API. When I write tests first I am looking for an API. The tests help guide me towards a nice API. I personally find TDD in this manner works best when taking a bottom-up approach with your units. I start with the lowest-level, simple operations. I then build in layers on top of the prior layers. Eventually you end up with a high-level API that starts to feel right and is already verified by the lower-level unit tests. Although this style of development has become less prevalent in my work with Haskell as I can rely on the type system to guarantee assertions I used to have to test for. This tends to make a top-down approach more amenable where I can start with a higher-level API and fill-in the lower-level details as I go.
- tome 8y ago
  > > Writing tests before the implementation code implies that you are certain about your final API, which may or may not be the case. > How does this myth continue to persist? Yes indeed. TDD helps you find a nice API.
  - barrkel 8y ago
    TDD encourages you to implement an overly complex API with too many possible configurations. It encourages delivery of a box of parts, rather than sub-assemblies.
    - spronkey 8y ago
      IMO TDD has almost exactly zero to do with the design of your API. You can TDD top down by writing an end to end test, and recursing TDD down each level until you get to a single unit, repeat until all inception layers of the tests pass. You can TDD bottom up, by starting with components you'll know you'll need, working out some APIs at their level, then doing the same to piece together larger components. Ultimately you're in control of how your API looks, and this has nothing to do with TDD itself - if you get caught up exposing a bunch of inner workings and configurations, then you're simply not designing well. It has nothing to do with TDD. Likewise, if you get caught short with an API that's too simple, well, that has little to do with TDD either.
falcolas 8y ago
A small quibble with #8 (no manual testing) - I can't recall the number of times a program has passed all the tests the creator thought to write, only to fail when someone actually looked at the output. Automated testing should not be the end of the testing. It should be the beginning. Manual testing should be the last step, even if every manual test is immediately automated - there's always more to test. Also, we should be sure to include combinatorial and fuzz testing in that pyramid as well, since skipping them leads to someone coming in with AFL and exploiting the hell out of your app.
- scaryclam 8y ago
  Agreed. The number of times that manual tests have unearthed completely unthought of situations is huge. A good QA tester will do weird and wonderful things that no developer ever thinks of doing, never mind testing for. Saying you shouldn't manually test is like saying you shouldn't shower. Sure, you can get away with it for a time, and it saves time! But eventually someone's going to point out you smell bad. Having a customer have to tell you your product is broken (smells bad), because you didn't know, because you didn't bother testing, is seriously not what you want.
pimmen 8y ago
"Anti-Pattern 2 - Having integration tests without unit tests" The first project I worked from after graduating had this problem when I came onboard, all the tests were slow as all hell and 80% of each test did the same thing (clearing the entire database and seeded in new test data, for every test). I felt like I made a mistake becoming a programmer because of how gruesome this anti-pattern is when you work with big Java applications. Then we hired a senior developer three months later who promptly started breaking up the tests after checking with the team if that was ok. The productivity of the entire team increased by many orders of magnitude!
- kkapelon 8y ago
  Yes this is exactly what I was talking about. The anti-patterns are mentioned in order of appearance, so this is very common. Don't let problems like these disappoint you. There are companies that don't suffer from any of these anti-patterns
steindavidb 8y ago
This website is unreadable on iPad, the font size changes every time I scroll.
- kkapelon 8y ago
  I have an iPad 1 and it shows just fine. Can you post a screenshot/video somewhere with the problem?
  - hcs 8y ago
    Here it is on a 5th gen iPad with iOS 10.3.3: https://youtu.be/5gqbL_fhy0U https://youtu.be/5gqbL_fhy0U Works fine with Reader View, though.
- jboles 8y ago
  Same problem, it’s a shame because it looks like a good article, but too distracting with the layout jumping around. Not sure of a way to take a screen recording on an iPad though.
koonsolo 8y ago
I don't agree with Anti-Pattern 10 - Not converting production bugs to tests I used to think this was a good idea too, until I saw the real statistics on a project on this. This project (50+ developer team) tracked all bugs, and also if they were regressions or not. Almost 0 regression occurred of bugs that were fixed before. All testing needs to consider return on investment. The reality, at least for that project, was that testing time was best spent elsewhere.
- convolvatron 8y ago
  what seems to be lacking from these discussions is the notino that forethought and consideration can win the day. if there is some brain fart, and its tracked down and fixed, the value of encoding that in a unit to be run until heat death. as you say, exactly zero. but if there is a problem area that keeps throwing bugs like a metastatic tumor - say race conditions or memory footprint, or unicode handling. in that case the return on writing tests to try to shake out those areas is pretty high. I think the real failure here is to try to substitute informed judgement calls and meaningful discussion with simple checklists of dos and donts.
  - koonsolo 8y ago
    > In that case the return on writing tests to try to shake out those areas is pretty high. I agree, but in that case you shouldn't test for that one specific case that failed, but try to make a test that handles similar issues, or tries catch bugs of the same kind. Of course it depends how far you can push this.
- hamstercat 8y ago
  Doesn't that just mean that the tests were effective? The goal of doing a test for each bug is to prevent them from happening again. Unless you meant that this project didn't apply this particular advice but was getting away with it just fine? I personally believe that this advice is one of the most important one actually for a very specific reason: it leaves a very bad impression to client when a bug keep happening again and again after being fixed. Unfortunately I've lived this experience when I was just starting out in my career on a similar project (around 45-50 developers) with basically no tests at all. It wasn't fun explaining to the client, even if they were internal to the company, that the bug we fixed last month had to be fixed again.
antoineMoPa 8y ago
In small companies where there is no time to "waste" on tests, my view is that 80% of the problems can be caught with 20% of the work by writing integration tests that cover large areas of the application. Writing unit tests would be ideal, but time-consuming. For a web project, that would involve testing all pages for HTTP 200 (< 1 hour bash script that will catch most major bugs), automatically testing most interfaces to see if filling data and clicking "save" works. Of course, for very important/dangerous/complex algorithms in the code, unit tests are useful, but generally, that represents a very low fraction of a web application's code.
- rdsubhas 8y ago
  Yep. There is a kind of goldilocks zone: - Company starts out, lots of change and RFCs, testing at a shallow depth (high-level tests wherever possible) helps a ton - [Goldilocks zone] Company and team settles down, predictable workflow, "not changing the world" anymore, unit tests help - Project goes into legacy, needs to be split/ merged/ refactored/ rearchitected/, again unit tests don't help. Back to shallow tests and stranglers Only very small part of programming today is doing hardcode datascience and algorithms. Rest are all just plumbing from A to B to C. I always argued in teams to focus on higher level tests, but then, its easy to get shouted down because of "TDD". Sadly, very few people seem to focus on objective rather than process.
  - murukesh_s 8y ago
    Yea exactly, it's often painful to see when all you need is an API level test which will anyway cover all the use cases instead of the low level unit tests written for the sake of TDD.. leaner the code more maintainable it is
  - humanrebar 8y ago
    > I always argued in teams to focus on higher level tests... In my experience, your inclinations are true, but it actually takes more discipline and buy in from leadership for it to work. If broken tests are lower priority than feature work, your test suite ends up rotting, getting ignored, then abandoned. At least with unit tests, individual contributors can maintain pockets and layers of verified-working code. I'm not arguing against any kind of test. But you do need the people handing out raises, planning sprints, scheduling deadlines, etc. to be on board for end-to-end testing to work. It also helps a lot to introduce those early in the system's life, since the team culture develops with the tests already being there.
rdsubhas 8y ago
Superb article (some small nitpicks but they are ignorable). I would like to add just one more anti-pattern: Focusing too much on fast feedback Nope. One should get priorities straight. The #1 purpose of tests is safety. The "sleep well in the night" test. The "deploy with eyes closed" test. Performance of the test suite, while "nice to have", is not the core objective. If push comes to shove, if it comes down to safety vs performance, safety should just win hands down as a principle. Doing a bunch of mocks for speeding up 80% of unit tests? Great, but its a borrowed debt, and must be balanced out with 20% of higher level tests.
- achamayou 8y ago
  Can't agree enough. Degrading the quality of the tests to make them quicker is particularly frustrating, especially when it's easy enough to split them out into something that runs at a lower frequency. Just because you can run all the tests and all the optimisations on every keypress doesn't mean you shouldn't have them.
- keithnz 8y ago
  I find this a bit of a strange statement. I started with Extreme Programming in 99 and grew and evolved through all the refinements to the process around testing. It's always been about providing safety. Fast has always been about not running silly tests that take too long Fast Feedback is a core objective. It is not in competition with safety, safety is an integral part. Safety is the feedback you are expecting. We want to know about that safety fast. A popular term that came around in the early 2000s was "Brain Engaged", meaning you needed always to be aware of why you were doing things and not following blind rules. Meaning you need to know the purpose of going fast. The whole point is to go as fast as possible safely. Some of the biggest challenges is how to get things quick while maintaining safety. Kind of makes no sense to have fast tests with no safety. Now you mention mocks, and I have seen people mock in very strange ways that devalue tests. I like the general guidance from Kent Beck "I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence"
BeetleB 8y ago
Reminding me of too many headaches at work. We have all integration tests, and no unit tests. I and several others have pushed for unit tests, but with little success. Our full test suite takes 10 hours to run. We have split it up so we can test what we think is the portion we are modifying, but we're never 100% sure something in the rest of the suite isn't dependent on our changes. I disagree a little with his complexity multiplying for his anitpattern about no unit tests. In theory, yes, you would multiply them. In practice, it is rarely the case you need to test all combinations - real code rarely looks like his diagram. In our experience with our project, the final number of branches we need to test is much closer to adding than to multiplying. I very much agree with the "don't test internal implementation". If the primary reason your tests fail is because you refactored or made API changes, your test suite is not robust. Am having to live with flaky tests right now. Horrible. Our team doesn't want to prioritize fixing them. One antipattern he left out: Making unit tests a 1:1 match with code, and insisting that a unit test should not test more than one funciton. I know the community is split, but I am very much in the camp that "unit" should not be tied to a function. Don't make it that granular.
d_burfoot 8y ago
I don't understand why people are so enamored of the distinction between "unit" tests and "integration" tests. That dichotomy has never seemed useful to me. In my mind the relevant distinction is speed. There's one fast test suite that you run before every code commit, and another slower suite that you run every night on a cloud instance or testing machine, and review the results in the morning. The fast/slow distinction is roughly comparable to the unit/integration distinction, but I don't have any problem with testing a DB interaction in the fast suite - and in fact consider it to be a good practice - if the interaction is fast enough. In general the fast suite should check as much as possible within the constraint of needing to be fast.
- kkapelon 8y ago
  >The fast/slow distinction is roughly comparable to the unit/integration distinction I think you answered the question yourself. Speed is also shown in the table that defines the types of tests. You can name the categories foo tests and bar tests if you wish. Anti-pattern 1 - is companies that have only the "fast suite" you are talking about and anti-pattern 2 is companies that have only the "slower suite".
  - nwatson 8y ago
    I'd prefer that all testing be done with live data in a real database ... maybe not at the very beginning but as soon as practicable. I came into a project recently with so many issues around differences between real data and assumed data, and unrealistic assumptions about real data that it's been hugely painful, with lots of back and forth between Dev and QA and Ops. With Docker now there's no excuse for not setting up soon in a project's lifecycle a realistic environment (infrastructure topology-wise) with real data on development and QA laptops. With this you still have fast and slow perhaps, but everybody can run all but the most extreme scaling tests.
    - kodr 8y ago
      That's E2E test. What's interesting with unit tests is that you can run them locally, offline. They are fast enough so that you can run them and see if you broke anything. If your tests take too much time to start or relies on a live env (which may be down or broken), you'll spend more time for maybe nothing.
Toine 8y ago
Refreshing article. We hear so many religious arguments about one side or the other in these days, we really need more of these. I think there's a deep issue that causes all the misunderstanding, the elephant in the room : the definition of a "unit". Words have a meaning in a certain context : if people don't mean the same thing when using the same word they're doomed to misunderstand each other forever. Just ask 5 different people what is a unit and you'll have at least 3 different definitions. The most common one is : in OOP, a unit is a class. From my experience, a unit should be defined as a much higher abstraction level than that. A better definition would be : "a set of use cases that belong to the same module". In other words, unit tests should be we written in a language as close as possible to your domain language. Or : "test your use cases, not your classes". When you do that, you usually write tests for a few major classes that use all your other classes that are just implementation details. This leads to tests that are far more reliable and easy to maintain, because they have very low coupling to the rest of your application. Typically, this means testing classes at the very edge of your app, classes that directly communicate with the end-user, usually services or something like that. Let's say you "unit test" a simple car with a steering module. It has all sorts of internal complex mechanism that IMO you generally don't need to unit test directly. What you need to know is if the business value is correctly delivered to the driver, ie : - when he turns the steering wheel left, does the car turn left - when he brakes, does the car stop - when he presses the gas pedal, does the car accelerates - etc Under the hood there are dozens of other classes that perform actions that you don't really need to care about when you test. They will be tested indirectly anyway, because they're used by the high-level classes you do test. I think many people blindly try to unit test almost all the classes they write and it leads to code duplication all over the place and all sorts of other problems that make projects fail, people angry and think unit tests are bad.
- TheCoelacanth 8y ago
  I have observed the same thing. I have seen way too many test written with the class = unit philosophy that look more like tests of the VM that is running the tests than they look like tests for the code that was supposed to be tested.
V-2 8y ago
"I have seen projects which have well designed feature code, but suffer from tests with huge code duplication, hardcoded variables, copy-paste segments and several other inefficiencies that would be considered inexcusable if found on the main code." I agree with "code duplication" and "copy-paste segments" (although I fail to see how they're two different things. It looks like example duplication to me?) I don't agree with hardcoded variables. In tests, they're okay. Tests are not production code. In this case I side with Misko Hevery - see https://youtu.be/jVxmk-tVo7M?t=2m54s https://youtu.be/jVxmk-tVo7M?t=2m54s Tests should hardcode values, and (ideally) not contain any logic, not even of the simplest sort. That's the point, that's how we can reliably confront them with production code without risking bugs that fall off off our radar because the same faulty logic leaked from production into tests. Tests should be kept all dumb, all naive, all hardcoded. That's the thing which proves to be the most difficult to convince fellow developers about, as they're typically used to mechanically transferring over all the usual best practice from production code into tests. "Ooh, this is so hardcoded" - "good". "This method name is so verbose" - "good; will we ever call this method from code?". "This could be private" - "it's a class with unit tests, how will this code possibly be called from some other class while it executes? Visibility modifiers are waste of space here". And so on
emilsedgh 8y ago
So we have a REST API created using node and most of the job happens on Postgres and sql files. Basically, 95% of the methods depend on database [and it's current state]. How can I unit test this? I've given up on unit tests. They wouldn't make any sense. E2E tests are helping us a lot but I had several attempts to have unit tests but they don't make any sense at all.
- abritinthebay 8y ago
  Those methods are still data in, data out tho right? So test that with mocked data. Unit tests should be able to run with no network connection, no other service, just your test file + data. Otherwise at that point you’re doing integration testing, not unit tests.
  - emilsedgh 8y ago
    Let's say I have a `SomeEntity.get` function. Basically it gets an array of id's and calls the database and fetches those entities. ~10 lines of Javascript code. 40 lines of SQL. Which also depends on the state of the database. So what do I gain by mocking the database? Making sure those 10 lines of javascript work fine? In a vacuum? They are the 10 easy lines anyways which are covered by my E2E test. What matters is the SQL and how it behaves in different states of the database.
    - abritinthebay 8y ago
      And that’s why you test the DB with an integration test (and likely a functional test too). For your unit test you’d just mock the dB call and make sure that function called the “db” and returned the faux data. That’s it. No one said unit tests had to be complex or test the whole stack. Quite the opposite actually.
  - filleduchaos 8y ago
    At that point, that's just testing for the sake of testing, which is just silly - nobody needs you to prove that the compiler/interpreter/library/framework in fact works the way its maintainers have already tested it to work
amleszk 8y ago
I personally don't like the word 'antipattern' because it sounds like a TDD religion term. None the less the article does a good job covering lots of the reasons writing tests will result in a bad outcome. Much like the reasons developers dislike certain architecture patterns, it's not the pattern that's at fault but the application of it by individuals and teams
- kkapelon 8y ago
  I thought about using "common pitfalls". You think this would be better?
luord 8y ago
Great read, and many points I'll consider, particularly the first anti-pattern. I consider that writing too much code that interfaces with other services (which in turn leads to more integration testing) can come from suffering NIH, as it's likely that that code might just be repeating code already provided by the api/library of that service.
grosjona 8y ago
>> Anti-Pattern 2 - Having integration tests without unit tests I strongly disagree with this one. I think that unit tests only make sense when the project's code has really settled down (not likely to change in the future) and you want to lock it down to prevent new developers on the team from accidentally breaking things. Unit tests severely slow down development. I've worked on projects where it takes 2 to 3 days to update a single property on a JSON object on a REST API endpoint because changing a single property means that you have to update a ton of unit tests. The cons of unit testing are: - It locks down your code, so if your code is not structurally perfect (which it definitely is not for most of the project's life-cycle) then you will have to keep updating the tests as you write more code and more functions around. - It encourages you to use certain patterns like dependency injection which might make sense for some (e.g. statically typed) programming languages but are unsuitable for other (e.g. dynamically typed) languages because they make it difficult to track down dependencies. - It only makes sense for parts of the project that have strict reliability requirements and where any downtime/failure in that part of the code would result in some loss of business. It's important not to underestimate the maintenance cost of unit tests. More unit tests means much slower development (cuts productivity to half or sometimes even a quarter of what it was without tests for small teams), which means that you need to hire many times more developers to get the same productivity that you could get from a single developer. Sometimes it's OK if a part of the code breaks in non-critical parts of the system; especially if you have some kind of user-feedback system in place.
- kkapelon 8y ago
  > I've worked on projects where it takes 2 to 3 days to update a single property on a JSON object on a REST API endpoint because changing a single property means that you have to update a ton of unit tests You just described anti-pattern 5. Did you read the full article?
  - grosjona 8y ago
    Anti-pattern 5 is contradictory to the rest of the article... >> Tests that need to be refactored all the time suffer from tight coupling with the main code. What is the author proposing? To write unit tests that are only 'loosely coupled' to the code that they are testing? In my entire career, I've never seen a single unit test case that matches this description. If it's loosely coupled with the internal code then by definition, it's called an integration test. Anti-pattern number 5 is basically the author admitting that internal unit testing is a problem in terms of productivity but then they fail to offer an actual solution which doesn't contradict the rest of the article. Sometimes your code needs refactoring, you need to change the fundamental structure of how some objects interact with each other and when that's the case, unit tests actually discourage you from making the necessary changes of pulling the whole class definition apart (thereby invalidating all the unit test cases for that class) and moving the code to smaller or more specialized classes.
johnwatson11218 8y ago
I have also noticed that many of the frameworks for writing unit/integration tests are so flexible that developers will easily develop a mini-framework that makes their specif tests able to be expressed in a declarative manner. The downside is that their setup and system isn't really usable for other tests. I think devs need to keep it simple, even it leads to more code, so that a wider range of tests can be written with the same abstractions.
keithnz 8y ago
One thing that seems to be not really commented on is "Play Testing / Exploratory Testing" ( and semi related, "dog fooding"). This is the manual test process where someone actually uses the software and looks for problems. Problems are then captured as a new automated test case. This is about finding undesirable unforeseen consequences ( sometimes you get desirable ones! though if it is desirable, you will want to capture it in a test also ).
fadzlan 8y ago
I understand that if the test hits database, then its an integration test. But what if a whole load of the logics are in stored procedure? Surely those needs test too. Of course, one can make an argument that depending too much on stored procedures is an anti pattern....
- atom-morgan 8y ago
  The stored procedure should be tested within the domain of the stored procedure. The consumer of its result just tests its ability to use the result it should be given (fulfilling its end of the contract).
bni 8y ago
Unit tests is a tool to help you write the implementations and verify it works like you intended to. You could do the same thing by "F5"-ing it, that is running the full app, but the problem with that it is slower and above all NOT repeatable later, without significant setup time. Running the test, sets breakpoints and debug implementation by running the unit test, is by far the fastest way to develop, while also having atleast some assurance that the code you are writing is "correct"
Yizahi 8y ago
Anti-pattern 14 - generalizing all testing in the industry to e-shops and websites
- kkapelon 8y ago
  I am open to other suggestions. I have really tried to spend a lot of time to another example and cannot seem to find any. The other alternative would be a loan approval application. Remember that the definition of a developer is very broad nowadays. You have hackers working with C/Assembly on firmware, all the way to AI/ML with high level languages. Do you have another suggestion for a good example?
sethammons 8y ago
> Anti-Pattern 5 - Testing internal implementation I run up against this the most. The internal state often does not matter. What is important is the bevhavior. Refactoring internal state should not break a bunch of tests.
- rimliu 8y ago
  Then you will probably like this: https://www.youtube.com/watch?v=URSWYvyc42M https://www.youtube.com/watch?v=URSWYvyc42M
  - sethammons 8y ago
    Blast from the past; I had forgotten about that one. Thanks
adamlett 8y ago
I had hoped that this would be a catalog of concrete code examples of common or easy to make mistakes, along with concrete prescriptions for what to instead. Instead it turns out that it is mostly a long winded collection of opinions passed off as insights along with a few general principles that basically amount to good tests are good and bad tests are bad. Who reads this and is any wiser as to how to write valuable tests? Writing tests that add value is hard, is a skill that must be acquired. In my experience, most developers who have given up on testing have done so, because they’d been sold the false notion that testing is trivial and always adds value. What they’ve found instead is that testing is difficult (because they lack the skill), and that the tests they did write, were of questionable value. No wonder they gave up on it. What these people need is not another article talking in loose terms about the wonders and virtues of testing.
- kkapelon 8y ago
  >of concrete code examples Examples of what? Python? Ruby? Java? C++? You cannot please everybody. The article is written in a way that touches all developers. And judging by the feedback I got it has succeeded in this way. >that basically amount to good tests are good and bad tests are bad And the article helps people to understand which tests are good and which are bad in the first place. If you are already an expert on the subject then maybe you are the wrong audience. >Who reads this and is any wiser as to how to write valuable tests? If you think you can do better then by all means I am expecting your article on the subject >What these people need is not another article talking in loose terms about the wonders and virtues of testing. Please write the correct article then.
dwheeler 8y ago
Most of this article I agree with, such as the need for BOTH unit and integration testing, the need to focus on automation, and the need to turn regressions (bugs) into new tests. I also agree that "Paying excessive attention to test coverage" is not good. However, I completely disagree with much of its supporting text. If your test code coverage is only 20%, then by DEFINITION your tests are awful. That would mean that 80% of your code is completely untested. I agree that for many programs 100% code coverage is not worth the effort, because those last few percentages cost more than their benefit, but that doesn't mean that such low coverage makes sense. Most organizations I've worked with recommend at least 80% statement coverage, as a rule of thumb. I haven't seen any studies justifying this, but this essay doesn't cite anything to justify its claims either :-). You'd want much higher statement coverage, and also measure branch coverage, if software errors are serious (e.g., if someone could be physically harmed by an error). You should focus on creating good bang-for-buck tests first; code coverage is then a useful tool to help identify "code that isn't getting well-tested at all." It's also useful as a warning: 100% coverage may still be poorly tested, but low coverage (say less than 70%) means the program definitely has a terrible test suite. This statement is misleading: "You can have a project with 100% code coverage that still has bugs and problems." That's true for ANY program, regardless of its testing regime, because any testing regime can only test an astronomically small fraction of the possible input space. A program that just adds 2 64-bit numbers has 2^128 possible inputs; real programs have more.
- kkapelon 8y ago
  >That would mean that 80% of your code is completely untested That is not always a bad thing. Maybe depending on the application this 80% is trivial and never breaks. It is explained in anti-pattern 4 that you should start with the critical code first. >Most organizations I've worked with recommend at least 80% statement coverage, as a rule of thumb This number is making MANY assumptions. I would demand different code coverage from an application that runs on a nuclear reactor and from an application that is used as point of sale in a small pizza restaurant.

465 points by kkapelon 8y ago

In cases where manual testing is the only option, is it an anti-pattern to let the testing be done by the person who developed the code? As a developer, I feel that others should test my code. Is that reasonable?

mic47 8y ago
Why should other test your code?
- ben-schaaf 8y ago
  Same reason you don't review your own code, or pair program by talking to yourself.
  - koolba 8y ago
    > ... or pair program by talking to yourself. Who says I don’t? https://en.m.wikipedia.org/wiki/Rubber_duck_debugging https://en.m.wikipedia.org/wiki/Rubber_duck_debugging
    - icebraining 8y ago
      It's not pair programming, though, even if can have some of its advantages.
  - mic47 8y ago
    You can solve this by test plan: You write into test plan what you have tested, and others in code review can suggest/require to test other things you missed. In my previous job this worked quite well.
    - icebraining 8y ago
      That's fair enough; I think when people are talking about manual testing, they're thinking about the decision of what should be tested, not the actual button pressing. But it's a distinction worth making.
  - stinos 8y ago
    This seems a bit of a false generalization to me. For a long time I've been the sole developer on multiple large-ish applications. So of course I review my own code, and I write my own tests. And yes as another commenter mentions: if you don't pay attention this can lead to tunnel vision, and yes it can be hard to be consistent about it. I discovered those drawbacks fairly early, so quickly made it a habit to be super-strict and critical about pretty much every single line of code I write (both with regards to style, functionality, how it adheres to the standard practices etc). What I also do consistently is review older code: if I have to add a function to existing code, it happens more often than not that I'll re-read and review the surrounding bits. Often to come up with better code. If time permits, else I take note and go back to it later. In practice it turns out that if I haven't touched code for months, reviewing it is almost the same as reviewing someone else's code, i.e. starting with a clean slate. Which is also why I'll sometimes write something, commit it, but not yet merge it to production. Then a week or so later I'll come back and review every bit again.
    - ben-schaaf 8y ago
      Oh, I completely agree, though I wouldn't call writing code carefully and methodically a "code review" if that makes sense. I personally find that me a couple months down the track is pretty much another person anyway. In a sense you're not reviewing your own code, you're letting it be reviewed by future you. But if you're in a team with multiple developers, having someone else review your code would be more efficient.
      - _asummers 8y ago
        I tend to review my code with a reviewer hat, not with the author hat, before I open the PR. Find all the places where I'd call myself out for lazy names, poor abstractions, missing docs, etc. then go in and fix those. Repeat until satisfied, then open PR.
    - rb808 8y ago
      > In practice it turns out that if I haven't touched code for months, reviewing it is almost the same as reviewing someone else's code, I really like this idea, haven't heard it before but it makes complete sense, thank you.
  - rasjani 8y ago
    Yellow rubber duck makes pair programming way more socially awkward ;)
tempodox 8y ago
Manually testing your own code is liable to tunnel vision. So yes, having others do it is safer. There are even production standards that make this mandatory.
dankohn1 8y ago
Why would manual testing be the only option? As an example, we wanted to automate the test for browser-based OAuth connections to GitHub and authorizing an app there. Here is the test that does so: https://github.com/coreinfrastructure/best-practices-badge/blob/master/test/features/github_login_test.rb https://github.com/coreinfrastructure/best-practices-badge/b...
kalecserk 8y ago
I have been working with teams that heavily use automated tests for over 16 years now and, at least on my experience, making others test your code would lead to untestable code and delegation of responsibility. Even for manual tests, the test spec should make part of the change request and be reviewed appropriately. Cheers!

amarant 8y ago

I can't say I agree with this. Partly because it's too generic. I'm of the opinion that you should focus on testing what you deliver. If you're building a framework or library, then unit tests are probably the way to go. If you're developing a micro service that's primarily going to be called by other services, built by other teams for example, you should probably focus on integration tests. If you have an external API, you should focus on e2e tests, as your clients won't know or care that you have several individually well tested microservices behind your API or if it's a monolith. They only care that a given API call has the expected result. The pyramid thing is pretty, but also meaningless IMHO.

diggan 8y ago
> I can't say I agree with this. Partly because it's too generic. Did you actually read the entire article? It specifically mentions this as a anti-pattern, having tests testing the wrong thing and you should figure out what's the most important thing to test and how. Read the section "Anti-Pattern 3 - Having the wrong kind of tests" and you'll see you actually agree with the article.
whack 8y ago
You've mostly repeated what he said in section #3 http://blog.codepipes.com/testing/software-testing-antipatterns.html#anti-pattern-3---having-the-wrong-kind-of-tests http://blog.codepipes.com/testing/software-testing-antipatte...
kkapelon 8y ago
You are just repeating exactly what I am saying as anti-pattern 3
- amarant 8y ago
  Upon re-reading, I think I may have misread large parts of your article.. Sorry about that, these are actually great tips unless you misunderstand them ;-) at first I took your comments about "the shape is not a pyramid" as there being something wrong with that and that the shape should be a pyramid.. reading a bit more carefully I now see that's the opposite of what you're saying...I blame everything on english being my second language O:-)

kalev 8y ago

These are exactly the posts I need to level-up my game. More abstract, top level views without going into detail on setting up system-x to make it work, i can figure that out myself! Any more of these types of articles you guys and girls recommend?

kkapelon 8y ago
Yes I wrote it because it is my main complaint. The internet is full or articles that deal with a specific topic that is very narrow and never give the big picture away. Frankly the only good source of such material is books (especially the initial chapters where they set the stage for everything) If somebody else knows where these types of articles exist, I would also like to know where they are.
- avinium 8y ago
  Great article. I don't agree with 100% of your points but testing is a field where everyone has their own opinion shaped by their own experience, and I think you've covered every approach (along with its pros and cons) exceptionally well. OOC, how long did this take you to write?
  - kkapelon 8y ago
    I started in November. You can see the full history here https://github.com/kkapelon/kkapelon.github.io/commits/master https://github.com/kkapelon/kkapelon.github.io/commits/maste... Most of my time was spent on thinking the examples and trying to structure the content. The actual writing was very easy when I knew in my mind what I wanted to say (as is always the case with technical writing)
- Cofike 8y ago
  After getting into some comp sci textbooks these last few months I definitely think these books are underrated for the information they provide and how they provide it.

jamesmccann 8y ago

Really enjoyed reading this article as a Sunday afternoon long-read. Well structured and covers a lot of things I have experienced but haven't formed into such a clear description. Also really appreciate the realistic examples!

kkapelon 8y ago
Thank you! Yes it took me a long time to write and thinking about good examples is always hard...

al2o3cr 8y ago

Got to quibble with Antipattern #9 about test code - there's certainly a place for extracting common patterns from test code (factories, complex setup) but too much DRY in tests can make for hard-to-modify spaghetti. Sometimes it's easier to just repeat a few lines of code in the name of clarity.

diggan 8y ago
Sure, the point is to treat test-code as "normal" production code. The things you mentioned here applies for that code as well.
arthurdenture 8y ago
Similarly, overly elaborate test code can itself be buggy. A specific instance of this anti-pattern is when the test code duplicates the same logic as the code under test, logic which turns out to be incorrect. In test code, I am a lot more tolerant of straight-line repetition in the name of simplicity, in a manner that would be worth abstracting were it in non-test-code.
- rdiddly 8y ago
  That's a sign that you need to write tests to test your tests. It's part of treating your test code as a first-class citizen. I always write a test suite for my test suite. Just kidding.
  - squeaky-clean 8y ago
    I know this is a joke, but I do have a few "don't panic" tests that do extremely simple stuff, like just run a test and return True immediately, connect to db, connect to mock db, etc. Sometimes I do something stupid like put an extra comma somewhere without noticing, or update my db driver library without also updating my mock library. I think we've all had that moment where tests fail or it doesn't compile because you thought you copy-pasted some text into an email, but it got pasted into your editor window. So when I see that literally every test has failed, I know it was some really dumb but likely simple mistake on my part. If 180/195 tests pass.... Shit.

UK-Al05 8y ago

I thought we got past the idea that a unit is a method or class. The original interpretation means a unit of behaviour. When you start thinking like that, intergration tests become less important. But still important. The idea that a unit in unit testing was a class was a misinterpretation.

kalecserk 8y ago
Totally agree, and this misinterpretation got crystallized into your tools that often recommend/generate tests that are named TestSubjectClassNameTest->eachMethodTest. We have to pay attention to that ;)

sorokod 8y ago

"Linux command line utility ... In this contrived example you would need: Lots and lots of unit tests for the mathematical equations. Some integration tests for the CSV reading and JSON writing No UI tests because there is no UI." As with all command line apps, the command line is the UI and requires testing ( input parameters, error messages, etc... )

kkapelon 8y ago
You are right. I will change the wording to say GUI instead of just UI

agentultra 8y ago

> Writing tests before the implementation code implies that you are certain about your final API, which may or may not be the case. How does this myth continue to persist? Writing the test first has nothing to do with knowing the final API. When I write tests first I am looking for an API. The tests help guide me towards a nice API. I personally find TDD in this manner works best when taking a bottom-up approach with your units. I start with the lowest-level, simple operations. I then build in layers on top of the prior layers. Eventually you end up with a high-level API that starts to feel right and is already verified by the lower-level unit tests. Although this style of development has become less prevalent in my work with Haskell as I can rely on the type system to guarantee assertions I used to have to test for. This tends to make a top-down approach more amenable where I can start with a higher-level API and fill-in the lower-level details as I go.

tome 8y ago
> > Writing tests before the implementation code implies that you are certain about your final API, which may or may not be the case. > How does this myth continue to persist? Yes indeed. TDD helps you find a nice API.
- barrkel 8y ago
  TDD encourages you to implement an overly complex API with too many possible configurations. It encourages delivery of a box of parts, rather than sub-assemblies.
  - spronkey 8y ago
    IMO TDD has almost exactly zero to do with the design of your API. You can TDD top down by writing an end to end test, and recursing TDD down each level until you get to a single unit, repeat until all inception layers of the tests pass. You can TDD bottom up, by starting with components you'll know you'll need, working out some APIs at their level, then doing the same to piece together larger components. Ultimately you're in control of how your API looks, and this has nothing to do with TDD itself - if you get caught up exposing a bunch of inner workings and configurations, then you're simply not designing well. It has nothing to do with TDD. Likewise, if you get caught short with an API that's too simple, well, that has little to do with TDD either.

falcolas 8y ago

A small quibble with #8 (no manual testing) - I can't recall the number of times a program has passed all the tests the creator thought to write, only to fail when someone actually looked at the output. Automated testing should not be the end of the testing. It should be the beginning. Manual testing should be the last step, even if every manual test is immediately automated - there's always more to test. Also, we should be sure to include combinatorial and fuzz testing in that pyramid as well, since skipping them leads to someone coming in with AFL and exploiting the hell out of your app.

scaryclam 8y ago
Agreed. The number of times that manual tests have unearthed completely unthought of situations is huge. A good QA tester will do weird and wonderful things that no developer ever thinks of doing, never mind testing for. Saying you shouldn't manually test is like saying you shouldn't shower. Sure, you can get away with it for a time, and it saves time! But eventually someone's going to point out you smell bad. Having a customer have to tell you your product is broken (smells bad), because you didn't know, because you didn't bother testing, is seriously not what you want.

pimmen 8y ago

"Anti-Pattern 2 - Having integration tests without unit tests" The first project I worked from after graduating had this problem when I came onboard, all the tests were slow as all hell and 80% of each test did the same thing (clearing the entire database and seeded in new test data, for every test). I felt like I made a mistake becoming a programmer because of how gruesome this anti-pattern is when you work with big Java applications. Then we hired a senior developer three months later who promptly started breaking up the tests after checking with the team if that was ok. The productivity of the entire team increased by many orders of magnitude!

kkapelon 8y ago
Yes this is exactly what I was talking about. The anti-patterns are mentioned in order of appearance, so this is very common. Don't let problems like these disappoint you. There are companies that don't suffer from any of these anti-patterns

steindavidb 8y ago

This website is unreadable on iPad, the font size changes every time I scroll.

kkapelon 8y ago
I have an iPad 1 and it shows just fine. Can you post a screenshot/video somewhere with the problem?
- hcs 8y ago
  Here it is on a 5th gen iPad with iOS 10.3.3: https://youtu.be/5gqbL_fhy0U https://youtu.be/5gqbL_fhy0U Works fine with Reader View, though.
jboles 8y ago
Same problem, it’s a shame because it looks like a good article, but too distracting with the layout jumping around. Not sure of a way to take a screen recording on an iPad though.

koonsolo 8y ago

I don't agree with Anti-Pattern 10 - Not converting production bugs to tests I used to think this was a good idea too, until I saw the real statistics on a project on this. This project (50+ developer team) tracked all bugs, and also if they were regressions or not. Almost 0 regression occurred of bugs that were fixed before. All testing needs to consider return on investment. The reality, at least for that project, was that testing time was best spent elsewhere.

convolvatron 8y ago
what seems to be lacking from these discussions is the notino that forethought and consideration can win the day. if there is some brain fart, and its tracked down and fixed, the value of encoding that in a unit to be run until heat death. as you say, exactly zero. but if there is a problem area that keeps throwing bugs like a metastatic tumor - say race conditions or memory footprint, or unicode handling. in that case the return on writing tests to try to shake out those areas is pretty high. I think the real failure here is to try to substitute informed judgement calls and meaningful discussion with simple checklists of dos and donts.
- koonsolo 8y ago
  > In that case the return on writing tests to try to shake out those areas is pretty high. I agree, but in that case you shouldn't test for that one specific case that failed, but try to make a test that handles similar issues, or tries catch bugs of the same kind. Of course it depends how far you can push this.
hamstercat 8y ago
Doesn't that just mean that the tests were effective? The goal of doing a test for each bug is to prevent them from happening again. Unless you meant that this project didn't apply this particular advice but was getting away with it just fine? I personally believe that this advice is one of the most important one actually for a very specific reason: it leaves a very bad impression to client when a bug keep happening again and again after being fixed. Unfortunately I've lived this experience when I was just starting out in my career on a similar project (around 45-50 developers) with basically no tests at all. It wasn't fun explaining to the client, even if they were internal to the company, that the bug we fixed last month had to be fixed again.

antoineMoPa 8y ago

In small companies where there is no time to "waste" on tests, my view is that 80% of the problems can be caught with 20% of the work by writing integration tests that cover large areas of the application. Writing unit tests would be ideal, but time-consuming. For a web project, that would involve testing all pages for HTTP 200 (< 1 hour bash script that will catch most major bugs), automatically testing most interfaces to see if filling data and clicking "save" works. Of course, for very important/dangerous/complex algorithms in the code, unit tests are useful, but generally, that represents a very low fraction of a web application's code.

rdsubhas 8y ago
Yep. There is a kind of goldilocks zone: - Company starts out, lots of change and RFCs, testing at a shallow depth (high-level tests wherever possible) helps a ton - [Goldilocks zone] Company and team settles down, predictable workflow, "not changing the world" anymore, unit tests help - Project goes into legacy, needs to be split/ merged/ refactored/ rearchitected/, again unit tests don't help. Back to shallow tests and stranglers Only very small part of programming today is doing hardcode datascience and algorithms. Rest are all just plumbing from A to B to C. I always argued in teams to focus on higher level tests, but then, its easy to get shouted down because of "TDD". Sadly, very few people seem to focus on objective rather than process.
- murukesh_s 8y ago
  Yea exactly, it's often painful to see when all you need is an API level test which will anyway cover all the use cases instead of the low level unit tests written for the sake of TDD.. leaner the code more maintainable it is
- humanrebar 8y ago
  > I always argued in teams to focus on higher level tests... In my experience, your inclinations are true, but it actually takes more discipline and buy in from leadership for it to work. If broken tests are lower priority than feature work, your test suite ends up rotting, getting ignored, then abandoned. At least with unit tests, individual contributors can maintain pockets and layers of verified-working code. I'm not arguing against any kind of test. But you do need the people handing out raises, planning sprints, scheduling deadlines, etc. to be on board for end-to-end testing to work. It also helps a lot to introduce those early in the system's life, since the team culture develops with the tests already being there.

rdsubhas 8y ago

Superb article (some small nitpicks but they are ignorable). I would like to add just one more anti-pattern: Focusing too much on fast feedback Nope. One should get priorities straight. The #1 purpose of tests is safety. The "sleep well in the night" test. The "deploy with eyes closed" test. Performance of the test suite, while "nice to have", is not the core objective. If push comes to shove, if it comes down to safety vs performance, safety should just win hands down as a principle. Doing a bunch of mocks for speeding up 80% of unit tests? Great, but its a borrowed debt, and must be balanced out with 20% of higher level tests.

achamayou 8y ago
Can't agree enough. Degrading the quality of the tests to make them quicker is particularly frustrating, especially when it's easy enough to split them out into something that runs at a lower frequency. Just because you can run all the tests and all the optimisations on every keypress doesn't mean you shouldn't have them.
keithnz 8y ago
I find this a bit of a strange statement. I started with Extreme Programming in 99 and grew and evolved through all the refinements to the process around testing. It's always been about providing safety. Fast has always been about not running silly tests that take too long Fast Feedback is a core objective. It is not in competition with safety, safety is an integral part. Safety is the feedback you are expecting. We want to know about that safety fast. A popular term that came around in the early 2000s was "Brain Engaged", meaning you needed always to be aware of why you were doing things and not following blind rules. Meaning you need to know the purpose of going fast. The whole point is to go as fast as possible safely. Some of the biggest challenges is how to get things quick while maintaining safety. Kind of makes no sense to have fast tests with no safety. Now you mention mocks, and I have seen people mock in very strange ways that devalue tests. I like the general guidance from Kent Beck "I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence"

BeetleB 8y ago

Reminding me of too many headaches at work. We have all integration tests, and no unit tests. I and several others have pushed for unit tests, but with little success. Our full test suite takes 10 hours to run. We have split it up so we can test what we think is the portion we are modifying, but we're never 100% sure something in the rest of the suite isn't dependent on our changes. I disagree a little with his complexity multiplying for his anitpattern about no unit tests. In theory, yes, you would multiply them. In practice, it is rarely the case you need to test all combinations - real code rarely looks like his diagram. In our experience with our project, the final number of branches we need to test is much closer to adding than to multiplying. I very much agree with the "don't test internal implementation". If the primary reason your tests fail is because you refactored or made API changes, your test suite is not robust. Am having to live with flaky tests right now. Horrible. Our team doesn't want to prioritize fixing them. One antipattern he left out: Making unit tests a 1:1 match with code, and insisting that a unit test should not test more than one funciton. I know the community is split, but I am very much in the camp that "unit" should not be tied to a function. Don't make it that granular.

d_burfoot 8y ago

I don't understand why people are so enamored of the distinction between "unit" tests and "integration" tests. That dichotomy has never seemed useful to me. In my mind the relevant distinction is speed. There's one fast test suite that you run before every code commit, and another slower suite that you run every night on a cloud instance or testing machine, and review the results in the morning. The fast/slow distinction is roughly comparable to the unit/integration distinction, but I don't have any problem with testing a DB interaction in the fast suite - and in fact consider it to be a good practice - if the interaction is fast enough. In general the fast suite should check as much as possible within the constraint of needing to be fast.

kkapelon 8y ago
>The fast/slow distinction is roughly comparable to the unit/integration distinction I think you answered the question yourself. Speed is also shown in the table that defines the types of tests. You can name the categories foo tests and bar tests if you wish. Anti-pattern 1 - is companies that have only the "fast suite" you are talking about and anti-pattern 2 is companies that have only the "slower suite".
- nwatson 8y ago
  I'd prefer that all testing be done with live data in a real database ... maybe not at the very beginning but as soon as practicable. I came into a project recently with so many issues around differences between real data and assumed data, and unrealistic assumptions about real data that it's been hugely painful, with lots of back and forth between Dev and QA and Ops. With Docker now there's no excuse for not setting up soon in a project's lifecycle a realistic environment (infrastructure topology-wise) with real data on development and QA laptops. With this you still have fast and slow perhaps, but everybody can run all but the most extreme scaling tests.
  - kodr 8y ago
    That's E2E test. What's interesting with unit tests is that you can run them locally, offline. They are fast enough so that you can run them and see if you broke anything. If your tests take too much time to start or relies on a live env (which may be down or broken), you'll spend more time for maybe nothing.

Toine 8y ago

Refreshing article. We hear so many religious arguments about one side or the other in these days, we really need more of these. I think there's a deep issue that causes all the misunderstanding, the elephant in the room : the definition of a "unit". Words have a meaning in a certain context : if people don't mean the same thing when using the same word they're doomed to misunderstand each other forever. Just ask 5 different people what is a unit and you'll have at least 3 different definitions. The most common one is : in OOP, a unit is a class. From my experience, a unit should be defined as a much higher abstraction level than that. A better definition would be : "a set of use cases that belong to the same module". In other words, unit tests should be we written in a language as close as possible to your domain language. Or : "test your use cases, not your classes". When you do that, you usually write tests for a few major classes that use all your other classes that are just implementation details. This leads to tests that are far more reliable and easy to maintain, because they have very low coupling to the rest of your application. Typically, this means testing classes at the very edge of your app, classes that directly communicate with the end-user, usually services or something like that. Let's say you "unit test" a simple car with a steering module. It has all sorts of internal complex mechanism that IMO you generally don't need to unit test directly. What you need to know is if the business value is correctly delivered to the driver, ie : - when he turns the steering wheel left, does the car turn left - when he brakes, does the car stop - when he presses the gas pedal, does the car accelerates - etc Under the hood there are dozens of other classes that perform actions that you don't really need to care about when you test. They will be tested indirectly anyway, because they're used by the high-level classes you do test. I think many people blindly try to unit test almost all the classes they write and it leads to code duplication all over the place and all sorts of other problems that make projects fail, people angry and think unit tests are bad.

TheCoelacanth 8y ago
I have observed the same thing. I have seen way too many test written with the class = unit philosophy that look more like tests of the VM that is running the tests than they look like tests for the code that was supposed to be tested.

V-2 8y ago

"I have seen projects which have well designed feature code, but suffer from tests with huge code duplication, hardcoded variables, copy-paste segments and several other inefficiencies that would be considered inexcusable if found on the main code." I agree with "code duplication" and "copy-paste segments" (although I fail to see how they're two different things. It looks like example duplication to me?) I don't agree with hardcoded variables. In tests, they're okay. Tests are not production code. In this case I side with Misko Hevery - see https://youtu.be/jVxmk-tVo7M?t=2m54s https://youtu.be/jVxmk-tVo7M?t=2m54s Tests should hardcode values, and (ideally) not contain any logic, not even of the simplest sort. That's the point, that's how we can reliably confront them with production code without risking bugs that fall off off our radar because the same faulty logic leaked from production into tests. Tests should be kept all dumb, all naive, all hardcoded. That's the thing which proves to be the most difficult to convince fellow developers about, as they're typically used to mechanically transferring over all the usual best practice from production code into tests. "Ooh, this is so hardcoded" - "good". "This method name is so verbose" - "good; will we ever call this method from code?". "This could be private" - "it's a class with unit tests, how will this code possibly be called from some other class while it executes? Visibility modifiers are waste of space here". And so on

emilsedgh 8y ago

So we have a REST API created using node and most of the job happens on Postgres and sql files. Basically, 95% of the methods depend on database [and it's current state]. How can I unit test this? I've given up on unit tests. They wouldn't make any sense. E2E tests are helping us a lot but I had several attempts to have unit tests but they don't make any sense at all.

abritinthebay 8y ago
Those methods are still data in, data out tho right? So test that with mocked data. Unit tests should be able to run with no network connection, no other service, just your test file + data. Otherwise at that point you’re doing integration testing, not unit tests.
- emilsedgh 8y ago
  Let's say I have a `SomeEntity.get` function. Basically it gets an array of id's and calls the database and fetches those entities. ~10 lines of Javascript code. 40 lines of SQL. Which also depends on the state of the database. So what do I gain by mocking the database? Making sure those 10 lines of javascript work fine? In a vacuum? They are the 10 easy lines anyways which are covered by my E2E test. What matters is the SQL and how it behaves in different states of the database.
  - abritinthebay 8y ago
    And that’s why you test the DB with an integration test (and likely a functional test too). For your unit test you’d just mock the dB call and make sure that function called the “db” and returned the faux data. That’s it. No one said unit tests had to be complex or test the whole stack. Quite the opposite actually.
- filleduchaos 8y ago
  At that point, that's just testing for the sake of testing, which is just silly - nobody needs you to prove that the compiler/interpreter/library/framework in fact works the way its maintainers have already tested it to work

amleszk 8y ago

I personally don't like the word 'antipattern' because it sounds like a TDD religion term. None the less the article does a good job covering lots of the reasons writing tests will result in a bad outcome. Much like the reasons developers dislike certain architecture patterns, it's not the pattern that's at fault but the application of it by individuals and teams

kkapelon 8y ago
I thought about using "common pitfalls". You think this would be better?

luord 8y ago

Great read, and many points I'll consider, particularly the first anti-pattern. I consider that writing too much code that interfaces with other services (which in turn leads to more integration testing) can come from suffering NIH, as it's likely that that code might just be repeating code already provided by the api/library of that service.

grosjona 8y ago

>> Anti-Pattern 2 - Having integration tests without unit tests I strongly disagree with this one. I think that unit tests only make sense when the project's code has really settled down (not likely to change in the future) and you want to lock it down to prevent new developers on the team from accidentally breaking things. Unit tests severely slow down development. I've worked on projects where it takes 2 to 3 days to update a single property on a JSON object on a REST API endpoint because changing a single property means that you have to update a ton of unit tests. The cons of unit testing are: - It locks down your code, so if your code is not structurally perfect (which it definitely is not for most of the project's life-cycle) then you will have to keep updating the tests as you write more code and more functions around. - It encourages you to use certain patterns like dependency injection which might make sense for some (e.g. statically typed) programming languages but are unsuitable for other (e.g. dynamically typed) languages because they make it difficult to track down dependencies. - It only makes sense for parts of the project that have strict reliability requirements and where any downtime/failure in that part of the code would result in some loss of business. It's important not to underestimate the maintenance cost of unit tests. More unit tests means much slower development (cuts productivity to half or sometimes even a quarter of what it was without tests for small teams), which means that you need to hire many times more developers to get the same productivity that you could get from a single developer. Sometimes it's OK if a part of the code breaks in non-critical parts of the system; especially if you have some kind of user-feedback system in place.

kkapelon 8y ago
> I've worked on projects where it takes 2 to 3 days to update a single property on a JSON object on a REST API endpoint because changing a single property means that you have to update a ton of unit tests You just described anti-pattern 5. Did you read the full article?
- grosjona 8y ago
  Anti-pattern 5 is contradictory to the rest of the article... >> Tests that need to be refactored all the time suffer from tight coupling with the main code. What is the author proposing? To write unit tests that are only 'loosely coupled' to the code that they are testing? In my entire career, I've never seen a single unit test case that matches this description. If it's loosely coupled with the internal code then by definition, it's called an integration test. Anti-pattern number 5 is basically the author admitting that internal unit testing is a problem in terms of productivity but then they fail to offer an actual solution which doesn't contradict the rest of the article. Sometimes your code needs refactoring, you need to change the fundamental structure of how some objects interact with each other and when that's the case, unit tests actually discourage you from making the necessary changes of pulling the whole class definition apart (thereby invalidating all the unit test cases for that class) and moving the code to smaller or more specialized classes.

johnwatson11218 8y ago

I have also noticed that many of the frameworks for writing unit/integration tests are so flexible that developers will easily develop a mini-framework that makes their specif tests able to be expressed in a declarative manner. The downside is that their setup and system isn't really usable for other tests. I think devs need to keep it simple, even it leads to more code, so that a wider range of tests can be written with the same abstractions.

keithnz 8y ago

One thing that seems to be not really commented on is "Play Testing / Exploratory Testing" ( and semi related, "dog fooding"). This is the manual test process where someone actually uses the software and looks for problems. Problems are then captured as a new automated test case. This is about finding undesirable unforeseen consequences ( sometimes you get desirable ones! though if it is desirable, you will want to capture it in a test also ).

fadzlan 8y ago

I understand that if the test hits database, then its an integration test. But what if a whole load of the logics are in stored procedure? Surely those needs test too. Of course, one can make an argument that depending too much on stored procedures is an anti pattern....

atom-morgan 8y ago
The stored procedure should be tested within the domain of the stored procedure. The consumer of its result just tests its ability to use the result it should be given (fulfilling its end of the contract).

bni 8y ago

Unit tests is a tool to help you write the implementations and verify it works like you intended to. You could do the same thing by "F5"-ing it, that is running the full app, but the problem with that it is slower and above all NOT repeatable later, without significant setup time. Running the test, sets breakpoints and debug implementation by running the unit test, is by far the fastest way to develop, while also having atleast some assurance that the code you are writing is "correct"

Yizahi 8y ago

Anti-pattern 14 - generalizing all testing in the industry to e-shops and websites

kkapelon 8y ago
I am open to other suggestions. I have really tried to spend a lot of time to another example and cannot seem to find any. The other alternative would be a loan approval application. Remember that the definition of a developer is very broad nowadays. You have hackers working with C/Assembly on firmware, all the way to AI/ML with high level languages. Do you have another suggestion for a good example?

sethammons 8y ago

> Anti-Pattern 5 - Testing internal implementation I run up against this the most. The internal state often does not matter. What is important is the bevhavior. Refactoring internal state should not break a bunch of tests.

rimliu 8y ago
Then you will probably like this: https://www.youtube.com/watch?v=URSWYvyc42M https://www.youtube.com/watch?v=URSWYvyc42M
- sethammons 8y ago
  Blast from the past; I had forgotten about that one. Thanks

adamlett 8y ago

I had hoped that this would be a catalog of concrete code examples of common or easy to make mistakes, along with concrete prescriptions for what to instead. Instead it turns out that it is mostly a long winded collection of opinions passed off as insights along with a few general principles that basically amount to good tests are good and bad tests are bad. Who reads this and is any wiser as to how to write valuable tests? Writing tests that add value is hard, is a skill that must be acquired. In my experience, most developers who have given up on testing have done so, because they’d been sold the false notion that testing is trivial and always adds value. What they’ve found instead is that testing is difficult (because they lack the skill), and that the tests they did write, were of questionable value. No wonder they gave up on it. What these people need is not another article talking in loose terms about the wonders and virtues of testing.

kkapelon 8y ago
>of concrete code examples Examples of what? Python? Ruby? Java? C++? You cannot please everybody. The article is written in a way that touches all developers. And judging by the feedback I got it has succeeded in this way. >that basically amount to good tests are good and bad tests are bad And the article helps people to understand which tests are good and which are bad in the first place. If you are already an expert on the subject then maybe you are the wrong audience. >Who reads this and is any wiser as to how to write valuable tests? If you think you can do better then by all means I am expecting your article on the subject >What these people need is not another article talking in loose terms about the wonders and virtues of testing. Please write the correct article then.

dwheeler 8y ago

Most of this article I agree with, such as the need for BOTH unit and integration testing, the need to focus on automation, and the need to turn regressions (bugs) into new tests. I also agree that "Paying excessive attention to test coverage" is not good. However, I completely disagree with much of its supporting text. If your test code coverage is only 20%, then by DEFINITION your tests are awful. That would mean that 80% of your code is completely untested. I agree that for many programs 100% code coverage is not worth the effort, because those last few percentages cost more than their benefit, but that doesn't mean that such low coverage makes sense. Most organizations I've worked with recommend at least 80% statement coverage, as a rule of thumb. I haven't seen any studies justifying this, but this essay doesn't cite anything to justify its claims either :-). You'd want much higher statement coverage, and also measure branch coverage, if software errors are serious (e.g., if someone could be physically harmed by an error). You should focus on creating good bang-for-buck tests first; code coverage is then a useful tool to help identify "code that isn't getting well-tested at all." It's also useful as a warning: 100% coverage may still be poorly tested, but low coverage (say less than 70%) means the program definitely has a terrible test suite. This statement is misleading: "You can have a project with 100% code coverage that still has bugs and problems." That's true for ANY program, regardless of its testing regime, because any testing regime can only test an astronomically small fraction of the possible input space. A program that just adds 2 64-bit numbers has 2^128 possible inputs; real programs have more.

kkapelon 8y ago
>That would mean that 80% of your code is completely untested That is not always a bad thing. Maybe depending on the application this 80% is trivial and never breaks. It is explained in anti-pattern 4 that you should start with the critical code first. >Most organizations I've worked with recommend at least 80% statement coverage, as a rule of thumb This number is making MANY assumptions. I would demand different code coverage from an application that runs on a nuclear reactor and from an application that is used as point of sale in a small pizza restaurant.